Lecture 9 Basic Convex Analysis
April 22 - 24 2020
Basic Convex Analysis Lecture 9 April 22 - 24 2020 1 30
1 Notation
R = (minusinfin+infin) field of real numbers
Rn n-dimensional Euclidean space with inner product 〈middot middot〉Givens sets A and B A sube B denotes that A is a subset (possiblyequal to) B and A sub B means that A is a strict subset of B intAand clA denote the interior and the closure of A respectively
Given a norm 983042 middot 983042 its dual norm 983042 middot 983042lowast is defined as
983042z983042lowast = sup〈zx〉 | 983042x983042 le 1
We have983042x983042 = sup〈zx〉 | 983042z983042lowast le 1
ℓp norm (1 le p le infin)
983042x983042p =
983091
983107n983131
j=1
|xj |p983092
9831081p
p isin [1infin) 983042x983042infin = maxj
|xj |
Basic Convex Analysis Lecture 9 April 22 - 24 2020 2 30
Holderrsquos inequality
for p q isin [1infin] satisfying1
p+
1
q= 1
〈xy〉 le 983042x983042p983042y983042q
and moreover that 983042 middot 983042p and 983042 middot 983042q are dual norms
Generalized Cauchy-Schwarz inequality
for any pair of dual norms 983042 middot 983042 and 983042 middot 983042lowast
〈xy〉 le 983042x983042983042y983042lowast
Fenchel-Young inequality
for any pair of dual norms 983042 middot 983042 983042 middot 983042lowast and any η gt 0
〈xy〉 le η
2983042x9830422 + 1
2η983042y9830422lowast
Basic Convex Analysis Lecture 9 April 22 - 24 2020 3 30
2 Convex sets
The term ldquoconvexrdquo can be applied both to sets and to functions
A set C isin Rn is a convex set if the straight line segment connectingany two points in C lies entirely inside C Formally
forall xy isin C α isin [0 1] αx+ (1minus α)y isin C
Example A convex set (left) and a non-convex set (right)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 4 30
21 Basic properties of convex sets
If α isin R and C is convex then
αC = αx x isin C
is convex
The Minkowski sum of two convex sets C1 and C2 defined by
C1 + C2 = x1 + x2 x1 isin C1x2 isin C2
is convex
If αi isin R and all Ci are convex then
C =
m983131
i=1
αiCi
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 5 30
If Cα are convex sets for each α isin A where A is an arbitrary indexset then the intersection
C =983135
αisinACα
is convex
The convex hull of a set of points x1 middot middot middot xm isin Rn defined by
convx1 middot middot middot xm =
983083m983131
i=1
λixi λi ge 0
m983131
i=1
λi = 1
983084
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 6 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
1 Notation
R = (minusinfin+infin) field of real numbers
Rn n-dimensional Euclidean space with inner product 〈middot middot〉Givens sets A and B A sube B denotes that A is a subset (possiblyequal to) B and A sub B means that A is a strict subset of B intAand clA denote the interior and the closure of A respectively
Given a norm 983042 middot 983042 its dual norm 983042 middot 983042lowast is defined as
983042z983042lowast = sup〈zx〉 | 983042x983042 le 1
We have983042x983042 = sup〈zx〉 | 983042z983042lowast le 1
ℓp norm (1 le p le infin)
983042x983042p =
983091
983107n983131
j=1
|xj |p983092
9831081p
p isin [1infin) 983042x983042infin = maxj
|xj |
Basic Convex Analysis Lecture 9 April 22 - 24 2020 2 30
Holderrsquos inequality
for p q isin [1infin] satisfying1
p+
1
q= 1
〈xy〉 le 983042x983042p983042y983042q
and moreover that 983042 middot 983042p and 983042 middot 983042q are dual norms
Generalized Cauchy-Schwarz inequality
for any pair of dual norms 983042 middot 983042 and 983042 middot 983042lowast
〈xy〉 le 983042x983042983042y983042lowast
Fenchel-Young inequality
for any pair of dual norms 983042 middot 983042 983042 middot 983042lowast and any η gt 0
〈xy〉 le η
2983042x9830422 + 1
2η983042y9830422lowast
Basic Convex Analysis Lecture 9 April 22 - 24 2020 3 30
2 Convex sets
The term ldquoconvexrdquo can be applied both to sets and to functions
A set C isin Rn is a convex set if the straight line segment connectingany two points in C lies entirely inside C Formally
forall xy isin C α isin [0 1] αx+ (1minus α)y isin C
Example A convex set (left) and a non-convex set (right)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 4 30
21 Basic properties of convex sets
If α isin R and C is convex then
αC = αx x isin C
is convex
The Minkowski sum of two convex sets C1 and C2 defined by
C1 + C2 = x1 + x2 x1 isin C1x2 isin C2
is convex
If αi isin R and all Ci are convex then
C =
m983131
i=1
αiCi
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 5 30
If Cα are convex sets for each α isin A where A is an arbitrary indexset then the intersection
C =983135
αisinACα
is convex
The convex hull of a set of points x1 middot middot middot xm isin Rn defined by
convx1 middot middot middot xm =
983083m983131
i=1
λixi λi ge 0
m983131
i=1
λi = 1
983084
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 6 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Holderrsquos inequality
for p q isin [1infin] satisfying1
p+
1
q= 1
〈xy〉 le 983042x983042p983042y983042q
and moreover that 983042 middot 983042p and 983042 middot 983042q are dual norms
Generalized Cauchy-Schwarz inequality
for any pair of dual norms 983042 middot 983042 and 983042 middot 983042lowast
〈xy〉 le 983042x983042983042y983042lowast
Fenchel-Young inequality
for any pair of dual norms 983042 middot 983042 983042 middot 983042lowast and any η gt 0
〈xy〉 le η
2983042x9830422 + 1
2η983042y9830422lowast
Basic Convex Analysis Lecture 9 April 22 - 24 2020 3 30
2 Convex sets
The term ldquoconvexrdquo can be applied both to sets and to functions
A set C isin Rn is a convex set if the straight line segment connectingany two points in C lies entirely inside C Formally
forall xy isin C α isin [0 1] αx+ (1minus α)y isin C
Example A convex set (left) and a non-convex set (right)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 4 30
21 Basic properties of convex sets
If α isin R and C is convex then
αC = αx x isin C
is convex
The Minkowski sum of two convex sets C1 and C2 defined by
C1 + C2 = x1 + x2 x1 isin C1x2 isin C2
is convex
If αi isin R and all Ci are convex then
C =
m983131
i=1
αiCi
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 5 30
If Cα are convex sets for each α isin A where A is an arbitrary indexset then the intersection
C =983135
αisinACα
is convex
The convex hull of a set of points x1 middot middot middot xm isin Rn defined by
convx1 middot middot middot xm =
983083m983131
i=1
λixi λi ge 0
m983131
i=1
λi = 1
983084
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 6 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
2 Convex sets
The term ldquoconvexrdquo can be applied both to sets and to functions
A set C isin Rn is a convex set if the straight line segment connectingany two points in C lies entirely inside C Formally
forall xy isin C α isin [0 1] αx+ (1minus α)y isin C
Example A convex set (left) and a non-convex set (right)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 4 30
21 Basic properties of convex sets
If α isin R and C is convex then
αC = αx x isin C
is convex
The Minkowski sum of two convex sets C1 and C2 defined by
C1 + C2 = x1 + x2 x1 isin C1x2 isin C2
is convex
If αi isin R and all Ci are convex then
C =
m983131
i=1
αiCi
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 5 30
If Cα are convex sets for each α isin A where A is an arbitrary indexset then the intersection
C =983135
αisinACα
is convex
The convex hull of a set of points x1 middot middot middot xm isin Rn defined by
convx1 middot middot middot xm =
983083m983131
i=1
λixi λi ge 0
m983131
i=1
λi = 1
983084
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 6 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
21 Basic properties of convex sets
If α isin R and C is convex then
αC = αx x isin C
is convex
The Minkowski sum of two convex sets C1 and C2 defined by
C1 + C2 = x1 + x2 x1 isin C1x2 isin C2
is convex
If αi isin R and all Ci are convex then
C =
m983131
i=1
αiCi
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 5 30
If Cα are convex sets for each α isin A where A is an arbitrary indexset then the intersection
C =983135
αisinACα
is convex
The convex hull of a set of points x1 middot middot middot xm isin Rn defined by
convx1 middot middot middot xm =
983083m983131
i=1
λixi λi ge 0
m983131
i=1
λi = 1
983084
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 6 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
If Cα are convex sets for each α isin A where A is an arbitrary indexset then the intersection
C =983135
αisinACα
is convex
The convex hull of a set of points x1 middot middot middot xm isin Rn defined by
convx1 middot middot middot xm =
983083m983131
i=1
λixi λi ge 0
m983131
i=1
λi = 1
983084
is convex
Basic Convex Analysis Lecture 9 April 22 - 24 2020 6 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Theorem 1 (Projection onto closed convex sets)
Let C be a closed convex set and x isin Rn Then there is a unique pointπC(x) called the projection of x onto C such that
983042xminus πC(x)9830422 = infyisinC
983042y minus x9830422
that isπC(x) = argmin
yisinC983042y minus x98304222
A point z = πC(x) is the projection of x onto C if and only if
〈xminus zy minus z〉 le 0
for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 7 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Projection of the point x onto the set C (with projection πC(x))exhibiting 〈xminus πC(x)y minus πC(x)〉 le 0
Basic Convex Analysis Lecture 9 April 22 - 24 2020 8 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Corollary 2 (Nonexpansiveness)
Projections onto convex sets are nonexpansive in particular
983042πC(x)minus y9830422 le 983042xminus y9830422
for any x isin Rn and y isin C
Theorem 3 (Strict separation of points)
Let C be a closed convex set Given any point x isin C there is a vector vsuch that
〈vx〉 gt supyisinC
〈vy〉
Moreover we can take the vector v = xminus πC(x) and
〈vx〉 ge supyisinC
〈vy〉+ 983042v98304222
Basic Convex Analysis Lecture 9 April 22 - 24 2020 9 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Separation of the point x from C by the vector v = xminus πC(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 10 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
For nonempty sets S1 and S2 satisfying S1 cap S2 = empty if there existvector v ∕= 0 and scalar b such that
〈vx〉 ge b for all x isin S1
and〈vx〉 le b for all x isin S2
thenx isin Rn | 〈vx〉 = b
is called a separating hyperplane for nonempty sets S1 and S2
Theorem 4 (Strict separation of convex sets)
Let C1 C2 be closed convex sets with C2 compact and C1 cap C2 = empty Thenthere is a vector v such that
infxisinC1
〈vx〉 gt supxisinC2
〈vx〉
Basic Convex Analysis Lecture 9 April 22 - 24 2020 11 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Strict separation of convex sets
Basic Convex Analysis Lecture 9 April 22 - 24 2020 12 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
For a set S and x isin bdS = clS intS if vector v ∕= 0 satisfies
〈vx〉 ge 〈vy〉 for all y isin S
thenz isin Rn | vT(zminus x) = 0
is called a supporting hyperplane supporting S at x
Theorem 5 (Supporting hyperplane theorem)
For convex set C and any x isin bdC theres exists a supportinghyperplane supporting C at x ie exist v ∕= 0 satisfying
〈vx〉 ge 〈vy〉 for all y isin C
Basic Convex Analysis Lecture 9 April 22 - 24 2020 13 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Supporting hyperplanes to a convex set
Basic Convex Analysis Lecture 9 April 22 - 24 2020 14 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Is a supporting hyperplane supporting C at x unique
Theorem 6 (Halfspace intersections)
Let C sub Rn be a closed convex set Then C is the intersection of all thehalfspaces containing it
Basic Convex Analysis Lecture 9 April 22 - 24 2020 15 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
3 Convex functions
A function f Rn 983041rarr R cup +infin is a convex function if its domaindom(f) is convex and for all xy isin dom(f) α isin [0 1]
f(αx+ (1minus α)y) le αf(x) + (1minus α)f(y)
(strictly convex means lt)The epigraph of a function f is defined as
epif = (x t) f(x) le tA function is convex if and only if its epigraph is a convex setExample Convex function f(x) = maxx2minus2xminus 02
Basic Convex Analysis Lecture 9 April 22 - 24 2020 16 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Theorem 7 (First-order convexity condition)
Differentiable f is convex if and only if dom(f) is convex and
f(y) ge f(x) +nablaf(x)T(y minus x) forall xy isin dom(f)
Theorem 8 (Second-order convexity conditions)
Assume f is twice continuously differentiable Then f is convex if andonly if dom(f) is convex and
nabla2f(x) ≽ 0 forall x isin dom(f)
that is nabla2f(x) is positive semidefinite
A function f is called closed if its epigraph is a closed set
Theorem 9 (Continuous over closed domain rArr closedness)
Suppose f Rn 983041rarr R cup +infin is continuous over its domain anddom(f) is closed Then f is closed
Basic Convex Analysis Lecture 9 April 22 - 24 2020 17 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Lemma 10 (Convexity + compactness rArr boundedness)
Let f be convex and defined on the ℓ1 ball in n dimensions
B1 = x isin Rn 983042x9830421 le 1
Then there exist minusinfin lt m le M lt infin such that
m le f(x) le M forall x isin B1
More general convex f on a compact domain is bounded
Theorem 11 (Convexity + compactness rArr L-continuity)
Let f be convex and defined on a convex set C with non-empty interiorLet B sube intC be compact Then there is a constant L such that
|f(x)minus f(y)| le L983042xminus y983042
on B that is f is L-Lipschitz continuous on B
Basic Convex Analysis Lecture 9 April 22 - 24 2020 18 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Definition The directional derivative of a function f at a point xin the direction d is
f prime(xd) = limαrarr0+
f(x+ αd)minus f(x)
α
Theorem 12 (Convexity rArr existence of directional derivative)
For convex f at any point x isin intdom(f) and for any d thedirectional derivative f prime(xd) exists and is
f prime(xd) = infαgt0
f(x+ αd)minus f(x)
α
Moreover g(d) = f prime(xd) is convex and there exists a constant L lt infinsuch that
|g(d)| = |f prime(xd)| le L983042d983042
for any d isin Rn If f is Lipschitz continuous with respect to the norm983042 middot 983042 we can take L to be the Lipschitz constant of f
Basic Convex Analysis Lecture 9 April 22 - 24 2020 19 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Theorem 13 (Any local minimizer of a convex function is global)
Let f Rn 983041rarr R be convex and x be a local minimizer of f (resp a localminimizer of f over a convex set C) Then x is a global minimizer of f(resp a global minimizer of f over C)
Proof If x is a local minimizer of f over a convex set C then for anyy isin C we have for small enough t gt 0 that
f(x) le f(x+ t(y minus x)) or 0 le f(x+ t(y minus x))minus f(x)
t
We now use the criterion of increasing slopes that is for any convexfunction f and any u isin Rn the function φ(t)
φ(t) =f(x+ tu)minus f(x)
t
is increasing in t gt 0 Therefore forallt gt 0 we have
0 le f(x+ t(y minus x))minus f(x)
t
Setting t = 1 yields f(x) le f(y) for any y isin CBasic Convex Analysis Lecture 9 April 22 - 24 2020 20 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
The slopesf(x+ t)minus f(x)
tincrease with t1 lt t2 lt t3
Basic Convex Analysis Lecture 9 April 22 - 24 2020 21 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
4 Subgradient and subdifferential
Definition A vector g isin Rn is a subgradient of f at a point x if
f(y) ge f(x) + 〈gy minus x〉 for all y isin Rn
The subdifferential denoted partf(x) is the set of all subgradients off at x
Example g1 = nablaf(x1) g2g3 isin partf(x2)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 22 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Examples
part983042x983042 =
983083g isin Rn 983042g983042lowast = 1 〈gx〉 = 983042x983042 if x ∕= 0
g isin Rn 983042g983042lowast le 1 if x = 0
For 983042x9830422 we have part983042x9830422 =983083x983042x9830422 if x ∕= 0
g isin Rn 983042g9830422 le 1 if x = 0
For case n = 1 we have part|x| =
983099983105983103
983105983101
minus1 if x lt 0
[minus1 1] if x = 0
1 if x gt 0
Theorem 14 (Nonemptiness closedness convexity boundedness ofthe subdifferential set at interior points of the domain)
Suppose f is convex Let x isin intdom(f) Then partf(x) is nonemptyclosed convex and bounded
Basic Convex Analysis Lecture 9 April 22 - 24 2020 23 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Theorem 15 (Nonemptiness of subdifferential sets rArr convexity)
Let f Rn 983041rarr R cup +infin be proper and assume that dom(f) is convexSuppose that for any x isin dom(f) the set partf(x) is nonempty Then fis convex
Theorem 16 (First-order characterizations of strong convexity)
Let f Rn 983041rarr R cup +infin be proper closed and convex Then for a givenγ gt 0 the following three claims are equivalent
(i) f is γ-strongly convex
(ii) For any x satisfying partf(x) ∕= empty y isin dom(f) and g isin partf(x)
f(y) ge f(x) + 〈gy minus x〉+ γ
2983042y minus x9830422
(iii) For any x and y satisfying partf(x) ∕= empty partf(y) ∕= empty and gx isin partf(x)gy isin partf(y)
〈gx minus gyxminus y〉 ge γ983042xminus y9830422
Basic Convex Analysis Lecture 9 April 22 - 24 2020 24 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Theorem 17 (Equivalent characterization of subdifferential)
An equivalent characterization of the subdifferential partf(x) of convex fat x is
partf(x) = g 〈gd〉 le f prime(xd) forall d isin Rn
Theorem 18 (Max formula of directional derivative)
Suppose f is closed convex and partf(x) ∕= empty Then
f prime(xd) = supgisinpartf(x)
〈gd〉
Theorem 19 (Subgradient bounded by Lipschitz constant)
Suppose that convex function f is L-Lipschitz continuous with respectto the norm 983042 middot 983042 over a set C where C sub intdom(f) Then
sup983042g983042lowast g isin partf(x)x isin C le L
Basic Convex Analysis Lecture 9 April 22 - 24 2020 25 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Theorem 20 (Minimizer of convex function over convex set)
Let f be convex The point x983183 isin intdom(f) minimizes f over a closedconvex set C if and only if there exists a subgradient g isin partf(x983183) suchthat 〈gy minus x983183〉 ge 0 for all y isin C
The point x983183 minimizes f
over C(the shown level curves)
Active case x983183 isin bdCminusg supporting hyperplane
Inactive case x983183 isin intCg = 0 rArr 0 isin partf(x983183)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 26 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
5 Calculus rules with subgradients
Scaling
If h(x) = αf(x) for some α ge 0 then parth(x) = αf(x)
Finite sums
Suppose that fi i = 1 m are convex functions and let f =
m983131
i=1
fi
If x isin intdom(fi) i = 1 m then partf(x) =
m983131
i=1
partfi(x)
Exercise x isin Rm 983042x9830421 =983123m
i=1 fi(x) fi(x) = |xi| part983042x9830421 =
Affine transformations
Let f Rm 983041rarr R be convex and A isin Rmtimesn and b isin Rm Thenh Rn 983041rarr R defined by h(x) = f(Ax+ b) is convex and hassubdifferential
parth(x) = ATpartf(Ax+ b)
Exercises (1) proof (2) part983042Ax+ b9830421= (3) part983042Ax+ b9830422 =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 27 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Maximum of a finite collection of convex functions
Let fi i = 1 m be convex functions and f(x) = max1leilem
fi(x)
Then we haveepif =
983135
1leilem
epifi
which is convex and therefore f is convex
If x isin intdom(fi) i = 1 m then the subdifferential partf(x) is theconvex hull of the subgradients of active functions (those attainingthe maximum) at x that is
partf(x) = convpartfi(x) fi(x) = f(x)
If there is only a single unique active function fi then
partf(x) = partfi(x)
Basic Convex Analysis Lecture 9 April 22 - 24 2020 28 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Exercise x isin Rm 983042x983042infin = max1leilem
fi(x) fi(x) = |xi| part983042x983042infin =
Exercise
f(x) = maxf1(x) f2(x) f1(x) = x2 f2(x) = minus2xminus 15
x0 = minus1 +983155
45 partf(x0) =
Basic Convex Analysis Lecture 9 April 22 - 24 2020 29 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30
Supremum of an infinite collection of convex functions
Considerf(x) = sup
αisinAfα(x)
where A is an arbitrary index set and fα is convex for each α
If the supremum is attained then
partf(x) supe convpartfα(x) fα(x) = f(x)
If the supremum is not attained the function f may not besubdifferentiable at x
Basic Convex Analysis Lecture 9 April 22 - 24 2020 30 30