Convex Functions (I)
Lijun [email protected]://cs.nju.edu.cn/zlj
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenβs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenβs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Convex Function
is convex if is convex
Convex Function
is convex if is convex
is strictly convex if
Convex Function
is convex if is convex
is concave if is convex is convex
Affine functions are both convex and concave, and vice versa.
Extended-value Extensions
The extended-value extension of is
Example
Extended-value Extensions
The extended-value extension of is
Example Indicator Function of a Set
Zeroth-order Condition
Definition High-dimensional space
A function is convex if and only if it is convex when restricted to any line that intersects its domain. , is convex is convex One-dimensional space
First-order Conditions
is differentiable. Then is convex if and only if is convex For all
π π¦ π π₯ π»π π₯ π¦ π₯
First-order Taylor approximation
First-order Conditions
is differentiable. Then is convex if and only if is convex For all
Local Information Global Information
is strictly convex if and only if
π π¦ π π₯ π»π π₯ π¦ π₯
π π¦ π π₯ π»π π₯ π¦ π₯
Proof
is convex
Necessary condition:π π₯ π‘ π¦ π₯ 1 π‘ π π₯ π‘π π¦ , 0 π‘ 1
β π π¦ π π₯β
π π¦ π π₯ π π₯ π¦ π₯
Sufficient condition:π§ ππ₯ 1 π π¦
π π₯ π π§ π π§ π₯ π§π π¦ π π§ π π§ π¦ π§
β π π₯ π π§ 1 π π π§ π₯ π¦π π¦ π π§ ππ π§ π₯ π¦
β ππ π₯ 1 π π π¦ π π§ β π ππ₯ 1 π π¦ ππ π₯ 1 π π π¦
Proof
is convex
is convex
π is convex β is convex β π 1 π 0π 0 β π π¦ π π₯ π»π π₯ π¦ π₯
π π‘ π π‘π¦ 1 π‘ π₯ , πβ² π‘ π»π π‘π¦ 1 π‘ π₯ π¦ π₯
π is convex
π is convex
First-order condition of π
First-order condition of π
Proof
is convex
is convex
β π π‘ π π‘ π π‘ π‘ π‘ β π π‘ is convex βπ is convex
π π‘ π π‘π¦ 1 π‘ π₯ ,
π π‘π¦ 1 π‘ π₯ π π‘π¦ 1 π‘ π₯ π»π π‘π¦ 1 π‘ π₯ π¦ π₯ π‘ π‘
πβ² π‘ π»π π‘π¦ 1 π‘ π₯ π¦ π₯
π is convex
π is convex
First-order condition of π
First-order condition of π
Second-order Conditions
is twice differentiable. Then is convex if and only if is convex For all ,
Attention is strictly convex is strict convex
is strict convex but is convex is necessary,
Examples
Functions on is convex on ,
is convex on when or ,
and concave for
, for , is convex on
is concave on
Negative entropy is convex on
Examples
Functions on Every norm on is convex
Quadratic-over-linear: dom π π₯, π¦ β π π¦ 0
/ is concave on
is concave on
Examples
Functions on Every norm on is convex
π π₯ is a norm on π
π ππ₯ 1 π π¦ π ππ₯ π 1 π π¦
ππ π₯ 1 π π π¦
π ππ₯ 1 π π¦ max ππ₯ 1 π π¦
πmax π₯ 1 π max π¦
Examples
Functions on
π» π π₯, π¦ π¦ π₯π¦π₯π¦ π₯
π¦π₯ π¦
π₯ β½ 0
Examples
Functions on
Examples
Functions on
π» π π₯π
π π§ diag π§ π§π§
π§ π , β¦ π
π£ π» π π₯ π£π
β π§ β π£ π§
β π£ π§ 0
Cauchy-Schwarz inequality: π π π π
π π
Examples
Functions on is concave on
π π‘ π π π‘π , π π‘π β» 0, π β» 0
π π‘ log det π π‘π
log det π πΌ π‘π ππ π
β log 1 π‘π log det π
π , β¦ π are the eigenvalues of π ππ
π π‘ β , π π‘ β
det π΄π΅ det π΄ det π΅ https://en.wikipedia.org/wiki/Determinant
Sublevel Sets
-sublevel set
is convex is convex is convex is convex
-superlevel set
is concave convex
is convex
πΆ π₯ β dom π π π₯ πΌ
πΆ π₯ β dom π π π₯ πΌ
Epigraph
Graph of function
Epigraph of function
Epigraph
Epigraph of function
Hypograph
Conditions is convex is convex is concave is convex
Example
Matrix Fractional Function
Quadratic-over-linear:
Schur complement condition is convex Recall Example 2.10 in the book
π π₯, π π₯ π π₯, dom π π π
Application of Epigraph
First order Condition π π¦ π π₯ π»π π₯ π¦ π₯
π¦, π‘ β epi π β π‘ π π¦ π π₯ π»π π₯ π¦ π₯
Application of Epigraph
First order Condition π π¦ π π₯ π»π π₯ π¦ π₯
π¦, π‘ β epi π β π‘ π π₯ π»π π₯ π¦ π₯
π¦, π‘ β epi π β π»π π₯1
π¦π‘
π₯π π₯ 0
Jensenβs Inequality
Basic inequality
points
Jensenβs Inequality
Infinite points
π π₯ ππ π₯ π§ , π§ is a zero-mean noisy
HΓΆlderβs inequality
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenβs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Nonnegative Weighted Sums Finite sums π€ 0, π is convex π π€ π β― π€ π is convex
Infinite sums π π₯, π¦ is convex in π₯, βπ¦ β π, π€ π¦ 0 π π₯ π π₯, π¦ π€ π¦π ππ¦ is convex
Epigraph interpretation ππ©π’ π€π π₯, π‘ |π€π π₯ π‘
πΌ 00 π€ ππ©π’ π π₯, π€π‘ |π π₯ π‘
ππ©π’ π€π πΌ 00 π€ ππ©π’ π
Composition with an affine mapping
Affine Mapping
If is convex, so is
If is concave, so is
π π₯ π π΄π₯ π
Pointwise Maximum
is convex
is convex with
max π ππ₯ 1 π π¦ , π ππ₯ 1 π π¦max ππ π₯ 1 π π π¦ , ππ π₯ 1
π π π¦π max π π₯ , π π₯ 1 π max π π¦ , π π¦ππ π₯ 1 π π π¦
π , β¦ π is convex β π π₯ max π π₯ , β¦ π π₯
Examples
Piecewise-linear functions
Sum of largest components
is convex
Pointwise maximum of !! !
linear functions
Pointwise Supremum
is convex in
is convex with
βπ
Epigraph interpretation βπ
Intersection of convex sets is convex Pointwise infimum of a set of
concave functions is concave
βπ
Examples
Support function of a set
β
Distance to farthest point of a set
β
Examples
Maximum eigenvalue of a symmetric matrix
Norm of a matrix is maximum singular value
of
Representation
Almost every convex function can be expressed as the pointwise supremum of a family of affine functions.
βΉ π π₯ sup π π₯ |π affine, π z π π§ βπ§
π: π β π is convex and dom π π
Compositions
Definition
Chain Rule
π π₯ β π π₯
π» π π₯ β π π₯ π» π π₯ β π π₯ π»π π₯ π»π π₯
Scalar Composition
and are twice differentiable
is convex, if
β is convex and nondecreasing π is convex
β is convex and nonincreasing, π is concave
π π₯ β π π₯ πβ² π₯ β π π₯ πβ²β² π₯
Scalar Composition
and are twice differentiable
is concave, if
β is concave and nondecreasing π is concave
β is concave and nonincreasing, π is convex
π π₯ β π π₯ πβ² π₯ β π π₯ πβ²β² π₯
Without differentiability assumption Without domain condition
β π₯ 0 with dom β 1,2 , which is convex and non-decreasing
π π₯ π₯ with dom π π, which is convex
dom π 2, 1 βͺ 1, 2
π π₯ β π π₯ 0
Scalar Composition
Without differentiability assumption Without domain condition
β is convex, β is nondecreasing, and π is convex β π is convex
β is convex, β is nonincreasing, and π is concave β π is convex
The conditions for concave are similar
Scalar Composition
Extended-value Extensions
Examples is convex is convex is concave and positive is
concave is concave and positive is
convex is convex and nonnegative and
is convex is convex is convex on
Vector Composition
β and π are twice differentiable dom π π, dom β π
π β β π β π π₯ , β¦ , π π₯
π π₯ π π₯ π» β π π₯ πβ² π₯ π»β π π₯ πβ²β² π₯
πβ² π₯ π»β π π₯ πβ² π₯
Vector Composition
β and π are twice differentiable dom π π, dom β π
π is convex, if π π₯ 0 β is convex, β is nondecreasing in each
argument, and π are convex β is convex, β is nonincreasing in each
argument, and π are concave
π β β π β π π₯ , β¦ , π π₯
π π₯ π π₯ π» β π π₯ πβ² π₯ π»β π π₯ πβ²β² π₯
Vector Composition
β and π are twice differentiable dom π π, dom β π
π is concave, if π π₯ 0 β is concave, β is nondecreasing in each
argument, and π are concave
The general case is similar
π β β π β π π₯ , β¦ , π π₯
π π₯ π π₯ π» β π π₯ πβ² π₯ π»β π π₯ πβ²β² π₯
Examples β π§ π§ β― π§ , π§ β π , π , β¦ , π is convex
β β β π is convex
β π§ log β π , π , β¦ , π is convex β β βπ is convex
β π§ β π§/ on π is concave for 0 π
1, and its extension is nondecreasing. If π is concave and nonnegative β β β π is concave
Minimization is convex in is convex π π₯ inf
βπ π₯, π¦ is convex if π π₯
β, β π₯ β dom π dom π π₯ π₯, π¦ β dom π for some π¦ β πΆ
Proof by Epigraph epi π π₯, π‘ | π₯, π¦, π‘ β epi π for some π¦ β πΆ The projection of a convex set is convex.
Examples Schur complement π π₯, π¦ π₯ π΄π₯ 2π₯ π΅π¦ π¦ πΆπ¦ π΄ π΅
π΅ πΆ β½ 0, π΄, πΆ is symmetric β π π₯, π¦ is convex π π₯ inf π π₯, π¦ π₯ π΄ π΅πΆ π΅ π₯ is convex
β π΄ π΅πΆ π΅ β½ 0, πΆ is the pseudo-inverse of πΆ
Distance to a set π is a convex nonempty set,π π₯, π¦ βπ₯ π¦β is
convex in π₯, π¦ π π₯ dist π₯, π inf
ββπ₯ π¦β
Examples
Affine domain β π¦ is convex π π₯ inf β π¦ |π΄π¦ π₯ is convex
Proof
π π₯, π¦ β π¦ if π΄π¦ π₯ β otherwise
π π₯, π¦ is convex in π₯, π¦ π is the minimum of π over π¦
Perspective of a function
defined as
is the perspective of dom π π₯, π‘ |π₯/π‘ β dom π, π‘ 0 π is convex β π is convex
Proof
Perspective mapping preserve convexity
π₯, π‘, π β epi π β π‘ππ₯π‘ π
β ππ₯π‘
π π‘
β π₯/π‘, π /π‘ β epi π
Example
Euclidean norm squared
Composition with an Affine function is convex
is convex
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenβs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Summary Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenβs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function