convex functions (i)vector composition ร ร โand ๐are twice differentiable dom ๐ ร๐,dom...
TRANSCRIPT
Convex Functions (I)
Lijun [email protected]://cs.nju.edu.cn/zlj
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenโs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenโs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Convex Function
is convex if is convex
Convex Function
is convex if is convex
is strictly convex if
Convex Function
is convex if is convex
is concave if is convex is convex
Affine functions are both convex and concave, and vice versa.
Extended-value Extensions
The extended-value extension of is
Example
Extended-value Extensions
The extended-value extension of is
Example Indicator Function of a Set
Zeroth-order Condition
Definition High-dimensional space
A function is convex if and only if it is convex when restricted to any line that intersects its domain. , is convex is convex One-dimensional space
First-order Conditions
is differentiable. Then is convex if and only if is convex For all
๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
First-order Taylor approximation
First-order Conditions
is differentiable. Then is convex if and only if is convex For all
Local Information Global Information
is strictly convex if and only if
๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
Proof
is convex
Necessary condition:๐ ๐ฅ ๐ก ๐ฆ ๐ฅ 1 ๐ก ๐ ๐ฅ ๐ก๐ ๐ฆ , 0 ๐ก 1
โ ๐ ๐ฆ ๐ ๐ฅโ
๐ ๐ฆ ๐ ๐ฅ ๐ ๐ฅ ๐ฆ ๐ฅ
Sufficient condition:๐ง ๐๐ฅ 1 ๐ ๐ฆ
๐ ๐ฅ ๐ ๐ง ๐ ๐ง ๐ฅ ๐ง๐ ๐ฆ ๐ ๐ง ๐ ๐ง ๐ฆ ๐ง
โ ๐ ๐ฅ ๐ ๐ง 1 ๐ ๐ ๐ง ๐ฅ ๐ฆ๐ ๐ฆ ๐ ๐ง ๐๐ ๐ง ๐ฅ ๐ฆ
โ ๐๐ ๐ฅ 1 ๐ ๐ ๐ฆ ๐ ๐ง โ ๐ ๐๐ฅ 1 ๐ ๐ฆ ๐๐ ๐ฅ 1 ๐ ๐ ๐ฆ
Proof
is convex
is convex
๐ is convex โ is convex โ ๐ 1 ๐ 0๐ 0 โ ๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
๐ ๐ก ๐ ๐ก๐ฆ 1 ๐ก ๐ฅ , ๐โฒ ๐ก ๐ป๐ ๐ก๐ฆ 1 ๐ก ๐ฅ ๐ฆ ๐ฅ
๐ is convex
๐ is convex
First-order condition of ๐
First-order condition of ๐
Proof
is convex
is convex
โ ๐ ๐ก ๐ ๐ก ๐ ๐ก ๐ก ๐ก โ ๐ ๐ก is convex โ๐ is convex
๐ ๐ก ๐ ๐ก๐ฆ 1 ๐ก ๐ฅ ,
๐ ๐ก๐ฆ 1 ๐ก ๐ฅ ๐ ๐ก๐ฆ 1 ๐ก ๐ฅ ๐ป๐ ๐ก๐ฆ 1 ๐ก ๐ฅ ๐ฆ ๐ฅ ๐ก ๐ก
๐โฒ ๐ก ๐ป๐ ๐ก๐ฆ 1 ๐ก ๐ฅ ๐ฆ ๐ฅ
๐ is convex
๐ is convex
First-order condition of ๐
First-order condition of ๐
Second-order Conditions
is twice differentiable. Then is convex if and only if is convex For all ,
Attention is strictly convex is strict convex
is strict convex but is convex is necessary,
Examples
Functions on is convex on ,
is convex on when or ,
and concave for
, for , is convex on
is concave on
Negative entropy is convex on
Examples
Functions on Every norm on is convex
Quadratic-over-linear: dom ๐ ๐ฅ, ๐ฆ โ ๐ ๐ฆ 0
/ is concave on
is concave on
Examples
Functions on Every norm on is convex
๐ ๐ฅ is a norm on ๐
๐ ๐๐ฅ 1 ๐ ๐ฆ ๐ ๐๐ฅ ๐ 1 ๐ ๐ฆ
๐๐ ๐ฅ 1 ๐ ๐ ๐ฆ
๐ ๐๐ฅ 1 ๐ ๐ฆ max ๐๐ฅ 1 ๐ ๐ฆ
๐max ๐ฅ 1 ๐ max ๐ฆ
Examples
Functions on
๐ป ๐ ๐ฅ, ๐ฆ ๐ฆ ๐ฅ๐ฆ๐ฅ๐ฆ ๐ฅ
๐ฆ๐ฅ ๐ฆ
๐ฅ โฝ 0
Examples
Functions on
Examples
Functions on
๐ป ๐ ๐ฅ๐
๐ ๐ง diag ๐ง ๐ง๐ง
๐ง ๐ , โฆ ๐
๐ฃ ๐ป ๐ ๐ฅ ๐ฃ๐
โ ๐ง โ ๐ฃ ๐ง
โ ๐ฃ ๐ง 0
Cauchy-Schwarz inequality: ๐ ๐ ๐ ๐
๐ ๐
Examples
Functions on is concave on
๐ ๐ก ๐ ๐ ๐ก๐ , ๐ ๐ก๐ โป 0, ๐ โป 0
๐ ๐ก log det ๐ ๐ก๐
log det ๐ ๐ผ ๐ก๐ ๐๐ ๐
โ log 1 ๐ก๐ log det ๐
๐ , โฆ ๐ are the eigenvalues of ๐ ๐๐
๐ ๐ก โ , ๐ ๐ก โ
det ๐ด๐ต det ๐ด det ๐ต https://en.wikipedia.org/wiki/Determinant
Sublevel Sets
-sublevel set
is convex is convex is convex is convex
-superlevel set
is concave convex
is convex
๐ถ ๐ฅ โ dom ๐ ๐ ๐ฅ ๐ผ
๐ถ ๐ฅ โ dom ๐ ๐ ๐ฅ ๐ผ
Epigraph
Graph of function
Epigraph of function
Epigraph
Epigraph of function
Hypograph
Conditions is convex is convex is concave is convex
Example
Matrix Fractional Function
Quadratic-over-linear:
Schur complement condition is convex Recall Example 2.10 in the book
๐ ๐ฅ, ๐ ๐ฅ ๐ ๐ฅ, dom ๐ ๐ ๐
Application of Epigraph
First order Condition ๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
๐ฆ, ๐ก โ epi ๐ โ ๐ก ๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
Application of Epigraph
First order Condition ๐ ๐ฆ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
๐ฆ, ๐ก โ epi ๐ โ ๐ก ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ฆ ๐ฅ
๐ฆ, ๐ก โ epi ๐ โ ๐ป๐ ๐ฅ1
๐ฆ๐ก
๐ฅ๐ ๐ฅ 0
Jensenโs Inequality
Basic inequality
points
Jensenโs Inequality
Infinite points
๐ ๐ฅ ๐๐ ๐ฅ ๐ง , ๐ง is a zero-mean noisy
Hรถlderโs inequality
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenโs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Nonnegative Weighted Sums Finite sums ๐ค 0, ๐ is convex ๐ ๐ค ๐ โฏ ๐ค ๐ is convex
Infinite sums ๐ ๐ฅ, ๐ฆ is convex in ๐ฅ, โ๐ฆ โ ๐, ๐ค ๐ฆ 0 ๐ ๐ฅ ๐ ๐ฅ, ๐ฆ ๐ค ๐ฆ๐ ๐๐ฆ is convex
Epigraph interpretation ๐๐ฉ๐ข ๐ค๐ ๐ฅ, ๐ก |๐ค๐ ๐ฅ ๐ก
๐ผ 00 ๐ค ๐๐ฉ๐ข ๐ ๐ฅ, ๐ค๐ก |๐ ๐ฅ ๐ก
๐๐ฉ๐ข ๐ค๐ ๐ผ 00 ๐ค ๐๐ฉ๐ข ๐
Composition with an affine mapping
Affine Mapping
If is convex, so is
If is concave, so is
๐ ๐ฅ ๐ ๐ด๐ฅ ๐
Pointwise Maximum
is convex
is convex with
max ๐ ๐๐ฅ 1 ๐ ๐ฆ , ๐ ๐๐ฅ 1 ๐ ๐ฆmax ๐๐ ๐ฅ 1 ๐ ๐ ๐ฆ , ๐๐ ๐ฅ 1
๐ ๐ ๐ฆ๐ max ๐ ๐ฅ , ๐ ๐ฅ 1 ๐ max ๐ ๐ฆ , ๐ ๐ฆ๐๐ ๐ฅ 1 ๐ ๐ ๐ฆ
๐ , โฆ ๐ is convex โ ๐ ๐ฅ max ๐ ๐ฅ , โฆ ๐ ๐ฅ
Examples
Piecewise-linear functions
Sum of largest components
is convex
Pointwise maximum of !! !
linear functions
Pointwise Supremum
is convex in
is convex with
โ๐
Epigraph interpretation โ๐
Intersection of convex sets is convex Pointwise infimum of a set of
concave functions is concave
โ๐
Examples
Support function of a set
โ
Distance to farthest point of a set
โ
Examples
Maximum eigenvalue of a symmetric matrix
Norm of a matrix is maximum singular value
of
Representation
Almost every convex function can be expressed as the pointwise supremum of a family of affine functions.
โน ๐ ๐ฅ sup ๐ ๐ฅ |๐ affine, ๐ z ๐ ๐ง โ๐ง
๐: ๐ โ ๐ is convex and dom ๐ ๐
Compositions
Definition
Chain Rule
๐ ๐ฅ โ ๐ ๐ฅ
๐ป ๐ ๐ฅ โ ๐ ๐ฅ ๐ป ๐ ๐ฅ โ ๐ ๐ฅ ๐ป๐ ๐ฅ ๐ป๐ ๐ฅ
Scalar Composition
and are twice differentiable
is convex, if
โ is convex and nondecreasing ๐ is convex
โ is convex and nonincreasing, ๐ is concave
๐ ๐ฅ โ ๐ ๐ฅ ๐โฒ ๐ฅ โ ๐ ๐ฅ ๐โฒโฒ ๐ฅ
Scalar Composition
and are twice differentiable
is concave, if
โ is concave and nondecreasing ๐ is concave
โ is concave and nonincreasing, ๐ is convex
๐ ๐ฅ โ ๐ ๐ฅ ๐โฒ ๐ฅ โ ๐ ๐ฅ ๐โฒโฒ ๐ฅ
Without differentiability assumption Without domain condition
โ ๐ฅ 0 with dom โ 1,2 , which is convex and non-decreasing
๐ ๐ฅ ๐ฅ with dom ๐ ๐, which is convex
dom ๐ 2, 1 โช 1, 2
๐ ๐ฅ โ ๐ ๐ฅ 0
Scalar Composition
Without differentiability assumption Without domain condition
โ is convex, โ is nondecreasing, and ๐ is convex โ ๐ is convex
โ is convex, โ is nonincreasing, and ๐ is concave โ ๐ is convex
The conditions for concave are similar
Scalar Composition
Extended-value Extensions
Examples is convex is convex is concave and positive is
concave is concave and positive is
convex is convex and nonnegative and
is convex is convex is convex on
Vector Composition
โ and ๐ are twice differentiable dom ๐ ๐, dom โ ๐
๐ โ โ ๐ โ ๐ ๐ฅ , โฆ , ๐ ๐ฅ
๐ ๐ฅ ๐ ๐ฅ ๐ป โ ๐ ๐ฅ ๐โฒ ๐ฅ ๐ปโ ๐ ๐ฅ ๐โฒโฒ ๐ฅ
๐โฒ ๐ฅ ๐ปโ ๐ ๐ฅ ๐โฒ ๐ฅ
Vector Composition
โ and ๐ are twice differentiable dom ๐ ๐, dom โ ๐
๐ is convex, if ๐ ๐ฅ 0 โ is convex, โ is nondecreasing in each
argument, and ๐ are convex โ is convex, โ is nonincreasing in each
argument, and ๐ are concave
๐ โ โ ๐ โ ๐ ๐ฅ , โฆ , ๐ ๐ฅ
๐ ๐ฅ ๐ ๐ฅ ๐ป โ ๐ ๐ฅ ๐โฒ ๐ฅ ๐ปโ ๐ ๐ฅ ๐โฒโฒ ๐ฅ
Vector Composition
โ and ๐ are twice differentiable dom ๐ ๐, dom โ ๐
๐ is concave, if ๐ ๐ฅ 0 โ is concave, โ is nondecreasing in each
argument, and ๐ are concave
The general case is similar
๐ โ โ ๐ โ ๐ ๐ฅ , โฆ , ๐ ๐ฅ
๐ ๐ฅ ๐ ๐ฅ ๐ป โ ๐ ๐ฅ ๐โฒ ๐ฅ ๐ปโ ๐ ๐ฅ ๐โฒโฒ ๐ฅ
Examples โ ๐ง ๐ง โฏ ๐ง , ๐ง โ ๐ , ๐ , โฆ , ๐ is convex
โ โ โ ๐ is convex
โ ๐ง log โ ๐ , ๐ , โฆ , ๐ is convex โ โ โ๐ is convex
โ ๐ง โ ๐ง/ on ๐ is concave for 0 ๐
1, and its extension is nondecreasing. If ๐ is concave and nonnegative โ โ โ ๐ is concave
Minimization is convex in is convex ๐ ๐ฅ inf
โ๐ ๐ฅ, ๐ฆ is convex if ๐ ๐ฅ
โ, โ ๐ฅ โ dom ๐ dom ๐ ๐ฅ ๐ฅ, ๐ฆ โ dom ๐ for some ๐ฆ โ ๐ถ
Proof by Epigraph epi ๐ ๐ฅ, ๐ก | ๐ฅ, ๐ฆ, ๐ก โ epi ๐ for some ๐ฆ โ ๐ถ The projection of a convex set is convex.
Examples Schur complement ๐ ๐ฅ, ๐ฆ ๐ฅ ๐ด๐ฅ 2๐ฅ ๐ต๐ฆ ๐ฆ ๐ถ๐ฆ ๐ด ๐ต
๐ต ๐ถ โฝ 0, ๐ด, ๐ถ is symmetric โ ๐ ๐ฅ, ๐ฆ is convex ๐ ๐ฅ inf ๐ ๐ฅ, ๐ฆ ๐ฅ ๐ด ๐ต๐ถ ๐ต ๐ฅ is convex
โ ๐ด ๐ต๐ถ ๐ต โฝ 0, ๐ถ is the pseudo-inverse of ๐ถ
Distance to a set ๐ is a convex nonempty set,๐ ๐ฅ, ๐ฆ โ๐ฅ ๐ฆโ is
convex in ๐ฅ, ๐ฆ ๐ ๐ฅ dist ๐ฅ, ๐ inf
โโ๐ฅ ๐ฆโ
Examples
Affine domain โ ๐ฆ is convex ๐ ๐ฅ inf โ ๐ฆ |๐ด๐ฆ ๐ฅ is convex
Proof
๐ ๐ฅ, ๐ฆ โ ๐ฆ if ๐ด๐ฆ ๐ฅ โ otherwise
๐ ๐ฅ, ๐ฆ is convex in ๐ฅ, ๐ฆ ๐ is the minimum of ๐ over ๐ฆ
Perspective of a function
defined as
is the perspective of dom ๐ ๐ฅ, ๐ก |๐ฅ/๐ก โ dom ๐, ๐ก 0 ๐ is convex โ ๐ is convex
Proof
Perspective mapping preserve convexity
๐ฅ, ๐ก, ๐ โ epi ๐ โ ๐ก๐๐ฅ๐ก ๐
โ ๐๐ฅ๐ก
๐ ๐ก
โ ๐ฅ/๐ก, ๐ /๐ก โ epi ๐
Example
Euclidean norm squared
Composition with an Affine function is convex
is convex
Outline Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenโs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function
Summary
Summary Basic Properties
Definition First-order Conditions, Second-order Conditions Jensenโs inequality and extensions Epigraph
Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function