Convexity and Optimization
Richard Lusby
Department of Management EngineeringTechnical University of Denmark
Today’s Material(jg
Extrema
Convex Function
Convex Sets
Other Convexity Concepts
Unconstrained Optimization
R Lusby (42111) Convexity & Optimization 2/38
Extrema(jg
Problem
max f (x) s.t. x ∈ S
max {f (x) : x ∈ S}
Global maximum x∗
f (x∗) ≥ f (x) ∀x ∈ S
Local maximum xo :
f (xo) ≥ f (x) ∀x in a neighborhood around xo
strict maximum/minimum defined similarly
R Lusby (42111) Convexity & Optimization 3/38
Weierstrass theorem:(jg
Theorem
A continuous function achieves its max/min on a closed and bounded set
R Lusby (42111) Convexity & Optimization 4/38
Supremum and Infimum(jg
Supremum
The supremum of a set S having a partial order is the leastupper bound of S (if it exists) and is denoted sup S.
Infimum
The infimum of a set S having a partial order is the greatestlower bound of S (if it exists) and is denoted inf S.
If the extrema are not achieved:I max → sup
I min → inf
ExamplesI sup{2, 3, 4, 5}?I sup{x ∈ Q : x2 < 2}?I inf{1/x : x > 0}?
R Lusby (42111) Convexity & Optimization 5/38
Finding Optimal SolutionsNecessary and Sufficient Conditions(jg
Every method for finding and characterizing optimal solutions is based onoptimality conditions - either necessary or sufficient
Necessary Condition
A condition C1(x) is necessary if C1(x∗) is satisfied by everyoptimal solution x∗ (and possibly some other solutions as well).
Sufficient Condition
A condition C2(x) is sufficient if C2(x∗) ensures that x∗ isoptimal (but some optimal solutions may not satisfy C2(x∗).
Mathematically
{x|C2(x)} ⊆ {x|x optimal solution } ⊆ {x|C1(x)}
R Lusby (42111) Convexity & Optimization 6/38
Finding Optimal Solutions(jg
An example of a necessary condition in the case S is “well-behaved” noimproving feasible direction
Feasible Direction
Consider xo ∈ S , s ∈ Rn is called a feasible direction if there existsε(s) > 0 such that
xo + εs ∈ S ∀ε : 0 < ε ≤ ε(s)
We denote the cone of feasible directions from xo in S as S(xo)
Improving Direction
s ∈ Rn is called an improving direction if there exists ε(s) > 0 such that
f (xo + εs) < f (xo) ∀ε : 0 < ε ≤ ε(s)
The cone of improving directions from xo in S is denoted F (xo)R Lusby (42111) Convexity & Optimization 7/38
Finding Optimal Solutions(jg
Local Optima
If xo is a local minimum, then there exist no s ∈ S(xo) for which f(.)decreases along s, i.e. for which
f (x1 + ε2s) < f (x + ε1s) for 0 ≤ ε1 < ε2 ≤ ε(s)
Stated otherwise: A necessary condition for local optimality is
F (xo) ∩ S(xo) = ∅
R Lusby (42111) Convexity & Optimization 8/38
Improving Feasible Directions(jg
If for a given direction s it holds that
∇f (xo)s < 0
Then s is an improving direction
Well known necessary condition for local optimality of xo for adifferentiable function:
∇f (xo) = 0
In other words, xo is stationary with respect to f (.)
R Lusby (42111) Convexity & Optimization 9/38
What if Stationarity is not Enough?(jg
Suppose f is twice continuously differentiable
Analyse the Hessian matrix for f at xo
∇2f (xo) =
{∂2(f (xo))
∂xi∂xj
}, i , j = 1, . . . , n
Sufficient Condition
If ∇f (xo) = 0 and ∇2f (xo) is positive definite:
xT∇2f (xo)x > 0 ∀ x ∈ Rn\{0}
then xo is a local minimum
R Lusby (42111) Convexity & Optimization 10/38
What if Stationarity is not Enough?continued(jg
A necessary condition for local optimality is “Stationarity + positivesemidefiniteness of ∇2f (xo)”
Note that positive definiteness is not a necessary conditionI E.g. look at f (x) = x4 for xo = 0
Similar statements hold for maximization problemsI Key concept here is negative definiteness
R Lusby (42111) Convexity & Optimization 11/38
Definiteness of a Matrix(jg
A number of criteria regarding the definiteness of a matrix exist
A symmetric n × n matrix A is positive definite if and only if
xTAx > 0 ∀x ∈ Rn\{0}
Positive semidefinite is defined likewise with ′′ ≥′′ instead of ′′ >′′
Negative (semi) definite is defined by reversing the inequality signs to′′ <′′ and ′′ ≤′′, respectively.
Necessary conditions for positive definiteness:
A is regular with det(A) > 0
A−1 is positive definite
R Lusby (42111) Convexity & Optimization 12/38
Definiteness of a Matrix(jg
Necessary+Sufficient conditions for positive definiteness:
Sylvestor’s Criterion: All principal submatrices have positivedeterminants
(a11)
(a11 a12
a21 a22
) a11 a12 a13
a21 a22 a23
a31 a32 a33
All eigenvalues of A are positive
R Lusby (42111) Convexity & Optimization 13/38
Necessary and Sufficient ConditionsDifferentiable f (.)(jg
Theorem
Suppose that f (.) is differentiable at a local minimum xo . Then∇f (xo)s ≥ 0 for s ∈ S(xo). If f (.) is twice differentiable at xo and∇f (xo) = 0, then sT∇2f (xo)s ≥ 0 ∀s ∈ S(xo)
Theorem
Suppose that S is convex and non-empty, f (.) differentiable, and thatxo ∈ S . Suppose furthermore that f (.) is convex.
xo is a local minimum if and only if xo is a global minimumxo is a local (and hence global) minimum if and only if
∇f (xo)(x− xo) ≥ 0 ∀x ∈ S
R Lusby (42111) Convexity & Optimization 14/38
Convex Combination(jg
Convex combination
The convex combination of two points is the line segment between them
α1x1 + α2x2 for α1, α2 ≥ 0 and α1 + α2 = 1
R Lusby (42111) Convexity & Optimization 15/38
Convex function(jg
Convex Functions
A convex function lies below its chord
f (α1x1 + α2x2) ≤ α1f (x1) + α2f (x2)
A strictly convex function has no more than one minimum
Examples: y = x2, y = x4, y = x
The sum of convex functions is also convex
A differentiable convex function lies above its tangent
A differentiable function is convex if its Hessian is positivesemi-definite
I Strictly convex not analogous!
A function f is concave iff −f is convex
EOQ objective function: T (Q) = dK/Q + cd + hQ/2?
R Lusby (42111) Convexity & Optimization 16/38
Economic Order Quantity Model(jg
The problem
The Economic Order Quantity Model is an inventory model that helpsmanafacturers, retailers, and wholesalers determine how they shouldoptimally replenish their stock levels.
Costs
K = Setup cost for odering one batchc = unit cost for producing/purchasingh = holding cost per unit per unit of time in inventory
Assumptions
d = A known constant demand rateQ = The order quantity (arrives all at once)Planned shortages are not allowed
R Lusby (42111) Convexity & Optimization 17/38
Convex sets(jg
Definition
A convex set contains all convex combinations of its elements
α1x1 + α2x2 ∈ S ∀ x1, x2 ∈ S
Some examples of E.g. (1, 2], x2 + y2 < 4, ∅Level curve (2 dimensions):
{(x , y) : f (x , y) = β}
Level set:{x : f (x) ≤ β}
R Lusby (42111) Convexity & Optimization 18/38
Lower Level Set Example(jg
−4−2
02
4 −4−2
02
4
0
100
x y
f(x)
Plot of 2x2 + 4y2
0
50
100
150
R Lusby (42111) Convexity & Optimization 19/38
Lower Level Set Example(jg
140140
140140
130130
130130
120120
120120
110110
110110
100 100
100
100100
9090
90
9090
90
8080
80
8080
80
7070
70
7070
70
6060
60
6060
6050
50
50
50
50 50
50
40
40
40
40
40
40
30
3030
30
30
20
20
20
20
10
10
10
−4 −2 0 2 4
−4
−2
0
2
4
x
y
Plot of 2x2 + 4y2
R Lusby (42111) Convexity & Optimization 20/38
Upper Level Set Example(jg
0
2
40
2
4
0
100
200
x y
f(x)
Plot of 54x +−9x2 + 78y − 13y2
0
50
100
150
R Lusby (42111) Convexity & Optimization 21/38
Upper Level Set Example(jg
180
180
180
180
160160
160
160160
160
140140
140
140 140
120
120
120
120120
100
100100
100
80
80
80
60604020
0 1 2 3 4 50
1
2
3
4
5
x
y
Plot of 54x +−9x2 + 78y − 13y2
R Lusby (42111) Convexity & Optimization 22/38
Example(jg
−20
2 −2
0
2−1
0
1
xy
ZZ= 7xy
exp(x2+y2)
−1
−0.5
0
0.5
1
R Lusby (42111) Convexity & Optimization 23/38
Example(jg
1.2
1.2
1
1
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2 0.2
0.20.2
0.2
0.2
0
00
0−0.2
−0.2
−0.2
−0.2
−0.2
−0.2
−0.4
−0.4
−0.4
−0.4
−0.6
−0.6
−0.6
−0.6
−0.8
−0.8
−1
−1
−1.2
−1.2
−2 −1 0 1 2
−2
−1
0
1
2
x
y
Z= 7xyexp(x2+y2)
R Lusby (42111) Convexity & Optimization 24/38
Convexity, Concavity, and OptimaConstrained optimization problems(jg
Theorem
Suppose that S is convex and that f (x) is convex on S for the problemminx∈S f (x), then
If x∗ is locally minimal, then x∗ is globally minimalThe set X ∗ of global optimal solutions is convexIf f is strictly convex, then x∗ is unique
R Lusby (42111) Convexity & Optimization 25/38
Examples(jg
Problem 1
Minimize −x2lnx1 + x19 + x2
2
Subject to: 1.0 ≤ x1 ≤ 5.0
0.6 ≤ x2 ≤ 3.6
R Lusby (42111) Convexity & Optimization 26/38
What does the function look like?(jg
12
34
5 12
34
5
0
20
x1 x2
f(x)
Plot of − x2ln(x1) +x19 + x22
0
5
10
15
20
25
R Lusby (42111) Convexity & Optimization 27/38
Problem 2
Minimize∑3
i=1−ilnxi
Subject to:∑3
i=1 xi = 6
xi ≤ 3.5 i = 1, 2, 3
xi ≥ 1.5 i = 1, 2, 3
R Lusby (42111) Convexity & Optimization 28/38
Class exercises(jg
Show that f (x) = ||x|| =√∑
i x2i is convex
Prove that any level set of a convex function is a convex set
R Lusby (42111) Convexity & Optimization 29/38
Other Types of Convexity(jg
The idea of pseudoconvexity of a function is to extend the class offunctions for which stationarity is a sufficient condition for globaloptimality. If f is defined on an open set X and is differentiable wedefine the concept of pseudoconvexity.
A differentiable function f is pseudoconvex if
∇f (x) · (x′ − x) ≥ 0⇒ f (x′) ≥ f (x) ∀x , x ′ ∈ X
or alternatively ..
f (x′) < f (x)⇒ ∇f (x)(x′ − x) < 0 ∀x , x ′ ∈ X
A function f is pseudoconcave iff −f is pseudoconvex
Note that if f is convex and differentiable, and X is open, then f isalso pseudoconvex
R Lusby (42111) Convexity & Optimization 30/38
Other Types of Convexity(jg
A function is quasiconvex if all lower level sets are convex
That is, the following sets are convex
S ′ = {x : f (x) ≤ β}
A function is quasiconcave if all upper level sets are convex
That is, the following sets are convex
S ′ = {x : f (x) ≥ β}
Note that if f is convex and differentiable, and X is open, then f isalso quasiconvex
Convexity properties
Convex ⇒ pseudoconvex ⇒ quasiconvex
R Lusby (42111) Convexity & Optimization 31/38
Exercises(jg
Show the following
f (x) = x + x3 is pseudoconvex but not convexf (x) = x3 is quasiconvex but not pseudoconvex
R Lusby (42111) Convexity & Optimization 32/38
Graphically(jg
−1.5 −1 −0.5 0 0.5 1 1.5
−4
−2
0
2
4
x
f(x)
Plot of x3 + x and x3
R Lusby (42111) Convexity & Optimization 33/38
Exercises(jg
Convexity Questions
Can a function be both convex and concave?Is a convex function of a convex function convex?Is a convex combination of convex functions convex?Is the intersection of convex sets convex?
R Lusby (42111) Convexity & Optimization 34/38
Unconstrained problem(jg
min f (x) s.t. x ∈ Rn
Necessary optimality condition for xo to be a local minimum
∇f (xo) = 0 and H(xo) is positive semidefinite
Sufficient optimality condition for xo to be a local minimum
∇f (xo) = 0 and H(xo) is positive definite
Necessary and sufficientI Suppose f is pseudoconvex
I x∗ is a global minimum iff ∇f (x∗) = 0
R Lusby (42111) Convexity & Optimization 35/38
Unconstrained example(jg
min f (x) = (x2 − 1)3
f ′(x) = 6x(x2 − 1)2 = 0 for x = 0,±1
H(x) = 24x2(x2 − 1) + 6(x2 − 1)2
H(0) = 6 and H(±1) = 0
Therefore x = 0 is a local minimum (actually the global minimum)
x = ±1 are saddle points
R Lusby (42111) Convexity & Optimization 36/38
What does the function look like?(jg
−1.5 −1 −0.5 0 0.5 1 1.5
−1
0
1
2
x
f(x)
f (x)=(x2 − 1)3
R Lusby (42111) Convexity & Optimization 37/38
Class Exercise(jg
Problem
Suppose A is an m ∗ n matrix, b is a given m vector, find
min ||Ax− b||2
R Lusby (42111) Convexity & Optimization 38/38