A. Nedich and U. Shanbhag Lecture 2
Outline
• Vectors, Matrices, Norms, Convergence
• Open and Closed Sets
• Special Sets: Subspace, Affine Set, Cone, Convex Set
• Special Convex Sets: Hyperplane, Polyhedral Set, Ball
• Special Cones: Normal Cone, Dual Cone, Tangent Cone
• Some Operations with Sets:
• Set Closure, Interior, Boundary
• Convex Hull, Conical Hull, Affine Hull
• Relative Closure, Interior, Boundary
• Functions
• Special Functions: Linear, Quadratic
Game Theory: Models, Algorithms and Applications 1
A. Nedich and U. Shanbhag Lecture 2
Notation: Vectors and Matrices
We consider the space of n-dimensional real vectors denoted by Rn
• Vector x ∈ Rn is viewed as a column: x =
x1...
xn
• The prime denotes a transpose: x′ = [x1, . . . , xn]
• For a matrix A, we use aij or [A]ij to denote its (i, j)-th entry
• The matrix A is positive semidefinite (A ≥ 0) when
x′Ax ≥ 0 for all x ∈ Rn
• The matrix A is positive definite (A > 0) when
x′Ax > 0 for all x ∈ Rn, x 6= 0
Game Theory: Models, Algorithms and Applications 2
A. Nedich and U. Shanbhag Lecture 2
Inner Product and Norms
• For vectors x ∈ Rn and y ∈ Rn, the inner product is
x′y =n∑
i=1
xiyi
• Norm (length) of a vector:
• Euclidean norm ‖x‖ =√
x′x =√∑n
i=1 x2i (l2-norm)
• Sum-norm ‖x‖1 =∑n
i=1 |xi| (l1-norm)
• Max-norm ‖x‖∞ = max1≤i≤n |xi| (l∞-norm)
• Matrix norm induced by (Euclidean) vector norm
‖A‖ = supx 6=0
‖Ax‖‖x‖
= sup‖x‖=1
‖Ax‖‖x‖
Game Theory: Models, Algorithms and Applications 3
A. Nedich and U. Shanbhag Lecture 2
Norm Properties, Distance, Orthogonal Vectors
• Properties of a norm
• Nonnegativity: ‖x‖ ≥ 0 for every x ∈ Rn
• Uniqueness of the zero-value: ‖x‖ = 0 if and only if x = 0
• Homogeneity: ‖λx‖ = |λ| ‖x‖ for any scalar λ and any x ∈ Rn
• Triangle relation: ‖x + y‖ ≤ ‖x‖+ ‖y‖ for every x and y ∈ Rn
• (Euclidean) Distance between two vectors x ∈ Rn and y ∈ Rn
‖x− y‖ =
√√√√ n∑i=1
(xi − yi)2
• Vectors x and y are orthogonal when x′y = 0
• For orthogonal vectors x and y:
Pythagorean Theorem ‖x + y‖2 = ‖x‖2 + ‖y‖2
• For any vectors x and y:
Schwartz Inequality: |x′y| ≤ ‖x‖ ‖y‖
Game Theory: Models, Algorithms and Applications 4
A. Nedich and U. Shanbhag Lecture 2
Vector Sequence
Given a vector sequence {xk} ⊂ Rn, we say that
• The sequence is bounded when for some scalar C, we have
‖xk‖ ≤ C for all k
• The sequence converges to a vector x ∈ Rn when
limk→∞ ‖xk − x‖ = 0 (xk → x)
• Vector y is an accumulation point of the sequence when there is a
subsequence {xki} such that xki → y as i →∞
• Example: xk = (1 + (−1)ka)k with 0 < a < 1. Accum. points?
• Bolzano Theorem
A bounded sequence {xk} ⊂ Rn has at least one limit point
Game Theory: Models, Algorithms and Applications 5
A. Nedich and U. Shanbhag Lecture 2
Set TopologyTypically denoted by capital letters C, X, Y , Z, etc.
Let X be a set in Rn.
• A vector x is an accumulation (or a limit) point of the set X when
there is a sequence {xk} ⊆ X such that xk → x
• The set X is closed when it contains all of its accumulation points
• The set X is open when its complement set Xc = {x ∈ Rn | x 6∈ X}is closed
• Rn and ∅ are the only sets that are both open and closed
• The set X is bounded when for some C: ‖x‖ ≤ C for all x ∈ X
• (Th.) The set X is compact when it is both closed and bounded
Let {Xi | i ∈ I} be a family of closed sets Xi ⊆ Rn
• The intersection ∩i∈IXi is closed
• When the set I is finite (I = {1, . . . , m}) the union ∪i∈IXi is closed
Game Theory: Models, Algorithms and Applications 6
A. Nedich and U. Shanbhag Lecture 2
Special SetsLet X be a set in Rn. We say that X is:
• subspace when αx + βy ∈ X for every x, y ∈ X and any α, β ∈ R• Example: {x | x′e = 0} where e ∈ Rn with ei = 1 for all i
• Is a subspace a closed set? Bounded?
• affine when it is a translated subspace:
X = x + S for an x ∈ X and a subspace S
• cone when λx ∈ X for every λ ≥ 0 and every x ∈ X
• convex when αx + (1− α)y ∈ X for any x, y ∈ X and α ∈ [0,1]
Game Theory: Models, Algorithms and Applications 7
A. Nedich and U. Shanbhag Lecture 2
Convex Sets
Is a subspace convex set? Affine set?
Special convex sets:
• (Euclidean) Ball centered at x0 ∈ Rn with a radius r > 0
{x ∈ Rn | ‖x− x0‖ ≤ r} (closed ball)
{x ∈ Rn | ‖x− x0‖ < r} (open ball)
• A closed ball is convex; An open ball is convex
Game Theory: Models, Algorithms and Applications 8
A. Nedich and U. Shanbhag Lecture 2
More Convex Sets• Half-space with normal vector a 6= 0: {x | a′x ≤ b}• Hyperplane with normal vector a 6= 0: {x | a′x = b} (b is a scalar)
• Polyhedral set (given by finitely many linear inequalities):
{x | a′ix ≤ bi, i = 1, . . . , m} = {x | Ax ≤ b}
A is an m× n matrix and b ∈ Rm
Game Theory: Models, Algorithms and Applications 9
A. Nedich and U. Shanbhag Lecture 2
Dual ConeLet K ⊆ Rn be a cone.
Def. A dual cone of K is the following set
K∗ = {d ∈ Rn | d′y ≥ 0 for all y ∈ K}
• Dual cone appears in formulations of complementarity problems
• Dual cone is always closed and convex (regardless of K)
Game Theory: Models, Algorithms and Applications 10
A. Nedich and U. Shanbhag Lecture 2
Normal Cone of a SetLet X ⊆ Rn be a nonempty set, and let x ∈ X.
Def. A normal cone of the set X at the point x is the following set
N(x;X) = {y ∈ Rn | y′(x− x) ≤ 0 for all x ∈ X}Vectors in this set are called normal vectors to the set X at x
X
N(x ; X )
x
^
x
N(x ; X ) = {0}
~
~
• Normal cone plays an important role in optimality conditions
• Normal cone is always closed and convex (regardless of X)
Game Theory: Models, Algorithms and Applications 11
A. Nedich and U. Shanbhag Lecture 2
Tangent Cone of a Set
Let X ⊆ Rn be a nonempty set, and let x ∈ X.
Def. A tangent cone of the set X at the point x is the following set
T (x;X) ={y ∈ Rn | y = lim
k→∞
xk − x
λk
with {xk} ⊂ X, {λk} ⊂ R++
s. t. xk → x, λk → 0}
Vectors in this set are called tangent vectors to the set X at x
Prop. The tangent cone T (x;X) is equivalently given by
T (x;X) =
{βy ∈ Rn | y = lim
k→∞
xk − x
‖xk − x‖with {xk} ⊂ X,
xk → x, xk 6= x, β ≥ 0}
Game Theory: Models, Algorithms and Applications 12
A. Nedich and U. Shanbhag Lecture 2
Tangent Cone Illustration
X
x
x~
T(x ; X )^
Tangent cone T (x;X) for any set X and any x ∈ X:
• plays an important role in optimality conditions
• always closed regardless of the properties of X
• when X is convex, then T (x;X) is convex
Game Theory: Models, Algorithms and Applications 13
A. Nedich and U. Shanbhag Lecture 2
Tangent and Normal Cone
Let X ⊂ Rn and let x ∈ X. Then, we have −N(x;X) ⊆ T (x;X)∗
X
x
x~
N(x ; X )^
T(x ; X )*^
Prop. If X is convex, then
T (x;X)∗ = −N(x;X) for all x ∈ X
Game Theory: Models, Algorithms and Applications 14
A. Nedich and U. Shanbhag Lecture 2
Set Interior, Closure and Boundary
Let X ⊆ Rn.
• A vector x ∈ X is an interior point of X if there exist a ball B(x, r)
such that B(x, r) ⊂ X
• The interior of the set X is the set of all interior points of X; it is
denoted by int(X)
• A vector x ∈ X is a accumulation point of X if there exist a sequence
{xk} ⊂ X converging to x
• The closure of the set X is the set of all accumulation points of X; it
is denoted by cl(X)
• The boundary of the set X is the intersection of the closure set of X
and the closure set of its complement (Rn \X), i.e.,
bd(X) = cl(X) ∩ cl(Rn \X)
Example: X = {(x1, x2) ∈ R2 | 0 ≤ x1 ≤ 1, 0 < x2 < 1}
Game Theory: Models, Algorithms and Applications 15
A. Nedich and U. Shanbhag Lecture 2
Convex and Conical HullsA convex combination of vectors x1, . . . , xm is a vector of the form
α1x1 + . . . + αmxm αi ≥ 0 for all i and∑m
i=1 αi = 1
The convex hull of a set X is the set of all convex combinations of the
vectors in X, denoted conv(X)
A conical combination of vectors x1, . . . , xm is a vector of the form
λ1x1 + . . . + λmxm with λi ≥ 0 for all i
The conical hull of a set X is the set of all conical combinations of the
vectors in X, denoted by cone(X)
Game Theory: Models, Algorithms and Applications 16
A. Nedich and U. Shanbhag Lecture 2
Affine Hull
An affine combination of vectors x1, . . . , xm is a vector of the form
t1x1 + . . . + tmxm with∑m
i=1 ti = 1, ti ∈ R for all i
The affine hull of a set X is the set of all affine combinations of the vectors
in X, denoted aff(X)
The dimension of a set X is the dimension of the affine hull of X
dim(X) = dim(aff(X))
Game Theory: Models, Algorithms and Applications 17
A. Nedich and U. Shanbhag Lecture 2
Relative Interior, Closure, and Boundary
Let X ⊆ Rn.
• A vector x ∈ X is a relative interior point of X if there exist a ball
B(x, r) such that B(x, r) ∩ aff(X) ⊂ X
• The relative interior of the set X is the set of all interior points of X;
it is denoted by rint(X)
• The relative boundary of the set X is the intersection of the affine
hull aff(X) and the boundary bd(X)
Example: X = {(x1, x2) ∈ R2 | 0 ≤ x1 ≤ 1, x2 = 1}
aff(X) =?
Game Theory: Models, Algorithms and Applications 18
A. Nedich and U. Shanbhag Lecture 2
Mappings
Let X ⊆ Rn.
• We write F : X → Rm to denote a mapping F from the set X to Rm
• F is a function when F : X → R (m = 1)
• We refer to X as a domain of the mapping F
• The image of X under mapping F is the following set
F (X) = {y ∈ Rm | y = F (x) for some x ∈ X}
• The inverse image of Y ⊆ Rm under mapping F is the following set
F−1(Y ) = {x ∈ X | F (x) ∈ Y }
Game Theory: Models, Algorithms and Applications 19
A. Nedich and U. Shanbhag Lecture 2
Special Mappings and Functions
A mapping F : X → Rm with X ⊆ Rn is
• affine when for all x ∈ X,
F (x) = Ax + b for some matrix A and a vector b ∈ Rm
Example: Space translation (by a given x0 ∈ Rn) is an affine mapping
from Rn to Rn: F (x) = x + x0 for all x ∈ Rn
• linear when for all x ∈ X, we have F (x) = Ax for a matrix A
Example: For a coordinate subspace S = {x ∈ Rn | xi = 0, for i ∈ I},where I is a subset of coordinate indices (I ⊆ {1, . . . , n}, the projection
on S is a linear mapping from Rn to Rn:
F (x) = Px with [P ]ii = 1 for i 6∈ I and Pij = 0 otherwise
Game Theory: Models, Algorithms and Applications 20
A. Nedich and U. Shanbhag Lecture 2
Special Functions
A function f : Rn → R is
• quadratic when for all x
f(x) = x′Qx + a′x + b for a matrix Q, a ∈ Rn and b ∈ R
• affine when for all x
f(x) = a′x + b for a vector a ∈ Rn and b ∈ R
• linear when for all x
f(x) = a′x for a vector a ∈ Rn
Game Theory: Models, Algorithms and Applications 21
A. Nedich and U. Shanbhag Lecture 2
References for this lecture
The material for this lecture:
• (B) Bertsekas D.P. Nonlinear Programming
• See Appendix there for topology in Rn and linear algebra
• (BNO) Bertsekas, Nedic, Ozdaglar Convex Analysis and Optimization
• Chapter 1 for convex sets and functions
• see also lecture slides for Convex Optimization (Lectures 2–5) at
https://netfiles.uiuc.edu/angelia/www/angelia.html
• (FP) Facchinei and Pang Finite Dimensional ..., Vol I
• Chapter 1 for Normal Cone, Dual Cone, and Tangent Cone
• NOTE!!!
• Normal cone definition used in (FP) is different than that of (B,BNO)
Game Theory: Models, Algorithms and Applications 22