CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNINGConstrained optimization

KEY CONCEPTS Constraint formulations Necessary optimality conditions

Lagrange multipliers: equality constraints KKT conditions: equalities and inequalities

Objective function fFigure 1

S

+ feasible set S

Objective function fFigure 1

S

Local minima are either local minima of f or on the boundary of S

Figure 2

S

l1 u1

l2

u2

Bound constraints Linear inequalities Ax b

S Ai

(row i)

bi

Linear equalities Ax = b

b1

A1

S

General, nonlinear constraints

Figure 3

h1(x)0

g1(x)=0

h2(x)0

S

Lagrange multipliers: one equality constraintAt a local minimum (or maximum), the gradient of the objectiveand the constraint must be parallel

Figure 4

g(x)=0x1

f(x1)

g(x1)

g(x2)

f(x2)

x2

If the constraint gradient and the objective gradient are notparallel, then there exists some direction v that you can move into change f without changing g(x)

Figure 5

f(x)

x

g(x)

v

Interpretation: Suppose x* is a global minimum.I were to relax the constraint g(x)=0 at a constant ratetoward g(x)=1, the value of tells me the rate of decrease off(x*).

Figure 6

x*

f(x*) = - g(x*)

g(x*)

One inequality constraint h(x) 0. Either:Figure 7

h(x) < 0

h(x) > 0

h(x) = 0

One inequality constraint h(x) 0. Either:1. x is a critical point of f with h(x) < 0, or

Figure 7

h(x) < 0

h(x) > 0

h(x) = 0

f(x1)=0

Figure 7

h(x) < 0

h(x) > 0

h(x) = 0

x2

f(x2)

g(x2)

One inequality constraint h(x) 0. Either:1. x is a critical point of f with h(x) < 0, or2. x is on boundary h(x) = 0 and satisfies a Lagrangian condition

g(x3)

f(x3)

x3

Figure 8

h1(x) < 0

Multiple inequality constraints

h2(x) < 0

x

x