department of management engineering technical university

Solution Methods

Richard Lusby

Department of Management EngineeringTechnical University of Denmark

Lecture Overview(jg

Unconstrained

Several Variables

Quadratic Programming

Separable Programming

SUMT

R Lusby (42111) Solution Methods 2/37

One Variable Unconstrained Optimization(jg

Let’s consider the simplest case: Unconstrained optimization with just asingle variable x , where the differentiable function to be maximized isconcave

The necessary and sufficient condition for a particular solution x = x∗

to be a global maximum is:

df

dx= 0 x = x∗

If this can be solved directly you are done

What if it cannot be solved easily analytically?

We can utilize search procedures to solve it numerically

Find a sequence of trial solutions that lead towards the optimalsolution


Bisection MethodOne Variable Unconstrained Optimization(jg

Can always be applied when f (x) concave

It can also be used for certain other functions

If x∗ denotes the optimal solution, all that is needed is that

df

dx> 0 if x < x∗

df

dx= 0 if x = x∗

df

dx< 0 if x > x∗

These conditions automatically hold when f (x) is concave

The sign of the gradient indicates the direction of improvement


Example(jg

f (x)

x

dfdx = 0

x∗


Bisection MethodOverview(jg

Bisection

Given two values, x < x , with f ′(x) > 0, f ′(x) < 0Find the midpoint x̂ = x+x

2Find the sign of the slope of the midpointThe next two values are:

I x = x̂ if f ′(x̂) < 0I x = x̂ if f ′(x̂) > 0

What is the stopping criterion?

x − x < 2ε


Bisection MethodExample(jg

The Problem

maximize f (x) = 12x − 3x4 − 2x6


Bisection MethodWhat does the function look like?(jg

0.5 1.0 1.5

1

2

3

4

5

6

7

8

9

−1

−2


Bisection MethodCalculations(jg

Iteration f ′(x̂) x x x̂ f (x̂)

0 0 2 1 7.0000

1 -12.00 0 1 0.5 5.7812

2 10.12 0.5 1 0.75 7.6948

3 4.09 0.75 1 0.875 7.8439

4 -2.19 0.75 0.875 0.8125 7.8672

5 1.31 0.8125 0.875 0.84375 7.8829

6 -0.34 0.8125 0.84375 0.828125 7.8815

7 0.51 0.828125 0.84375 0.8359375 7.8839

x∗ ≈ 0.836

0.828125 < x∗ < 0.84375

f (x∗) = 7.8839


Bisection MethodContinued(jg

Intuitive and straightforward procedure

Converges slowly

An iteration decreases the difference between the bounds by one half

Only information on the derivative of f (x) is used

More information could be obtained by looking at f ′′(x)


Newton MethodIntroduction(jg

Basic Idea: Approximate f (x) within the neighbourhood of thecurrent trial solution by a quadratic function

This approximation is obtained by truncating the Taylor series afterthe second derivative

f (xi+1) ≈ f (xi ) + f ′(xi )(xi+1 − xi ) +f ′′(xi )

2(xi+1 − xi )

2

Having set xi at iteration i , this is just a quadratic function of xi+1

Can be maximized by setting its derivative to zero


Newton Overview

max f (xi+1) ≈ f (xi ) + f ′(xi )(xi+1 − xi ) +f ′′(xi )

2(xi+1 − xi )

2

f ′(xi+1) ≈ f ′(xi ) + f ′′(xi )(xi+1 − xi )

xi+1 = xi −f ′(xi )

f ′′(xi )

What is the stopping criterion?

|xi+1 − xi | < ε


Same Example(jg

The Problem

maximize: f (x) = 12x − 3x4 − 2x6

f ′(x) = 12− 12x3 − 12x5

f ′′(x) = −36x3 − 60x4

xi+1 = xi +1− x3 − x5

3x3 + 5x4


Newton MethodThe function again(jg

0.5 1.0 1.5

1

2

3

4

5

6

7

8

9

−1

−2


Newton MethodCalculations(jg

Iteration xi f (xi ) f ′(xi ) f ′′(xi ) f (x̂)

1 1 7 -12 -96 0.875

2 0.875 7.8439 -2.1940 -62.733 0.84003

3 0.84003 7.8838 -0.1325 -55.279 0.83763

4 0.83763 7.8839 -0.0006 -54.790 0.83762

x∗ = 083763

f (x∗) = 7.8839


Several variablesNewton still works(jg

Newton: Multivariable

Given x1, the next iterate maximizes the quadratic approximation

f (x1) +∇f (x1)(x− x1) + (x− x1)TH(x1)(x− x1)

2

x2 = x1 − H(x1)−1∇f (x1)T

Gradient search

The next iterate maximizes f along the gradient ray

maximize: g(t) = f (x1 + t∇f (x1)T ) s.t. t ≥ 0

x2 = x1 + t∗∇f (x1)T


Example(jg

The Problem

maximize: f (x , y) = 2xy + 2y − x2 − 2y2


ExampleContinued(jg

The vector of partial derivatives is given as

∇f (x) =

(∂f

∂x1,∂f

∂x2, . . . ,

∂f

∂xn

)

Here∂f

∂x= 2y − 2x

∂f

∂y= 2x + 2− 4y

Suppose we select the point (x,y)=(0,0) as our initial point

∇f (0, 0) = (0, 2)


ExampleContinued(jg

Perform an iterationx = 0 + t(0) = 0

y = 0 + t(2) = 2t

Substituting these expressions in f (x) we get

f (x + t ∗ ∇f (x)) = f (0, 2t) = 4t − 8t2

Differentiate wrt to t

d

dt(4t − 8t2) = 4− 16t = 0

Therefore t∗ = 14 , and x = (0, 0) + 1

4(0, 2) =(0, 12)


ExamplePerform a second iteration(jg

Gradient at x = (0, 12) is ∇f (0, 12) = (1, 0)

Determine step length

x =

(0,

1

2

)+ t(1, 0)

Substituting these expressions in f (x) we get

f (x + t ∗ ∇f (x)) = f

(t,

1

2

)= t − t2 +

1

2

Differentiate wrt to t

d

dt(t − t2 +

1

2) = 1− 2t = 0

Therefore t∗ = 12 , and x =

(0, 12)

+ 12(1, 0) =

(12 ,

12

)


1

1

y

x

(0, 1

2

) (12 ,

12

)

(12 ,

34

) (34 ,

34

)

(34 ,

78

) (78 ,

78

)


Quadratic programming(jg

maximize: cTx− 12x

TQx

subject to: Ax ≤ b (λ)

x ≥ 0 (µ)

Q is symmetric and positive semidefinite

Lagrangian:

L(x,λ,µ) = cTx− 1

2xTQx + λT (b− Ax) + µTx

Applying KKT yields

Qx + ATλ− µ = c

Ax + v = b

x,λ,µ, v ≥ 0

xTµ = 0,λTv = 0 complementarity constraints


Example(jg

Problem

minimize: f (x)

subject to: g1(x) ≤ 0

g2(x) ≤ 0

KKT Conditions

∇f (x) + u1∇g1(x) + u2∇g2(x) = 0

u1g1(x) = 0

u1 ≥ 0

u2g2(x) = 0

u2 ≥ 0


QP solution(jg

KKT Conditions

Qx + ATλ− µ = c

Ax + v = b

x,λ,µ, v ≥ 0

xTµ = 0

λTv = 0

Add artificial variables to constraints with positive cj

Subtract artificial variables to constraints with negative bi

Initial basic variables are artificial variables some of µ, and some of v

Do phase 1 of the Simplex method with a restricted entry rule,I Ensures complementarity constraints are always satisfied


Example(jg

The Problem

maximize: 15x + 30y + 4xy − 2x2 − 4y2

subject to: x + 2y ≤ 30

x , y ≥ 0


ExampleContinued(jg

The Parameters are as follows:

c =

[15

30

],A =

[1

2

], x =

[x

y

],Q =

[4 −4

−4 8

],b = 30

The Non-Linear Component of the objective function is

−1

2xTQx = −2x2 + 4xy − 4y2


ExampleNew Linear Program(jg

Minimize: Z = z1 + z2

subject to: 4x − 4y + λ1 − µ1 + z1 = 15

−4x + 8y + 2λ1 − µ2 + z2 = 30

x + 2y + v1 = 30

x , y , λ1, µ1, µ2, v1, z1, z2 ≥ 0

Complementarity Conditions

xµ1 = 0, yµ2 = 0, v1λ1 = 0


Modified SimplexInitial Tableau(jg

BV Z x y λ1 µ1 µ2 v1 z1 z2 rhs

Z -1 0 -4 -3 1 1 0 0 0 -45

z1 0 4 -4 1 -1 0 0 1 0 15

z2 0 -4 8 2 0 -1 0 0 1 30

v1 0 1 2 0 0 0 1 0 0 30


Modified SimplexFirst Pivot(jg


Z -1 -2 0 -2 1 12 0 0 1

2 -30

z1 0 2 0 2 -1 -12 0 1 12 30

y 0 -12 1 14 0 -18 0 0 1

8308

v1 0 2 0 -12 0 14 1 0 -14

452


Modified SimplexSecond Pivot(jg


Z -1 0 0 -52 1 34 1 0 1

4 -152z1 0 0 0 5

2 -1 -34 -1 1 34

152

y 0 0 1 18 0 - 1

1614 0 1

16758

x 0 1 0 -14 0 18

12 0 -18

454


Modified SimplexFinal Tableau(jg


Z -1 0 0 0 0 0 0 1 1 0

λ1 0 0 0 1 -25 - 310 -25

25

310 3

y 0 0 1 0 120 - 1

40310 - 1

20140 9

x 0 1 0 0 - 110

120

25

110 - 1

20 12


Separable Programming(jg

The Problem

maximize:∑

j fj(xj)

subject to: Ax ≤ b

x ≥ 0

Each fj is approximated by a piece-wise linear function

f (y) = s1y1 + s2y2 + s3y3

y = y1 + y2 + y3

0 ≤ y1 ≤ u1

0 ≤ y2 ≤ u2

0 ≤ y3 ≤ u3


Separable ProgrammingContinued(jg

Special restrictions:I y2 = 0 whenever y1 < u1I y3 = 0 whenever y2 < u2

If each fj is concave,I Automatically satisfied by the simplex method

Why?


Sequential Unconstrained Minimization Technique(jg

The Problem

maximize: f (x)

subject to: g(x) ≤ b

x ≥ 0

For a sequence of decreasing positive r ’s, solve

maximize f (x)− rB(x)

B is a barrier function approaching ∞ as a feasible point approachesthe boundary of the feasible region

For example

B(x) =∑

i

1

bi − gi (x)+∑

j

1

xj


The Problem

maximize: xy

subject to: x2 + y ≤ 3

x , y ≥ 0

r x y

1 1

1 0.90 1.36

10−2 0.987 1.925

10−4 0.998 1.993

Class exercise

Verify that the KKT conditions are satisfied at x = 1 & y = 2


Class exercises(jg

Separable programming

maximize: 32x − x4 + 4y − y2

subject to: x2 + y2 ≤ 9

x , y ≥ 0

Formulate this as an LP model using x = 0, 1, 2, 3 and y = 0, 1, 2, 3as breakpoints for the approximating piece-wise linear functions


Class exercisesContinued(jg

Sequential Unconstrained Minimization Technique

If f is concave and gi is convex ∀ i , show that the following is concave

f (x)− r

(∑

i

1

bi − gi (x)+∑

j

1

xj


department of management engineering technical university

Documents