ie 5531: engineering optimization i - university of … · ie 5531: engineering optimization i ......

IE 5531: Engineering Optimization I

Lecture 7: Duality and applications

Prof. John Gunnar Carlsson

September 29, 2010

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 1 / 30

Administrivia

PS 2 posted this evening

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 2 / 30

Recap

Any linear program

minimize cTx s.t.

Ax = b

x ≥ 0

has a dual of the form

maximize bTy s.t.

ATy ≤ c

If both problems are feasible, we have cTx ≥ bTy for all feasible x and

y

At optimality, we have cTx∗ = bTy∗

�The dual of the dual is the primal�

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 3 / 30

Duality relationships

Primal minimize cTx maximize bTy Dual

A AT

≥ bi ≥ 0

constraints ≤ bi ≤ 0 variables

= bi free

≥ 0 ≤ cjvariables ≤ 0 ≥ cj constraints

free = cj

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 4 / 30

Today

Weak & strong duality

Duality examples

Complementary slackness

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 5 / 30

Weak duality

The weak duality theorem says that if x is a feasible point for the

primal (minimization) problem and y is a feasible point for the dual

(maximization) problem, then we have

bTy ≤ cTx

If optimal solutions exist for both problems, the strong duality theorem

says that

bTy∗ = cTx∗

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 6 / 30

Proof of weak duality

Let x and y be feasible primal-dual pairs for a standard-form LP, so

that

Ax = b; x ≥ 0; ATy ≤ c

We �nd that

bTy = xTATy ≤ cTx

since x ≥ 0

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 7 / 30

Proof of strong duality

Suppose that x∗ is an optimal solution to a standard-form LP that we

found via the simplex method, let B denote the basic indices at x∗,and let N denote the non-basic indices

Let x∗B = A−1B b and x∗N = 0

Recall the de�nition of the reduced cost vector r in the simplex

method (the top row):

r = c− AT(A−1B

)TcB

When the simplex method terminates, the reduced cost vector must

be nonnegative, thus

c− AT(A−1B

)TcB︸ ︷︷ ︸

y

≥ 0

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 8 / 30

Proof of strong duality

De�ne y by setting y =(A−1B

)TcB as in the preceding; clearly we have

c ≥ ATy

and thus y is dual feasible

Finally, note that

bTy = bT(A−1B

)TcB =

(A−1B b

)TcB = cTB x

∗B = cTx∗

which completes the proof

y must be optimal for the dual problem, by weak duality

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 9 / 30

Unboundedness

Note that if one of the problems is unbounded, then the other must be

infeasible

Suppose that cTx is unbounded below in a standard-form LP; weak

duality says that

bTy ≤ cTx

for all x and y

Suppose that y is feasible in the dual; we can choose x to make cTx

arbitrarily small and thus bTy ≤ −∞, a contradiction

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 10 / 30

Infeasibility

On the other hand, it is not true that if one problem is infeasible, the

other must be unbounded

The following primal-dual pairs are both infeasible:

minimize x1 + 2x2 s.t.

x1 + x2 = 1

2x1 + 2x2 = 3

with dual

maximize y1 + 3y2 s.t.

y1 + 2y2 = 1

y1 + 2y2 = 2

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 11 / 30

Feasibility and boundedness

An obvious corollary: if the primal problem is infeasible and the dual

problem has a feasible point, then it must be unbounded

A less obvious corollary: unless both problems are infeasible, at least

one of them must have an unbounded feasible set

The following relationships between primal and dual LPs are possible:

Primal

Finite optimum Unbounded Infeasible

Finite optimum Possible X X

Dual Unbounded X X Possible

Infeasible X Possible Possible

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 12 / 30

Duality example

Consider a transportation problem:

We have m warehouses, each containing an amount ai of a product

We have n stores that need to receive an amount bj of that product

It costs cij to transport one unit of the product from warehouse i to

store j

How can we minimize our costs while satisfying all stores' demands?

Assume that∑m

i=1ai =

∑nj=1

bj (total supply equals total demand)

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 13 / 30

Duality example

The LP formulation is

minimize

m∑i=1

n∑j=1

cijxij s.t.

m∑j=1

xij = ai ∀i

n∑i=1

xij = bj ∀j

xij ≥ 0 ∀i , j

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 14 / 30

Duality example

The dual of the transportation problem is

maximize

m∑i=1

aiui +n∑

j=1

bjvj s.t.

ui + vj ≤ cij ∀i , j

Remember:

ai is the amount of the product available at warehouse i

bj is the amount of the product required at store j

cij is the transportation cost from i to j

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 15 / 30

Duality example

Since there is no sign constraint, we can set ui 7→ −ui (forinterpretation purposes):

maximize

n∑j=1

bjvj −m∑i=1

aiui s.t.

vj − ui ≤ cij ∀i , j

Interpretation: think of the ui 's and vj 's as prices

Speci�cally, think of a new logistics company that wants to ship these

products

The company will buy the goods at each warehouse at a price ui and

sell them back to the warehouses at a price vjIn order to be competitive with the original company's transportation

modes, the new company must have vj − ui ≤ cij ∀i , j (otherwise the

original transportation company won't use them)

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 16 / 30

Duality example

Since there is no sign constraint, we can set ui 7→ −ui (forinterpretation purposes):

maximize

n∑j=1

bjvj −m∑i=1

aiui s.t.

vj − ui ≤ cij ∀i , j

Interpretation: think of the ui 's and vj 's as prices

Speci�cally, think of a new logistics company that wants to ship these

products

The company will buy the goods at each warehouse at a price ui and

sell them back to the warehouses at a price vjIn order to be competitive with the original company's transportation

modes, the new company must have vj − ui ≤ cij ∀i , j (otherwise the

original transportation company won't use them)

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 16 / 30

Maximum �ow

Recall from earlier the maximum �ow problem:

Given a directed, weighted graph G = (V ,E ,W ) and a pair of nodes

s and t

Think of the edge weights wij as the �capacity� of that edge

What's the largest amount of ��ow� we can send from s to t, subject

to the capacity constraints?

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 17 / 30

Maximum �ow

In lecture 1 we derived an LP formulation: let xij denote the amount of

�ow across edge (i , j); the problem is

maximize

∑(s,i)∈E

xsi s.t.

xij ≤ wij ∀ (i , j) ∈ E∑(i ,j)∈E

xij =∑

(j ,i)∈E

xji ∀i 6= s, t

xij ≥ 0 ∀ (i , j) ∈ E

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 18 / 30

Maximum �ow

The dual of the maximum �ow problem is

minimize

∑(i ,j)∈E

wijzij s.t.

zij = yi − yj ∀ (i , j) ∈ E

ys − yt = 1

zij ≥ 0 ∀ (i , j) ∈ E

(see B&T 7.5 for derivation)

We're assigning values yi to the nodes, and incurring costs yi − yj for

all (i , j) ∈ E

We can assume WLOG that ys = 1 and yt = 0

It is not hard to show that, in fact, yi ∈ {0, 1} for all i

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 19 / 30

Maximum �ow

The dual of the maximum �ow problem is

minimize

∑(i ,j)∈E

wijzij s.t.

zij = yi − yj ∀ (i , j) ∈ E

ys − yt = 1

zij ≥ 0 ∀ (i , j) ∈ E

(see B&T 7.5 for derivation)

We're assigning values yi to the nodes, and incurring costs yi − yj for

all (i , j) ∈ E

We can assume WLOG that ys = 1 and yt = 0

It is not hard to show that, in fact, yi ∈ {0, 1} for all i

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 19 / 30

Min cut

The preceding problem is equivalent to �nding a subset S of vertices,

such that s ∈ S , that minimizes the cut length of S , i.e.∑i∈S :j /∈S

wij

Thus, strong duality says that the maximum �ow we can push through

the network is equal to the smallest cut size of any subset S of vertices

Corollary: if the value of a �ow is equal to the value of some cut, then

both are optimal

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 20 / 30

Max-�ow min-cut theorem

Input graph

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 21 / 30

Max-�ow min-cut theorem

Maximum �ow

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 22 / 30

Max-�ow min-cut theorem

Minimum cut

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 23 / 30

Complementary slackness

Consider a standard-form linear program and let Ai denote column i of

A

We know that any constraint in the primal corresponds to a variable in

the dual, and vice versa

An important relationship between optimal primal and dual solutions is

known as complementary slackness, or complementarity:

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 24 / 30

Complementary slackness

Theorem

Let x and y be feasible solutions to the primal and dual problem,

respectively. The vectors x and y are optimal solutions for the two

respective problems if and only if

xi > 0 =⇒ ATi y = ci

ATi y < ci =⇒ xi = 0

in other words,

xi

(ATi y − ci

)= 0 ∀i

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 25 / 30

Proof of complementary slackness

Suppose that

xi > 0 =⇒ ATi y = ci

ATi y < ci =⇒ xi = 0

as in the theorem; then certainly(ATy − c

)Tx =

(yTA− cT

)x

=n∑

i=1

(yTAi − ci

)xi

= 0

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 26 / 30

Proof of complementary slackness

Since(yTA− cT

)x = 0, we know that

0 =(yTA− cT

)x

= yTAx− cTx

= yTb− cTx

cTx = bTy

which completes one direction of the proof by weak duality.

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 27 / 30

Proof of complementary slackness

To prove the other direction, suppose that x and y are optimal, so

that cTx = bTy (strong duality)

We can write

cTx− bTy = 0

cTx− (Ax)T y = 0

cTx− yTAx = 0(cT − yTA

)x = 0

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 28 / 30

Proof of complementary slackness

We can rewrite the equation(cT − yTA

)x = 0

as a summation:n∑

i=1

(ci − yTAi

)xi = 0

Since x and y are feasible, we have

ci − yTAi ≥ 0 ∀ixi ≥ 0 ∀i

so(ci − yTAi

)xi for all i

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 29 / 30

Non-standard form LPs

Complementarity holds for LPs that are not in standard form as well

Let aTj denote the jth row of A and let Ai denote the ith column of A

A primal-dual pair x, y is optimal if and only if

yj

(aTj x− bj

)= 0 ∀j ∈ {1, . . . ,m}(

ci − yTAi

)xi = 0 ∀i ∈ {1, . . . , n}

where A ∈ Rm×n

In other words: take each variable, multiply it by its associated

constraint; the product must be 0

Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 30 / 30