ie 5531: engineering optimization i - university of … · ie 5531: engineering optimization i ......
TRANSCRIPT
IE 5531: Engineering Optimization I
Lecture 7: Duality and applications
Prof. John Gunnar Carlsson
September 29, 2010
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 1 / 30
Administrivia
PS 2 posted this evening
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 2 / 30
Recap
Any linear program
minimize cTx s.t.
Ax = b
x ≥ 0
has a dual of the form
maximize bTy s.t.
ATy ≤ c
If both problems are feasible, we have cTx ≥ bTy for all feasible x and
y
At optimality, we have cTx∗ = bTy∗
�The dual of the dual is the primal�
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 3 / 30
Duality relationships
Primal minimize cTx maximize bTy Dual
A AT
≥ bi ≥ 0
constraints ≤ bi ≤ 0 variables
= bi free
≥ 0 ≤ cjvariables ≤ 0 ≥ cj constraints
free = cj
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 4 / 30
Today
Weak & strong duality
Duality examples
Complementary slackness
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 5 / 30
Weak duality
The weak duality theorem says that if x is a feasible point for the
primal (minimization) problem and y is a feasible point for the dual
(maximization) problem, then we have
bTy ≤ cTx
If optimal solutions exist for both problems, the strong duality theorem
says that
bTy∗ = cTx∗
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 6 / 30
Proof of weak duality
Let x and y be feasible primal-dual pairs for a standard-form LP, so
that
Ax = b; x ≥ 0; ATy ≤ c
We �nd that
bTy = xTATy ≤ cTx
since x ≥ 0
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 7 / 30
Proof of strong duality
Suppose that x∗ is an optimal solution to a standard-form LP that we
found via the simplex method, let B denote the basic indices at x∗,and let N denote the non-basic indices
Let x∗B = A−1B b and x∗N = 0
Recall the de�nition of the reduced cost vector r in the simplex
method (the top row):
r = c− AT(A−1B
)TcB
When the simplex method terminates, the reduced cost vector must
be nonnegative, thus
c− AT(A−1B
)TcB︸ ︷︷ ︸
y
≥ 0
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 8 / 30
Proof of strong duality
De�ne y by setting y =(A−1B
)TcB as in the preceding; clearly we have
c ≥ ATy
and thus y is dual feasible
Finally, note that
bTy = bT(A−1B
)TcB =
(A−1B b
)TcB = cTB x
∗B = cTx∗
which completes the proof
y must be optimal for the dual problem, by weak duality
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 9 / 30
Unboundedness
Note that if one of the problems is unbounded, then the other must be
infeasible
Suppose that cTx is unbounded below in a standard-form LP; weak
duality says that
bTy ≤ cTx
for all x and y
Suppose that y is feasible in the dual; we can choose x to make cTx
arbitrarily small and thus bTy ≤ −∞, a contradiction
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 10 / 30
Infeasibility
On the other hand, it is not true that if one problem is infeasible, the
other must be unbounded
The following primal-dual pairs are both infeasible:
minimize x1 + 2x2 s.t.
x1 + x2 = 1
2x1 + 2x2 = 3
with dual
maximize y1 + 3y2 s.t.
y1 + 2y2 = 1
y1 + 2y2 = 2
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 11 / 30
Feasibility and boundedness
An obvious corollary: if the primal problem is infeasible and the dual
problem has a feasible point, then it must be unbounded
A less obvious corollary: unless both problems are infeasible, at least
one of them must have an unbounded feasible set
The following relationships between primal and dual LPs are possible:
Primal
Finite optimum Unbounded Infeasible
Finite optimum Possible X X
Dual Unbounded X X Possible
Infeasible X Possible Possible
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 12 / 30
Duality example
Consider a transportation problem:
We have m warehouses, each containing an amount ai of a product
We have n stores that need to receive an amount bj of that product
It costs cij to transport one unit of the product from warehouse i to
store j
How can we minimize our costs while satisfying all stores' demands?
Assume that∑m
i=1ai =
∑nj=1
bj (total supply equals total demand)
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 13 / 30
Duality example
The LP formulation is
minimize
m∑i=1
n∑j=1
cijxij s.t.
m∑j=1
xij = ai ∀i
n∑i=1
xij = bj ∀j
xij ≥ 0 ∀i , j
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 14 / 30
Duality example
The dual of the transportation problem is
maximize
m∑i=1
aiui +n∑
j=1
bjvj s.t.
ui + vj ≤ cij ∀i , j
Remember:
ai is the amount of the product available at warehouse i
bj is the amount of the product required at store j
cij is the transportation cost from i to j
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 15 / 30
Duality example
Since there is no sign constraint, we can set ui 7→ −ui (forinterpretation purposes):
maximize
n∑j=1
bjvj −m∑i=1
aiui s.t.
vj − ui ≤ cij ∀i , j
Interpretation: think of the ui 's and vj 's as prices
Speci�cally, think of a new logistics company that wants to ship these
products
The company will buy the goods at each warehouse at a price ui and
sell them back to the warehouses at a price vjIn order to be competitive with the original company's transportation
modes, the new company must have vj − ui ≤ cij ∀i , j (otherwise the
original transportation company won't use them)
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 16 / 30
Duality example
Since there is no sign constraint, we can set ui 7→ −ui (forinterpretation purposes):
maximize
n∑j=1
bjvj −m∑i=1
aiui s.t.
vj − ui ≤ cij ∀i , j
Interpretation: think of the ui 's and vj 's as prices
Speci�cally, think of a new logistics company that wants to ship these
products
The company will buy the goods at each warehouse at a price ui and
sell them back to the warehouses at a price vjIn order to be competitive with the original company's transportation
modes, the new company must have vj − ui ≤ cij ∀i , j (otherwise the
original transportation company won't use them)
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 16 / 30
Maximum �ow
Recall from earlier the maximum �ow problem:
Given a directed, weighted graph G = (V ,E ,W ) and a pair of nodes
s and t
Think of the edge weights wij as the �capacity� of that edge
What's the largest amount of ��ow� we can send from s to t, subject
to the capacity constraints?
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 17 / 30
Maximum �ow
In lecture 1 we derived an LP formulation: let xij denote the amount of
�ow across edge (i , j); the problem is
maximize
∑(s,i)∈E
xsi s.t.
xij ≤ wij ∀ (i , j) ∈ E∑(i ,j)∈E
xij =∑
(j ,i)∈E
xji ∀i 6= s, t
xij ≥ 0 ∀ (i , j) ∈ E
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 18 / 30
Maximum �ow
The dual of the maximum �ow problem is
minimize
∑(i ,j)∈E
wijzij s.t.
zij = yi − yj ∀ (i , j) ∈ E
ys − yt = 1
zij ≥ 0 ∀ (i , j) ∈ E
(see B&T 7.5 for derivation)
We're assigning values yi to the nodes, and incurring costs yi − yj for
all (i , j) ∈ E
We can assume WLOG that ys = 1 and yt = 0
It is not hard to show that, in fact, yi ∈ {0, 1} for all i
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 19 / 30
Maximum �ow
The dual of the maximum �ow problem is
minimize
∑(i ,j)∈E
wijzij s.t.
zij = yi − yj ∀ (i , j) ∈ E
ys − yt = 1
zij ≥ 0 ∀ (i , j) ∈ E
(see B&T 7.5 for derivation)
We're assigning values yi to the nodes, and incurring costs yi − yj for
all (i , j) ∈ E
We can assume WLOG that ys = 1 and yt = 0
It is not hard to show that, in fact, yi ∈ {0, 1} for all i
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 19 / 30
Min cut
The preceding problem is equivalent to �nding a subset S of vertices,
such that s ∈ S , that minimizes the cut length of S , i.e.∑i∈S :j /∈S
wij
Thus, strong duality says that the maximum �ow we can push through
the network is equal to the smallest cut size of any subset S of vertices
Corollary: if the value of a �ow is equal to the value of some cut, then
both are optimal
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 20 / 30
Max-�ow min-cut theorem
Input graph
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 21 / 30
Max-�ow min-cut theorem
Maximum �ow
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 22 / 30
Max-�ow min-cut theorem
Minimum cut
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 23 / 30
Complementary slackness
Consider a standard-form linear program and let Ai denote column i of
A
We know that any constraint in the primal corresponds to a variable in
the dual, and vice versa
An important relationship between optimal primal and dual solutions is
known as complementary slackness, or complementarity:
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 24 / 30
Complementary slackness
Theorem
Let x and y be feasible solutions to the primal and dual problem,
respectively. The vectors x and y are optimal solutions for the two
respective problems if and only if
xi > 0 =⇒ ATi y = ci
ATi y < ci =⇒ xi = 0
in other words,
xi
(ATi y − ci
)= 0 ∀i
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 25 / 30
Proof of complementary slackness
Suppose that
xi > 0 =⇒ ATi y = ci
ATi y < ci =⇒ xi = 0
as in the theorem; then certainly(ATy − c
)Tx =
(yTA− cT
)x
=n∑
i=1
(yTAi − ci
)xi
= 0
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 26 / 30
Proof of complementary slackness
Since(yTA− cT
)x = 0, we know that
0 =(yTA− cT
)x
= yTAx− cTx
= yTb− cTx
cTx = bTy
which completes one direction of the proof by weak duality.
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 27 / 30
Proof of complementary slackness
To prove the other direction, suppose that x and y are optimal, so
that cTx = bTy (strong duality)
We can write
cTx− bTy = 0
cTx− (Ax)T y = 0
cTx− yTAx = 0(cT − yTA
)x = 0
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 28 / 30
Proof of complementary slackness
We can rewrite the equation(cT − yTA
)x = 0
as a summation:n∑
i=1
(ci − yTAi
)xi = 0
Since x and y are feasible, we have
ci − yTAi ≥ 0 ∀ixi ≥ 0 ∀i
so(ci − yTAi
)xi for all i
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 29 / 30
Non-standard form LPs
Complementarity holds for LPs that are not in standard form as well
Let aTj denote the jth row of A and let Ai denote the ith column of A
A primal-dual pair x, y is optimal if and only if
yj
(aTj x− bj
)= 0 ∀j ∈ {1, . . . ,m}(
ci − yTAi
)xi = 0 ∀i ∈ {1, . . . , n}
where A ∈ Rm×n
In other words: take each variable, multiply it by its associated
constraint; the product must be 0
Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 30 / 30