EE8950TomLuo
Lecture6:Duality
•Lagrangedualfunction
•Lagrangedualproblem
•strongdualityandSlater’scondition
•KKToptimalityconditions
•sensitivityanalysis
•equalityconstraints
•generalizedinequalities
•theoremsofalternatives
1
EE8950TomLuo
Lagrangian
standardformproblem(withoutequalityconstraints)
minimizef0(x)
subjecttofi(x)≤0,i=1,...,m
•optimalvaluep?,domainD
•calledprimalproblem(incontextofduality)
(fornow)wedon’tassumeconvexity
LagrangianL:Rn+m
→R
L(x,λ)=f0(x)+λ1f1(x)+···+λmfm(x)
•λicalledLagrangemultipliersordualvariables
•objectiveisaugmentedwithweightedsumofconstraintfunctions
2
EE8950TomLuo
Lagrangedualfunction
(Lagrange)dualfunctiong:Rm
→R∪{−∞}
g(λ)=infx
L(x,λ)=infx
(f0(x)+λ1f1(x)+···+λmfm(x))
•minimumofaugmentedcostasfunctionofweights
•canbe−∞forsomeλ
•gisconcave(eveniffinotconvex!)
example:LPminimizec
Tx
subjecttoaTix−bi≤0,i=1,...,m
NotethatL(x,λ)=cTx+
m∑
i=1
λi(aTix−bi)=−b
Tλ+(A
Tλ+c)
Tx
henceg(λ)=
{
−bTλifA
Tλ+c=0
−∞otherwise
3
EE8950TomLuo
Lowerboundproperty
ifλ�0andxisprimalfeasible,then
g(λ)≤f0(x)
proof:iffi(x)≤0andλi≥0,
f0(x)≥f0(x)+∑
i
λifi(x)≥infz
(
f0(z)+∑
i
λifi(z)
)
=g(λ)
f0(x)−g(λ)iscalledthedualitygapof(primalfeasible)xandλ�0
minimizeoverprimalfeasiblextoget,foranyλ�0,
g(λ)≤p?
λ∈Rm
isdualfeasibleifλ�0andg(λ)>−∞
dualfeasiblepointsyieldlowerboundsonoptimalvalue!
4
EE8950TomLuo
Lagrangedualproblem
let’sfindbestlowerboundonp?:
maximizeg(λ)
subjecttoλ�0
•called(Lagrange)dualproblem
(associatedwithprimalproblem)
•alwaysaconvexproblem,evenifprimalisn’t!
•optimalvaluedenotedd?
•wealwayshaved?≤p
?(calledweakduality)
•p?−d
?isoptimaldualitygap
5
EE8950TomLuo
Strongduality
forconvexproblems,we(usually)havestrongduality:
d?=p
?
whenstrongdualityholds,dualoptimalλ?
servesascertificateofoptimalityforprimal
optimalpointx?
manyconditionsorconstraintqualificationsguaranteestrongdualityforconvexproblems
Slater’scondition:ifprimalproblemisstrictlyfeasible(andconvex),i.e.,thereexists
x∈relintDwith
fi(x)<0,i=1,...,m
thenwehavep?=d
?
6
EE8950TomLuo
Dualoflinearprogram
(primal)LPminimizec
Tx
subjecttoAx�b
•nvariables,minequalityconstraints
dualofLPis(aftermakingimplicitequalityconstraintsexplicit)
maximize−bTλ
subjecttoATλ+c=0
λ�0
•dualofLPisalsoanLP(indeed,instdLPformat)
•mvariables,nequalityconstraints,mnonnegativitycontraints
forLPwehavestrongdualityexceptinone(pathological)case:primalanddualboth
infeasible(p?=+∞,d
?=−∞)
7
EE8950TomLuo
Dualofquadraticprogram
(primal)QPminimizex
TPx
subjecttoAx�b
weassumeP�0forsimplicityLagrangianisL(x,λ)=xTPx+λ
T(Ax−b)
∇xL(x,λ)=0yieldsx=−(1/2)P−1
ATλ,hencedualfunctionis
g(λ)=−(1/4)λTAP
−1A
Tλ−b
Tλ
•concavequadraticfunction
•allλ�0aredualfeasible
dualofQPismaximize−(1/4)λ
TAP
−1A
Tλ−b
Tλ
subjecttoλ�0
...anotherQP
8
EE8950TomLuo
Dualityinalgorithms
manyalgorithmsproduceatiterationk
•aprimalfeasiblex(k)
•andadualfeasibleλ(k)
withf0(x(k)
)−g(λ(k)
)→0ask→∞
henceatiterationkweknowp?∈[
g(λ(k)
),f0(x(k)
)]
•usefulforstoppingcriteria
•algorithmsthatusedualsolutionareoftenmoreefficient(e.g.,LP)
9
EE8950TomLuo
Nonheuristicstoppingcriteria
absoluteerror=f0(x(k)
)−p?≤ε
stoppingcriterion:until(
f0(x(k)
)−g(λ(k)
)≤ε)
relativeerror=f0(x
(k))−p
?
|p?|≤ε
stoppingcriterion:
until
(
g(λ(k)
)>0&f0(x(k))−g(λ(k))
g(λ(k))≤ε
)
or
(
f0(x(k)
)<0&f0(x(k))−g(λ(k))
−f0(x(k))≤ε
)
achievetargetvalue`or,prove`isunachievable
(i.e.,determineeitherp?≤`orp
?>`)
stoppingcriterion:until(
f0(x(k)
)≤`org(λ(k)
)>`)
10
EE8950TomLuo
Complementaryslackness
supposex?,λ
?areprimal,dualfeasiblewithzerodualitygap(hence,theyareprimal,dual
optimal)
f0(x?)=g(λ
?)=inf
x
(
f0(x)+
m∑
i=1
λ?ifi(x)
)
≤f0(x?)+
m∑
i=1
λ?ifi(x
?)
hencewehave∑
mi=1λ
?ifi(x
?)=0,andso
λ?ifi(x
?)=0,i=1,...,m
•calledcomplementaryslacknesscondition
•ithconstraintinactiveatoptimum=⇒λi=0
•λ?i>0atoptimum=⇒ithconstraintactiveatoptimum
11
EE8950TomLuo
KKToptimalityconditions
suppose
•fiaredifferentiable
•x?,λ
?are(primal,dual)optimal,withzerodualitygap
bycomplementaryslacknesswehave
f0(x?)+
∑
i
λ?ifi(x
?)=inf
x
(
f0(x)+∑
i
λ?ifi(x)
)
i.e.,x?
minimizesL(x,λ?)
therefore
∇f0(x?)+
∑
i
λ?i∇fi(x
?)=0
12
EE8950TomLuo
soifx?,λ
?are(primal,dual)optimal,withzerodualitygap,theysatisfy
fi(x?)≤0
λ?i≥0
λ?ifi(x
?)=0
∇f0(x?)+
∑
iλ?i∇fi(x
?)=0
theKarush-Kuhn-Tucker(KKT)optimalityconditions
conversely,iftheproblemisconvexandx?,λ
?satisfyKKT,thentheyare(primal,dual)
optimal
13
EE8950TomLuo
Geometricinterpretationofduality
considerset
A={(u,t)∈Rm+1
|∃xfi(x)≤ui,f0(x)≤t}
•Aisconvexiffiare
•forλ�0,
g(λ)=inf
{
[
λ
1
]
T[
u
t
]
∣
∣
∣
∣
∣
[
u
t
]
∈A
}
PSfragreplacements
u
t
A
t+λT
u=g(λ)
g(λ)[
λ1
]
14
EE8950TomLuo
(Ideaof)proofofSlater’stheoremproblemconvex,strictlyfeasible=⇒strongduality
PSfragreplacements
u
t
A
[
1
λ?
]
p?
•(0,p?)∈∂A⇒∃supportinghyperplaneat(0,p
?):
(u,t)∈A=⇒µ0(t−p?)+µ
Tu≥0
•µ0≥0,µ�0,(µ,µ0)6=0
•strongduality⇔∃supportinghyperplanewithµ0>0:forλ?=µ/µ0,wehave
p?≤t+λ
?Tu∀(t,u)∈A,p
?≤g(λ
?)
•Slater’scondition:thereexists(u,t)∈Awithu≺0;impliesthatallsupporting
hyperplanesat(0,p?)arenon-vertical(µ0>0)
15
EE8950TomLuo
Sensitivityanalysisviaduality
definep?(u)astheoptimalvalueof
minimizef0(x),subjecttofi(x)≤ui,i=1,...,m
0
0
PSfragreplacements
u
p?(u
)
epip?
p?(0)−λ?Tu
λ?
giveslowerboundonp?(u):p
?(u)≥p
?−∑
mi=1λ
?iui
•ifλ?ilarge:ui<0greatlyincreasesp
?
•ifλ?ismall:ui>0doesnotdecreasep
?toomuch
ifp?(u)isdifferentiable,λ
?i=−
∂p?(0)
∂ui
,λ?iissensitivityofp
?w.r.t.ithconstraint
16
EE8950TomLuo
Equalityconstraints
minimizef0(x)
subjecttofi(x)≤0,i=1,...,m
hi(x)=0,i=1,...,p
•optimalvaluep?
•againassume(fornow)notnecessarilyconvex
defineLagrangianL:Rn+m+p
→Ras
L(x,λ,ν)=f0(x)+
m∑
i=1
λifi(x)+
p∑
i=1
νihi(x)
dualfunctionisg(λ,ν)=infxL(x,λ,ν)
(λ,ν)isdualfeasibleifλ�0andg(λ,ν)>−∞
(nosignconditiononν)
17
EE8950TomLuo
lowerboundproperty:ifxisprimalfeasibleand(λ,ν)isdualfeasible,theng(λ,ν)≤
f0(x),hence
g(λ,ν)≤p?
dualproblem:findbestlowerbound
maximizeg(λ,ν)
subjecttoλ�0
(noteνunconstrained),optimalvalued?
weakduality:d?≤p
?always
strongduality:ifprimalisconvexthen(usually)d?=p
?
Slatercondition:ifprimalisconvex(i.e.,ficonvex,hiaffine)andstrictlyfeasible,i.e.,
thereexistsx∈relintDs.t.
fi(x)<0,hi(x)=0,
thend?=p
?
18
EE8950TomLuo
Example:equalityconstrainedleast-squares
minimizexTx
subjecttoAx=b
Aisfat,fullrank(solutionisx?=A
T(AA
T)−1
b)
dualfunctionis
g(ν)=infx
(
xTx+ν
T(Ax−b)
)
=−1
4ν
TAA
Tν−b
Tν
dualproblemis
maximize−14ν
TAA
Tν−b
Tν
solution:ν?=−2(AA
T)−1
b
cancheckd?=p
?
19
EE8950TomLuo
KKToptimalityconditions
assumefi,hidifferentiable
ifx?,λ
?,ν
?areoptimal,withzerodualitygap,thentheysatisfyKKTconditions
fi(x?)≤0,hi(x
?)=0
λ?i≥0
λ?ifi(x
?)=0
∇f0(x?)+
∑
iλ?i∇fi(x
?)+
∑
iν?i∇hi(x
?)=0
conversely,iftheysatisfyKKTandtheproblemisconvex,thenx?,λ
?,ν
?areoptimal
example:optimalityconditionsforequalityconstraintsonly
minimizef0(x)
subjecttoAx=b
x?
optimal⇐⇒if∃ν?
s.t.∇f0(x?)+A
Tν
?=0
20
EE8950TomLuo
Introducingequalityconstraints
idea:simpletransformationofprimalproblemcanleadtoverydifferentdual
example:unconstrainedgeometricprogramming
primalproblem:
minimizelog
m∑
i=1
exp(aTix−bi)
dualfunctionisconstantg=p?
(wehavestrongduality,butit’suseless)
nowrewriteprimalproblemas
minimizelog
m∑
i=1
expyi
subjecttoy=Ax−b
21
EE8950TomLuo
letusintroduce
•mnewvariablesy1,...,ym
•mnewequalityconstraintsy=Ax−b
dualfunction
g(ν)=infx,y
(
log
m∑
i=1
expyi+νT(Ax−b−y)
)
•infimumis−∞ifATν6=0
•assumingATν=0,let’sminimizeovery:
eyi
∑
nj=1e
yj=νi
solvableiffνi>0,1Tν=1
g(ν)=−∑
i
νilogνi−bTν
22
EE8950TomLuo
•sameexpressionifν�0,1Tν=1(0log0=0)
dualproblem
maximize−bTν−
∑
i
νilogνi
subjectto1Tν=1,(ν�0)
ATν=0
moral:trivialreformulationcanyielddifferentdual
23
EE8950TomLuo
Generalizedinequalities
minimizef0(x)
subjecttofi(x)�Ki0,i=1,...,L
•�KiaregeneralizedinequalitiesonRmi
•fi:Rn→R
miareKi-convex
LagrangianL:Rn×R
m1×···×RmL→R,
L(x,λ1,...,λL)=f0(x)+λT1f1(x)+···+λ
TLfL(x)
dualfunction
g(λ1,...,λL)=infx
(
f0(x)+λT1f1(x)+···+λ
TLfL(x)
)
λidualfeasibleifλi�K?i
0,g(λ1,...,λL)>−∞
24
EE8950TomLuo
lowerboundproperty:ifxprimalfeasibleand
(λ1,...,λL)isdualfeasible,then
g(λ1,...,λL)≤f0(x)
(hence,g(λ1,...,λL)≤p?)
dualproblemmaximizeg(λ1,...,λL)
subjecttoλi�K?i
0,i=1,...,L
weakduality:d?≤p
?always
strongduality:d?=p
?usually
Slatercondition:ifprimalisstrictlyfeasible,i.e.,
∃x∈relintD:fi(x)≺Ki0,i=1,...,L
thend?=p
?
25
EE8950TomLuo
Example:semidefiniteprogramming
minimizecTx
subjecttoF0+x1F1+···+xnFn�0
Lagrangian(multiplierZ=ZT∈R
m×m)
L(x,Z)=cTx+TrZ(F0+x1F1+···+xnFn)
dualfunction
g(Z)=infx
(
cTx+TrZ(F0+x1F1+···+xnFn)
)
=
{
TrF0ZifTrFiZ+ci=0,i=1,...,n
−∞otherwise
dualproblemmaximizeTrF0Z
subjecttoTrFiZ+ci=0,i=1,...,n
Z=ZT�0
strongdualityholdsifthereexistsxwithF0+x1F1+···+xnFn≺0
26
EE8950TomLuo
Theoremofalternatives
f1,...,fmconvexwithdomfi=Rn
exactlyoneofthefollowingistrue:
1.thereexistsxwithfi(x)<0,i=1,...,m
2.thereexistsλ6=0withλ�0,
g(λ)=infx
(λ1f1(x)+···+λmfm(x))≥0
•calledalternatives
•useinpractice:λthatsatisfies2ndconditionprovesfi(x)<0isinfeasible
example:fi(x)=aTix−bi
1.thereexistsxwithAx≺b
2.thereexistsλ�0,λ6=0,bTλ≤0,A
Tλ=0
27
EE8950TomLuo
proof.fromconvexduality:
primalproblemminimizet
subjecttofi(x)≤t,i=1,...,m
(variablesx,t)
dualproblemmaximizeg(λ)
subjecttoλ�0
1Tλ=1
•Slater’sconditionissatisfied,hencep?=d
?
•1stalternative:⇐⇒p?
<0
•2ndalternative:⇐⇒p?≥0
28