interior-point methods - university of michiganmepelman/teaching/ioe611/... · stopping criterion....

Interior-point methods

I inequality constrained minimization

I logarithmic barrier function and central path

I barrier method

I complexity analysis via self-concordance

I generalized inequalities

I feasibility and initialization

IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods 0–1

Inequality constrained minimization

minimize f0(x)subject to f

i

(x) 0, i = 1, . . . ,mAx = b

(18)

I fi

’s convex, twice continuously di↵erentiable

I A 2 Rp⇥n with rankA = p

I we assume p? is finite and attained

I we assume problem is strictly feasible: there exists x̃ with

x̃ 2 dom f0, fi

(x̃) < 0, i = 1, . . . ,m, Ax̃ = b

hence, strong duality holds and dual optimum is attained

IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–2

Examples

I LP, QP, QCQP, GP

I entropy maximization with linear inequality constraints

minimizeP

n

i=1 xi log xi

subject to Fx � gAx = b

with dom f0 = Rn

++

I di↵erentiability may require reformulating the problem, e.g.,piecewise-linear minimization or `1-norm approximation viaLP

I SDPs and SOCPs are better handled as problems withgeneralized inequalities (see later)


Logarithmic barrierreformulation of (18) via indicator function:

minimize f0(x) +P

m

i=1 I�(fi

(x))subject to Ax = b

where I�(u) = 0 if u 0, I�(u) = 1 otherwise (indicator functionof R�)approximation via logarithmic barrier

minimize f0(x) � (1/t)P

m

i=1 log(�fi

(x))subject to Ax = b

I an equality constrained problem

I for t > 0, �(1/t) log(�u) is asmooth approximation of I�(u)

I approximation improves ast ! 1

Logarithmic barrier

reformulation of (1) via indicator function:

minimize f0(x) +!m

i=1 I−(fi(x))subject to Ax = b

where I−(u) = 0 if u ≤ 0, I−(u) = ∞ otherwise (indicator function of R−)

approximation via logarithmic barrier

minimize f0(x) − (1/t)!m

i=1 log(−fi(x))subject to Ax = b

• an equality constrained problem

• for t > 0, −(1/t) log(−u) is asmooth approximation of I−

• approximation improves as t → ∞

PSfrag replacements

u−3 −2 −1 0 1

−5

0

5

10

Interior-point methods 12–4IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–4

Logarithmic barrier function

�(x) = �mX

i=1

log(�fi

(x)), dom� = {x | f1(x) < 0, . . . , fm

(x) < 0}

I convex (follows from composition rules)

I twice continuously di↵erentiable, with derivatives

r�(x) =mX

i=1

1

�fi

(x)rf

i

(x)

r2�(x) =mX

i=1

1

fi

(x)2rf

i

(x)rfi

(x)T +mX

i=1

1

�fi

(x)r2f

i

(x)


Central path

I for t > 0, define x?(t) as the solution of

minimize tf0(x) + �(x)subject to Ax = b

(for now, assume x?(t) exists and is unique for each t > 0)

I central path is {x?(t) | t > 0}example: central path for an LP

minimize cT xsubject to aT

i

x bi

, i = 1, . . . , 6

hyperplane cT x = cT x?(t) istangent to level curve of � throughx?(t)

Central path

• for t > 0, define x⋆(t) as the solution of

minimize tf0(x) + φ(x)subject to Ax = b

(for now, assume x⋆(t) exists and is unique for each t > 0)

• central path is {x⋆(t) | t > 0}

example: central path for an LP

minimize cTxsubject to aT

i x ≤ bi, i = 1, . . . , 6

hyperplane cTx = cTx⋆(t) is tangent tolevel curve of φ through x⋆(t)

PSfrag replacements

c

x⋆ x⋆(10)


Dual points on central pathx = x?(t) if there exists a w such that

trf0(x) +mX

i=1

1

�fi

(x)rf

i

(x) + ATw = 0, Ax = b

I therefore, x?(t) minimizes the Lagrangian

L(x ,�?(t), ⌫?(t)) = f0(x) +mX

i=1

�?i

(t)fi

(x) + ⌫?(t)T (Ax � b)

where we define �?i

(t) = 1/(�tfi

(x?(t)) and ⌫?(t) = w/t

I this confirms the intuitive idea that f0(x?(t)) ! p? if t ! 1:

p? � g(�?(t), ⌫?(t))

= L(x?(t),�?(t), ⌫?(t))

= f0(x?(t)) � m/t


Interpretation via KKT conditions

x = x?(t), � = �?(t), ⌫ = ⌫?(t) satisfy

1. primal constraints: fi

(x) 0, i = 1, . . . ,m, Ax = b

2. dual constraints: � ⌫ 0

3. approximate complementary slackness: ��i

fi

(x) = 1/t,i = 1, . . . ,m

4. gradient of Lagrangian with respect to x vanishes:

rf0(x) +mX

i=1

�i

rfi

(x) + AT⌫ = 0

di↵erence with KKT is that condition 3 replaces �i

fi

(x) = 0


Force field interpretationcentering problem (for problem with no equality constraints)

minimize tf0(x) �P

m

i=1 log(�fi

(x))

force field interpretation

I Associate with each constraint a force field

Fi

(x) = �r(� log(�fi

(x))) =rf

i

(x)

fi

(x),

and with the objective function — force fieldF0(x) = �trf0(x)

I The forces balance at x?(t):

F0(x?(t)) +

mX

i=1

Fi

(x?(t)) = 0


Example

minimize cT xsubject to aT

i

x bi

, i = 1, . . . ,m

I objective force field is constant: F0(x) = �tcI constraint force field decays as inverse distance to constraint

hyperplane:

Fi

(x) =�a

i

bi

� aTi

x, kF

i

(x)k2 =1

dist(x , Hi

)

where Hi

= {x | aTi

x = bi

}

exampleminimize cTxsubject to aT

i x ≤ bi, i = 1, . . . ,m

• objective force field is constant: F0(x) = −tc

• constraint force field decays as inverse distance to constraint hyperplane:

Fi(x) =−ai

bi − aTi x

, ∥Fi(x)∥2 =1

dist(x,Hi)

where Hi = {x | aTi x = bi}

PSfrag replacements −c PSfrag replacements

−3ct = 1 t = 3

Interior-point methods 12–10


Barrier method

given strictly feasible x , t := t(0) > 0, µ > 1, tolerance ✏ > 0.

repeat

1. Centering step. Compute x?(t) by minimizing tf0 + �, subject toAx = b.

2. Update. x := x?(t).3. Stopping criterion. quit if m/t < ✏.4. Increase t. t := µt.

I terminates with f0(x) � p? ✏ (stopping criterion motivatedby f0(x?(t)) � p? m/t)

I centering usually done using Newton’s method, starting atcurrent x

I several heuristics for choice of t(0)

I choice of µ involves a trade-o↵: large µ means fewer outeriterations, more inner (Newton) iterations; typical values:µ = 10–20


Newton step for modified KKT equations

Newton step for centering problem solves

tr2f0(x) + r2�(x) AT

A 0

� �x

nt

⌫nt

�= �

trf0(x) + r�(x)

0

�

at each iteration


Examplesinequality form LP (m = 100 inequalities, n = 50 variables)

Examples

inequality form LP (m = 100 inequalities, n = 50 variables)

PSfrag replacements

Newton iterations

dual

ity

gap

µ = 2µ = 50 µ = 150

0 20 40 60 80

10−6

10−4

10−2

100

102

PSfrag replacements

µ

New

ton

iter

atio

ns

0 40 80 120 160 2000

20

40

60

80

100

120

140

• starts with x on central path (t(0) = 1, duality gap 100)

• terminates when t = 108 (gap 10−6)

• centering uses Newton’s method with backtracking

• total number of Newton iterations not very sensitive for µ ≥ 10


I starts with x (0) on central path (t(0) = 1, duality gap 100)I terminates when t = 108 (gap 10�6)I centering uses Newton’s method with backtrackingI total number of Newton iterations not very sensitive for

µ � 10IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–13

Geometric program(m = 100 inequalities and n = 50 variables)

minimize log⇣P5

k=1 exp(aT0kx + b0k)⌘

subject to log⇣P5

k=1 exp(aTik

x + bik

)⌘

0, i = 1, . . . ,m

geometric program (m = 100 inequalities and n = 50 variables)

minimize log!"5

k=1 exp(aT0kx + b0k)

#

subject to log!"5

k=1 exp(aTikx + bik)

#≤ 0, i = 1, . . . ,m

PSfrag replacements

Newton iterations

dual

ity

gap

µ = 2µ = 50µ = 150

0 20 40 60 80 100 120

10−6

10−4

10−2

100

102


Family of standard LPs

minimize cT xsubject to Ax = b, x ⌫ 0

A 2 Rm⇥2m, m = 10, . . . , 1000; for each m, solve 100 randomlygenerated instances

family of standard LPs (A ∈ Rm×2m)

minimize cTxsubject to Ax = b, x ≽ 0

m = 10, . . . , 1000; for each m, solve 100 randomly generated instances

PSfrag replacements

m

New

ton

iter

atio

ns

101 102 10315

20

25

30

35

number of iterations grows very slowly as m ranges over a 100 : 1 ratio


I Terminated when the gap is reduced by a factor of 104

I number of inner iterations grows very slowly with mI of course, each Newton step gets harder for larger m


Convergence analysis

Number of outer (centering) iterations: exactly

&log(m/(✏t(0)))

log µ

'

plus the initial centering step (to compute x?(t(0)))Centering problem

minimize tf0(x) + �(x)

see convergence analysis of Newton’s method

I tf0 + � must have closed sublevel sets for t � t(0)

I classical analysis requires strong convexity, Lipschitz condition

I analysis via self-concordance requires self-concordance oftf0 + �


Complexity analysis via self-concordance

Same assumptions as before, plus:

I sublevel sets (of f0, on the feasible set) are bounded

I tf0 + � is self-concordant with closed sublevel sets

Second condition

I holds for LP, QP, QCQP

I may require reformulating the problem, e.g.,

minimizeP

n

i=1 xi log xi

subject to Fx � g�! minimize

Pn

i=1 xi log xi

subject to Fx � g , x ⌫ 0

I needed for complexity analysis; barrier method works evenwhen self-concordance assumption does not apply


Newton iterations per centering step:Bound on e↵ort of computing x+ = x?(µt) starting at x = x?(t),from self-concordance theory

#Newton iterations µtf0(x) + �(x) � µtf0(x+) � �(x+)

�+ c

I �, c are constants (depend only on Newton algorithmparameters)

I from duality (with � = �?(t), ⌫ = ⌫?(t)):

µtf0(x) + �(x) � µtf0(x+) � �(x+)

= µtf0(x) � µtf0(x+) +

mX

i=1

log(�µt�i

fi

(x+)) � m log µ

µtf0(x) � µtf0(x+) � µt

mX

i=1

�i

fi

(x+) � m � m log µ

µtf0(x) � µtg(�, ⌫) � m � m log µ

= m(µ � 1 � log µ)IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–18

Total number of Newton iterations(excluding first centering step)

#Newton iterations N =

&log(m/(t(0)✏))

log µ

'✓m(µ � 1 � log µ)

�+ c

◆total number of Newton iterations (excluding first centering step)

#Newton iterations ≤ N =

!log(m/(t(0)ϵ))

log µ

"#m(µ − 1 − log µ)

γ+ c

$PSfrag replacements

µ

N

1 1.1 1.20

1 104

2 104

3 104

4 104

5 104

figure shows N for typical values of γ, c,

m = 100,m

t(0)ϵ= 105

• confirms trade-off in choice of µ

• in practice, #iterations is in the tens; not very sensitive for µ ≥ 10


Figure shows N for typicalvalues of �, c ,

m = 100,m

t(0)✏= 105

I confirms trade-o↵ in choice of µ

I in practice, #iterations is in the tens; not very sensitive forµ � 10

IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods –19

Polynomial-time complexity of barrier method

I for µ = 1 + 1/pm:

N = O

pm log

m/t(0)

✏

!!

I number of Newton iterations for fixed gap reduction is O(pm)

I multiply with cost of one Newton iteration (a polynomialfunction of problem dimensions), to get bound on number offlops

this choice of µ optimizes worst-case complexity; in practice wechoose µ fixed (µ = 10, . . . , 20)


Feasibility and phase I methodsFeasibility problem: find x such that

fi

(x) 0, i = 1, . . . ,m, Ax = b (19)

Phase I: computes strictly feasible starting point for barriermethodBasic phase I method

minimize (over x , s) ssubject to f

i

(x) s, i = 1, . . . ,mAx = b

(20)

I if x , s feasible, with s < 0, then x is strictly feasible for (19)I if optimal value p̄? of (20) is positive, then problem (19) is

infeasibleI if p̄? = 0 and attained, then problem (19) is feasible (but not

strictly);if p̄? = 0 and not attained, then problem (19) is infeasible


Example:Family of linear inequalities Ax � b + ��b

I data chosen to be strictly feasible for � > 0, infeasible for� 0

I use basic phase I, terminate when s < 0 or dual objective ispositive

example: family of linear inequalities Ax ≼ b + γ∆b

• data chosen to be strictly feasible for γ > 0, infeasible for γ ≤ 0

• use basic phase I, terminate when s < 0 or dual objective is positive

PSfrag replacements

γ

New

ton

iter

atio

ns

Infeasible Feasible

−1 −0.5 0 0.5 10

20

40

60

80

100

PSfrag replacements

γNewton iterations

InfeasibleFeasible

−1

−0.5

0

0.5

1

0

20

40

60

80

100

γ

New

ton

iter

atio

ns

−100 −10−2 −10−4 −10−60

20

40

60

80

100

PSfrag replacements

γNewton iterations

InfeasibleFeasible

−1

−0.5

0

0.5

1

0

20

40

60

80

100

γ

New

ton

iter

atio

ns

10−6 10−4 10−2 1000

20

40

60

80

100

number of iterations roughly proportional to log(1/|γ|)


number of iterations roughly proportional to log(1/|�|)IOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–22

Finding a well-centered initial pointLet x̄ (0) 2 \m

i=0 dom fi

and M > max{f0(x̄ (0)), p?}minimize (over x , s) ssubject to f

i

(x) s, i = 1, . . . ,mAx = bf0(x) M,

(21)

I (21) is solved by the barrier method, with points on the central pathsatisfying:

t̄ =mX

i=1

1

s̄ � fi

(x̄);

1

M � f0(x̄)rf0(x̄)+

mX

i=1

1

s̄ � fi

(x̄)rf

i

(x̄)+AT ⌫̄ = 0.

I For (x̄ , s̄) on the central path of (21) with s̄ = 0, we have:

1

M � f0(x̄)rf0(x̄) +

mX

i=1

1

�fi

(x̄)rf

i

(x̄) + AT ⌫̄ = 0,

I i.e., x̄ is on the central path of the original problem, witht = 1

M�f0(x̄)and optimality gap m/t = m(M � f0(x̄)) m(M � p?)


Generalized inequalitiesRecall: f : Rn ! Rk is convex w.r.t. a proper cone K ⇢ Rk if

f (✓x + (1 � ✓)y) �K

✓f (x) + (1 � ✓)f (y)

minimize f0(x)subject to f

i

(x) �K

i

0, i = 1, . . . ,mAx = b

I f0 convex, fi

: Rn ! Rk

i , i = 1, . . . ,m, convex with respect toproper cones K

i

⇢ Rk

i

I fi

, i = 0, 1, . . . ,m twice continuously di↵erentiableI A 2 Rp⇥n with rankA = pI we assume p? is finite and attainedI we assume problem is strictly feasible; hence strong duality

holds and dual optimum is attained

examples of greatest interest: SOCP, SDPIOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–24

Generalized logarithm for proper cone : Rq ! R is generalized logarithm for proper cone K ✓ Rq if:

I dom = intK and r2 (y) � 0 for y �K

0

I (sy) = (y) + ✓ log s for y �K

0, s > 0 (✓ is the degree of )

Examples

I nonnegative orthant K = Rn

+: (y) =P

n

i=1 log yi

, withdegree ✓ = n

I positive semidefinite cone K = Sn

+:

(Y ) = log detY (✓ = n)

I second-order coneK = {y 2 Rn+1 | (y21 + · · · + y2

n

)1/2 yn+1}:

(y) = log(y2n+1 � y21 � · · · � y2

n

) (✓ = 2)


Properties (without proof):

For y �K

0, r (y) ⌫K

⇤ 0, and yTr (y) = ✓

I nonnegative orthant Rn

+: (y) =P

n

i=1 log yi

r (y) = (1/y1, . . . , 1/yn

), yTr (y) = n

I positive semidefinite cone Sn

+: (Y ) = log detY

r (Y ) = Y�1, tr(Yr (Y )) = n

I second-order coneK = {y 2 Rn+1 | (y21 + · · · + y2

n

)1/2 yn+1}:

r (y) =2

y2n+1 � y21 � · · · � y2

n

2

6664

�y1...

�yn

yn+1

3

7775, yTr (y) = 2


Logarithmic barrier and central path

Logarithmic barrier for f1(x) �K1 0, . . . , f

m

(x) �K

m

0:

�(x) = �mX

i=1

i

(�fi

(x)), dom� = {x | fi

(x) �K

i

0, i = 1, . . . ,m}

I i

is generalized logarithm for Ki

, with degree ✓i

I � is convex, twice continuously di↵erentiable

Central path: {x?(t) | t > 0} where x?(t) solves

minimize tf0(x) + �(x)subject to Ax = b


Dual points on central pathx = x?(t) if there exists w 2 Rp,

trf0(x) +mX

i=1

Dfi

(x)Tr i

(�fi

(x)) + ATw = 0

(Dfi

(x) 2 Rk

i

⇥n is derivative, or Jacobian, matrix of fi

)I the Lagrangian (which is convex in x for �

i

�K

⇤i

0) is

L(x ,�, ⌫) = f0(x) +mX

i=1

�Ti

fi

(x) + ⌫T (Ax � b)

I therefore, x?(t) minimizes Lagrangian L(x ,�?(t), ⌫?(t)),where

�?i

(t) =1

tr

i

(�fi

(x?(t))), ⌫?(t) =w

t

I from properties of i

: �?i

(t) �K

⇤i

0, with duality gap

f0(x?(t)) � g(�?(t), ⌫?(t)) = (1/t)

mX

i=1

✓i


Example: semidefinite programming(with F

i

2 Sp)

minimize cT xsubject to F (x) =

Pn

i=1 xiFi + G � 0

I logarithmic barrier: �(x) = log det(�F (x)�1)I central path: x?(t) minimizes tcT x � log det(�F (x)); hence

tci

� tr(Fi

F (x?(t))�1) = 0, i = 1, . . . , n

I dual point on central path: Z ?(t) = �(1/t)F (x?(t))�1 isfeasible for

maximize tr(GZ )subject to tr(F

i

Z ) + ci

= 0, i = 1, . . . , nZ ⌫ 0

I duality gap on central path: cT x?(t) � tr(GZ ?(t)) = p/tIOE 611: Nonlinear Programming, Fall 2017 10. Interior-point methods Page 10–29

Barrier method

given strictly feasible x , t := t(0) > 0, µ > 1, tolerance ✏ > 0.

repeat

1. Centering step. Compute x?(t) by minimizing tf0 + �, subject toAx = b.

2. Update. x := x?(t).3. Stopping criterion. quit if (

Pi

✓i

)/t < ✏.4. Increase t. t := µt.

I only di↵erence is duality gap m/t on central path is replacedbyP

i

✓i

/t

I number of outer iterations:&

log((P

i

✓i

)/(✏t(0)))

log µ

'

I complexity analysis via self-concordance applies to SDP, SOCP


Examplessecond-order cone program (50 variables, 50 SOC constraints inR6)

Examples

second-order cone program (50 variables, 50 SOC constraints in R6)

PSfrag replacements

Newton iterations

dual

ity

gap

duality gap

µ = 2µ = 50 µ = 200

0 20 40 60 80

10−6

10−4

10−2

100

102

PSfrag replacements

µ

New

ton

iter

atio

ns

20 60 100 140 1800

40

80

120

semidefinite program (100 variables, LMI constraint in S100)

PSfrag replacements

Newton iterations

dual

ity

gap

µ = 2µ = 50µ = 150

0 20 40 60 80 100

10−6

10−4

10−2

100

102

PSfrag replacements

µ

New

ton

iter

atio

ns0 20 40 60 80 100 120

20

60

100

140



Examples

second-order cone program (50 variables, 50 SOC constraints in R6)

PSfrag replacements

Newton iterations

dual

ity

gap

duality gap

µ = 2µ = 50 µ = 200

0 20 40 60 80

10−6

10−4

10−2

100

102

PSfrag replacements

µ

New

ton

iter

atio

ns

20 60 100 140 1800

40

80

120


PSfrag replacements

Newton iterations

dual

ity

gap

µ = 2µ = 50µ = 150

0 20 40 60 80 100

10−6

10−4

10−2

100

102

PSfrag replacements

µ

New

ton

iter

atio

ns

0 20 40 60 80 100 120

20

60

100

140


family of SDPs(A 2 Sn, x 2 Rn)

minimize 1T xsubject to A + diag(x) ⌫ 0

n = 10, . . . , 1000, for each n solve 100 randomly generatedinstances

family of SDPs (A ∈ Sn, x ∈ Rn)

minimize 1Txsubject to A + diag(x) ≽ 0

n = 10, . . . , 1000, for each n solve 100 randomly generated instances

PSfrag replacements

n

New

ton

iter

atio

ns

101 102 10315

20

25

30

35



Primal-dual interior-point methods

more e�cient than barrier method when high accuracy is needed

I update primal and dual variables at each iteration; nodistinction between inner and outer iterations

I often exhibit superlinear asymptotic convergence

I search directions can be interpreted as Newton directions formodified KKT conditions

I can start at infeasible points

I cost per iteration same as barrier method


interior-point methods - university of michiganmepelman/teaching/ioe611/... · stopping criterion....

Documents