new complexity results for the iri-imai methodzhangs/reports/1996_sz-iri-imai.pdfnew complexity...

New complexity results for the Iri-Imai method

Jos F. Sturm Shuzhong Zhang

June 1993

Abstract

In this paper we show that the number of main iterations requiredby the Iri-Imai algorithm to solve a linear programming problem isO(nL). Moreover, we show that a modification of this algorithm re-quires only O(

√nL) main iterations. In this modification we measure

progress by means of the primal-dual potential function.

Key words: Linear Programming, Iri-Imai method, primal-dualpotential function.

1 Introduction

Since Karmarkar [12] showed in 1984 that his interior point methodsolves linear programming problems in polynomial time, a lot of re-search has been devoted to interior point methods. This researchcan be categorized according to the type of interior point methodanalyzed. In particular, we can distinguish affine scaling methods,path-following methods and potential reduction methods. The affinescaling method was introduced as early as in 1967 by Dikin [2]. Thatmethod searches along a scaled steepest descent direction of the objec-tive function and therefore is of a greedy nature. Currently, no pureaffine scaling algorithm has been proved to be polynomial. In fact, theconvergence of the large-step affine scaling method (without any non-degeneracy assumptions) has been proved only recently by Tsuchiyaand Muramutsu [15]. The path-following method was first studied by

1

J.F. Sturm and S. Zhang: New complexity results for the Iri-Imai method 2

Huard [8] in 1967. The method is less greedy in terms of improvingthe objective since it only searches for approximate analytical centersof a sequence of shrinking subsets of the feasible region that containthe optimal point. The objective function, however, is monotonical-ly improving along this sequence of analytical centers. Polynomialityof a path-following algorithm has first been proved by Renegar [13]in 1988. Karmarkar’s method is called a potential reduction methodbecause it minimizes a (quasi-convex) potential function which is acomposition of objective as well as constraint functions. Potential re-duction methods are not of greedy type with respect to the objectivefunction, since reducing the potential function locally can even resultin a worse objective value. In the long run, however, the objective willbe improved. The potential function φ is defined by

φ(x, z) := q ln(z − bTx)−n∑i=1

ln(ci − aTi x),

where q ≥ n, and z is an upper bound on the optimal value of (D):max{bTx : ATx ≤ c}.

With respect to the potential function, an important observationwas made by Iri and Imai [10], when they showed that the exponentialof Karmarkar’s potential function, the Iri-Imai function, is strictlyconvex in x if q ≥ n+1. The method of Iri and Imai is to minimize theIri-Imai function by means of Newton’s method. A polynomial boundon the number of main iterations required by this method was firstproved by Zhang and Shi [17] in 1988. The iteration bound obtainedby Zhang and Shi was O(n8L) and was reduced to O(n2L) by Imai [9]in 1991. Iri [11] obtained an O(nL) iteration bound for the case inwhich l is a multiple of n, where l := q − n. In 1992, Zhang [18]extended Iri’s result to the smooth convex programming case.

For the potential reduction approach Ye [16] made an importan-t contribution by showing that his primal-dual potential reductionmethod requires only O(

√nL) main iterations to solve linear pro-

gramming problems.In this paper we show that the Iri-Imai method actually requires

O(qL) iterations for any q ≥ n + 2. Moreover, we show that whenl = q − n =

√n+ 3, and a certain updating scheme of upper-bounds

on the optimal objective value is used, the number of main iterationsrequired reduces to O(

√nL) too.


This paper is organized as follows. At the end of this section, wepresent a glossary of symbols used in this paper for easy reference. InSection 2 we try to get the reader acquainted with the ideas behindpotential reduction methods in general and the Iri-Imai method inparticular. In Section 3, the behavior of relevant quantities along theNewton direction of the Iri-Imai function is analyzed. In addition,the Iri-Imai algorithm is described and the O(nL) iteration bound isproven for it. In Section 4, we present our primal-dual version of Iriand Imai’s method, and prove the number of main iterations neededby the algorithm thus obtained to be O(

√nL). We conclude the paper

in Section 5.

List of symbols:A, b, c, x, λ data and variables for (P) and (D)F ,F0 {x | ATx ≤ c}, {x | ATx < c}Φ, φ, ψ Iri-Imai, dual, and primal-dual potential functionsq, z, l parameters of the potential functions Φ,φ and ψd −∇2Φ(x)−1∇Φ(x)ai the i-th column of A, A = [a1, a2, · · · , an]

aiaTi d

ci−aTi xa

∑ni=1 ai/n

b bTdz−bTx

h√−∇φ(x, z)Td

e [1, 1, · · · , 1]T

ei the i-th column of the identity matrixIn identity matrix of order n, In = [e1, e2, · · · , en]rk qbk −

∑ni=1 a

ki

s√∑n

i=1(ai − b)2

z∗ optimal objective value, z∗ = maxx∈F bTx

2 Preliminaries

Consider the linear programming problem in the dual form

(D) maximize bTxsubject to ATx ≤ c


where A is an m×n matrix, b and c are m- and n-dimensional vectorsrespectively, and x ∈ Rm is the decision variable.

Let F be the feasible region, F := {x | ATx ≤ c} with interiorF0 := {x | ATx < c} and let z be an upper bound on the optimalvalue of (D), i.e. bTx ≤ z for all x ∈ F . We will make the followingassumptions.

Assumption 1 (Slater condition)

F0 6= ∅.

Assumption 2 F is bounded.

Further assume that an initial interior point x0 ∈ F0 is available.A potential reduction method tries to construct the next iterative

interior solution x′ based on an interior point x ∈ F0, in such a waythat a certain reduction can be guaranteed in the value of the potentialfunction φ, which is defined by

φ(x, z) := q ln(z − bTx)−n∑i=1

ln(ci − aTi x)

where q ≥ n, z is an upper bound on the optimal value of (D). Notethat the function −

∑ni=1 ln(ci − aT

i x) is Huard’s center function [8].The minimizer of this center function, which can be shown to be u-nique under Assumptions 1 and 2, is called the analytic center ofF [13, 14]. The concept of potential function was introduced by Kar-markar [12] in 1984 for linear programming in the primal form. As φis only defined for x ∈ F0, φ is also called a barrier function. By re-ducing the potential function φ by at least a constant amount at eachiteration, the gap z − bTx will decrease in the long run also, becausethe center function is bounded from below by its value in the analyticcenter of F .

Lemma 1 Suppose Assumptions 1 and 2 are satisfied, and an initialinterior point x0, and an initial upper bound z0 have been generated insuch a way that φ(x0, z0) = O(qL). If an algorithm reduces the poten-tial function φ(x, z) by at least δ at each iteration, then the algorithmcan be used to find an optimal solution of (D) in O(qL/δ) iterations.


Proof:See Gonzaga [5], Lemma 2.1.

2

This paper does not deal with the question how to generate (x0,z0);we refer the interested reader to Anstreicher [1] and Freund [4]. Tosimplify the notation, we will often use φ(x) to denote φ(x, z) in thisand the next section when there is no confusion.

Lemma 1 shows that by minimizing the potential function, onewould solve (D). Up till now, it is by no means clear, however, howto minimize φ(x). Especially it does not make much sense to applyNewton’s method, as φ(x) happens to be non-convex. Instead of min-imizing φ itself, we will minimize in this paper the Iri-Imai potentialfunction Φ, which is a monotone transformation of φ. The Iri-Imaipotential function is

Φ(x) := eφ(x) =(z − bTx)q∏ni=1(ci − aT

i x)

where q = n+ l.In the remaining of this section, we will prove that Φ is strictly

convex on F0 if l > 1.We first note, since Φ > 0 on F0 and φ(x) = ln(Φ(x)), that

∇φ(x) =∇Φ(x)

Φ(x)

∇2φ(x) = −∇Φ(x)∇Φ(x)T

Φ(x)2+∇2Φ(x)

Φ(x)

i.e.∇2Φ(x)

Φ(x)= ∇φ(x)∇φ(x)T +∇2φ(x). (1)

Because φ(x) = q ln(z − bTx)−∑ni=1 ln(ci − aT

i x) we have

∇φ(x) = −q b

z − bTx+

n∑i=1

aici − aT

i x

and

∇2φ(x) =n∑i=1

aiaTi

(ci − aTi x)2

− q bbT

(z − bTx)2.


It is well known that n∑ni=1 y

2i ≥ (

∑ni=1 yi)

2 for any y ∈ Rn. Wenow prove a somewhat stronger statement in the following lemma toproceed the analysis.

Lemma 2 For any α ≥ 0, y ∈ Rn it holds that

(n+ α)n∑i=1

y2i ≥

(n∑i=1

yi

)2

+α(n+ α)

α+ 1max

1≤i≤ny2i .

Proof:For α ≥ 0 we will prove the lemma by showing

(n+ α)In − eeT − βeneTn � 0⇔ β ≤ α(n+ α)

α+ 1(2)

where A � B means that B −A is a positive semi-definite matrix.Note that a matrix is positive semi-definite iff all of its eigenvalues

are nonnegative. Clearly, the matrix Aj := (n + α)Ij − eeT is semi-positive definite for any j ≤ n and α ≥ 0 since it has j − 1 positiveeigenvalues n + α and one nonnegative eigenvalue n + α − j. As thedeterminant is the product of the eigenvalues, we have

det(Aj) = (n+ α− j)(n+ α)j−1.

Therefore, the semi-positive definiteness of An − βeneTn is determined

solely by the sign of det(An − βeneTn ).

Finally, notice that the difference between An and An − βeneTn

exists only in one position and the last column of An − βeneTn can be

written as e − βen. This shows that det(An − βeneTn ) = det(An) −

β det(An−1), and so det(An − βeneTn ) ≥ 0 iff

β ≤ det(An)

det(An−1)=α(n+ α)

α+ 1

which proves equation (2). By pre-multiplying by yT and post-multiplyingby y in (2) the lemma follows.2

Now we are ready to give an alternative proof for a result whichwas obtained by Iri and Imai [10]. The new proof technique will beused in the analysis given in the next section.


Lemma 3 The Hessian matrix ∇2Φ(x) is positive definite for anyx ∈ F0 if l > 1.

Proof:(See also Iri and Imai [10]).Let v ∈ Rm and x ∈ F0 be arbitrary. Then we have, using (1),

vT∇2Φ(x)

Φ(x)v = (q2 − q) (bTv)2

(z − bTx)2

−2qbTv

z − bTx

n∑i=1

aTi v

ci − aTi x

+

(n∑i=1

aTi v

ci − aTi x

)2

+n∑i=1

(aTi v)2

(ci − aTi x)2

.

Therefore, we obtain for any j ∈ {1, 2, · · · , n},

vT∇2Φ(x)

Φ(x)v =

n∑i=1

(aTi v)2

(ci − aTi x)2

− 1

q − 1

(n∑i=1

aTi v

ci − aTi x

)2

+

(√q2 − q bTv

z − bTx−√

q

q − 1

n∑i=1

aTi v

ci − aTi x

)2

≥n∑i=1

(aTi v)2

(ci − aTi x)2

− 1

q − 1(n∑i=1

aTi v

ci − aTi x

)2

≥ l − 1

l

(aTj v)2

(cj − aTj x)2

≥ 0 (3)

where the last inequality follows using Lemma 2. Since v is arbitrary,this shows that∇2Φ(x) is positive semi-definite. To prove it is actuallypositive definite, we assume to the contrary that there exists a v 6= 0

such that vT∇2Φ(x)Φ(x) v = 0. Then we have for any t ∈ R, and for any

j ∈ {1, 2, · · · , n},

(taTj v)2

(cj − aTj x)2

≤ lt2

l − 1vT∇2Φ(x)

Φ(x)v = 0.

which contradicts the boundedness of F (Assumption 2).2


Knowing that the Hessian matrix ∇2Φ is positive definite on F0,it clearly makes sense to analyze the Newton direction d given by

d := −∇2Φ(x)−1∇Φ(x).

This analysis will be carried out in the next section.

3 Along the Newton direction

To simplify the analysis, we introduce the quantity

b :=bTd

z − bTx,

which is the change in objective value caused by a unit Newton step drelative to the gap z−bTx. In addition, we define aj to be the relativechange in the slack of constraint j,

aj :=aTj d

cj − aTj x

for j = 1, 2, · · · , n

and finally h2 is defined as a first order approximation of the reductionin potential function value along d,

h2 := −∇φ(x)Td = qb−n∑i=1

ai.

Due to equation (1) it follows that

h2 = h4 − qb2 +n∑i=1

a2i .

The following lemma estimates the change in slacks along d. Fortechnical reasons we assume from now on that l ≥ 2.

Lemma 4 (Change in slacks) The quantity h2 can be bounded fromabove by

h2 ≤ l

l − 1and the relative change in slack values along d can be estimated as

max1≤j≤n

a2j ≤

l + 3

l(h2 − 2l − 3

2lh4) ≤ l + 3

2(2l − 3).


Proof:Let 1

l ≤ λ ≤ 1.Because h4 = (qb−

∑ni=1 ai)

2 we have

dT∇2Φ(x)

Φ(x)d = h4 − qb2 +

n∑i=1

a2i − λ(h4 − (qb−

n∑i=1

ai)2)

= (1− λ)h4 + (λq − 1)qb2

−2λqbn∑i=1

ai + λ(n∑i=1

ai)2 +

n∑i=1

a2i . (4)

Therefore, we obtain for any j ∈ {1, 2, · · · , n},

dT∇2Φ(x)

Φ(x)d = (1− λ)h4 +

n∑i=1

a2i −

λ

λq − 1(n∑i=1

ai)2

+

(√λq2 − qb− λ

√q/(λq − 1)

n∑i=1

ai

)2

≥ (1− λ)h4 +n∑i=1

a2i −

λ

λq − 1(n∑i=1

ai)2

≥ (1− λ)h4 +lλ− 1

lλ+ λ− 1a2j (5)

where the last inequality follows using Lemma 2.By choosing λ = 1/l in (5) it follows that h2 ≥ l−1

l h4, or equiva-

lently,

h2 ≤ l

l − 1

which proves the first part of the lemma (see also Iri [11], equation(3.24)).

Moreover, from (5) we obtain

a2j ≤ min

1/l<λ≤1

lλ+ λ− 1

lλ− 1(h2 − (1− λ)h4)

≤ l + 3

l(h2 − 2l − 3

2lh4) ≤ l + 3

2(2l − 3)

where the second inequality follows by letting λ = 32l .

2

J.F. Sturm and S. Zhang: New complexity results for the Iri-Imai method10

The following lemma gives a bound on the change in objectivevalue along the Newton direction d by means of bounding the quantity

s :=√∑n

i=1(ai − b)2 which will prove useful in the subsequent analysisas well.

Lemma 5 Let s :=√∑n

i=1(ai − b)2. Then,

s2 = (1 + 2b)h2 − h4 − lb2 ≤ h2 − l − 1

lh4 ≤ 1

4

l

l − 1.

Moreover, we have

b =h2

l±

√1

l(h2 − l − 1

lh4 − s2)

and

| b |< 1

l − 1+

√1

2(l − 1).

Proof:We first notice that

s2 =n∑i=1

(ai − b)2

=n∑i=1

a2i + nb2 − 2b

n∑i=1

ai

= (h2 − h4 + qb2) + nb2 + 2b(h2 − qb)= (1 + 2b)h2 − h4 − lb2

which proves the first part of the lemma. Now solving for b the abovederived identity

lb2 − 2h2b+ s2 − h2 + h4 = 0

yields

b =h2

l±

√1

l(h2 − l − 1

lh4 − s2)

and consequently,

| b |≤ h2

l+

√1

l(h2 − l − 1

lh4) <

1

l − 1+

√1

2(l − 1).


2

Clearly (ai − b)2 ≤ s2 for any 1 ≤ i ≤ n and so we have for anyj ≥ 2

n∑i=1

| ai − b |j≤ sj−2n∑i=1

(ai − b)2 = sj (6)

where the equality follows from the definition of s.In view of Lemma 1, we want to estimate the improvement in

potential function that can be obtained by performing a line searchalong the Newton direction d. In order to do so, we will use the powerseries expansion of the potential function.

It is well known that for any | y |< 1 we have the power seriesexpansion

ln(1− y) = −∞∑k=1

yk

k.

Let t be such that t | b |< 1 and t | ai |< 1 for any i ∈ {1, 2, · · · , n}.Then we have

φ(x+ td) = q ln((z − bTx)(1− tb)

)−

n∑i=1

ln((ci − aTi x)(1− tai))

= φ(x) + q ln(1− tb)−n∑i=1

ln(1− tai)

= φ(x)−∞∑k=1

q(tb)k −∑ni=1(tai)

k

k

= φ(x)−∞∑k=1

tkrkk

where rk := qbk −∑ni=1 a

ki . Therefore, the reduction in potential

function value obtained by the step td is

φ(x)− φ(x+ td) =∞∑k=1

tkrkk. (7)

From this equation it is clear that estimating rk should be inter-esting. First notice that r1 = h2 and r2 = h4 − h2.

Lemma 6 Let rk := qbk −∑ni=1 a

ki . Then for any k ≥ 2 it holds that

| rk |≤ (l − 1) | b |k +k(h2 + l | b | −s) | b |k−1 +(s+ | b |)k.


Proof:Using Newton’s binomial formula, we have for any k ≥ 2

rk = qbk −n∑i=1

(ai − b+ b)k

= qbk −k∑j=0

n∑i=1

(kj

)(ai − b)j bk−j

= qbk − nbk − kn∑i=1

(ai − b)bk−1 −k∑j=2

n∑i=1

(kj

)(ai − b)j bk−j

= lbk + k(h2 − lb)bk−1 −k∑j=2

n∑i=1

(kj

)(ai − b)j bk−j

which implies

| rk | ≤∣∣∣lbk + k(h2 − lb)bk−1

∣∣∣+ k∑j=2

n∑i=1

(kj

)| ai − b |j | b |k−j

≤∣∣∣lbk + k(h2 − lb)bk−1

∣∣∣+ k∑j=2

n∑i=1

(kj

)sj | b |k−j

≤∣∣∣lbk + k(h2 − lb)bk−1

∣∣∣+ (s+ | b |)k− | b |k −ks | b |k−1

≤ (l − 1) | b |k +k(h2 + l | b | −s) | b |k−1 +(s+ | b |)k

where the second inequality follows from (6).2

The reduction in potential function value by searching along theNewton direction is estimated in the following lemma.

Lemma 7 (reduction by searching along Newton direction) Lett := 0.1, then

φ(x, z)− φ(x+ td, z) > 0.07h2.

Proof:First note that if t := 0.1 then it follows using Lemma 4 and

Lemma 5

t2a2j ≤ (0.1)2 l + 3

2(2l − 3)< 1 and t | b |< 0.1(

1

l − 1+

√1

2(l − 1)) < 1


which shows that td is a feasible step. Therefore, we can use (7) toobtain

φ(x, z)− φ(x+ td, z) =∞∑k=1

tkrkk

= th2 +t2

2(h4 − h2) +

∞∑k=3

tkrkk

≥ 2− t+ th2

2th2

−∞∑k=3

(l − 1) | b |k +k(h2 + l | b | −s) | b |k−1

ktk

−∞∑k=3

(s+ | b |)k

ktk

=2− t+ th2

2th2 − (l − 1)t3 | b |3

∞∑k=0

| tb |k

k + 3−

−(h2 + l | b | −s)t3b2∞∑k=0

| tb |k

−t3(s+ | b |)3∞∑k=0

tk(s+ | b |)k

k + 3

≥ 2− t+ th2

2th2 − (l − 1)t3 | b |3

3(1− t | b |)

−(h2 + l | b | −s)t3b2

1− t | b |− t3(s+ | b |)3

3(1− t(s+ | b |)). (8)

From Lemma 5 we know that s2 = (1 + 2b)h2 − h4 − lb2. Hence

(s+ | b |)2 = s2 + 2s | b | +b2

= h2 − h4 + 2h2b+ 2s | b | −(l − 1)b2

≤ h2 − h4 + (2(h2 + s)− (l − 1) | b |) | b |

≤ h2 − h4 +(h2 + s)2

l − 1

≤ h2 − h4 + 2h4 + s2

l − 1

≤ l + 1

l − 1h2 − (1− 2

l(l − 1))h4


≤ l + 1

l − 1h2,

where the fourth inequality follows by noticing s2 ≤ h2 − l−1l h

4.Clearly, t2(s+ | b |)2 ≤ 0.03h2 ≤ 0.06. Here notice that we assume

l ≥ 2 and so h2 ≤ 2. It follows then t(s+ | b |) < 0.3. Therefore,

t3(s+ | b |)3

3(1− t(s+ | b |))<

0.3× 0.03h2

3(1− 0.3)< 0.005h2. (9)

From Lemma 5 we know that | b |< 1l−1 +

√1

2(l−1) , which implies

t | b |< 0.1(1 +

√2

2) < 0.2.

Moreover,

b2 ≤ 2(h4

l2+ (

h2

l− (l − 1)h4

l2)) ≤ 2h2

l.

So,

(l − 1)t3 | b |3

3(1− t | b |)<

(l − 1)× 0.2× (0.1)2 × 2h2

l

3(1− 0.2)< 0.002h2 (10)

and

(h2 + l | b | −s)t3b2

1− t | b |<

t3 | b |2 h2

1− 0.2+lt3 | b |3

1− 0.2

<0.004h2

0.8+

0.004h2

0.8= 0.01h2. (11)

Combining (9),(10) and (11) with (8), we get

φ(x, z)− φ(x+ td, z) ≥ 2− t+ th2

2th2 − 0.017h2

> 0.07h2.

2

From Lemma 7, we know that for a certain step length along theNewton direction, the reduction in potential function value will be atleast 0.07h2. If the upper bound z is fixed to the optimal value z∗ :=maxx∈F b

Tx, this implies a constant reduction in potential functionvalue, due to the following lemma by Iri [11].


Lemma 8 If z = z∗, then h2 ≥ 1/2.

Proof:See Iri [11], inequality (3.15). Iri’s proof is based on Cauchy-

Schwartz’ inequality. As an alternative, we will show that the in-equality also follows from a duality relation, see Corollary 1.2

Combining Lemma 7 and Lemma 8 with Lemma 1, it follows thatthe following Iri-imai algorithm can be used to solve (D) in O(nL)iterations.

Algorithm 1 (Iri and Imai’s algorithm. Input data: A,b,c,l,x0,z∗)

Step 0 Set i = 0.

Step 1 If z∗ − bTxi ≤ 2−2L, stop.

Step 2 Set d = −∇2Φ(xi, z∗)−1∇Φ(xi, z∗).

Step 3 Compute xi+1 by minimizing φ(x, z∗) along d.

Step 4 Set i = i+ 1 and return to Step 1.

Theorem 1 For any l ≥ 2, Algorithm 1 solves (D) in O((n + l)L)main iterations.

Proof:Combining Lemma 7 and Lemma 8 with Lemma 1, the theorem

follows.2

4 Primal–dual potential reduction

So far the Iri-Imai method, like the original Karmarkar method, doesnot include both primal and dual variables. In this section we willmodify the Iri-Imai method to incorporate a primal–dual potentialfunction and so it becomes a primal–dual potential reduction method.


More specifically, we shall now try to construct the next iterativeinterior solution x′, or update the variable λ, based on the current xand λ, in such a way that a certain reduction can be guaranteed inthe value of the primal–dual potential function ψ defined by

ψ(x, λ) := q ln(cTλ− bTx)−n∑i=1

ln(ci − aTi x)−

n∑i=1

ln(λi)

where q = n+ l > n and x and λ are interior solutions of (D) and (P)respectively (i.e. x ∈ F0 and λ > 0, Aλ = b).

The primal–dual potential function was introduced by Tanabe in1987. Unlike Karmarkar’s potential function, the primal–dual poten-tial function needs to be reduced by an amount of only O(lL), com-pared to O(qL) in Karmarkar’s case, in order to obtain x and λ suchthat cTλ − bTx ≤ 2−2L. Here it is assumed that the initial dual in-terior point x0 and the initial upper bound z0 are generated in sucha way that there exists a primal feasible λ0 > 0 with z0 = cTλ0 andψ(x0, λ0) = O(lL). This is because the arithmetic mean inequality(

n∏i=1

yi

)1/n

≤n∑i=1

yi/n

implies

n∑i=1

ln(λi(ci − aix)) ≤ n ln(λT(c−ATx)

n)

= n ln(cTλ− bTx)− n ln(n)

i.e.,n∑i=1

ln(λi(ci − aix)

(cTλ− bTx)/n) ≤ 0. (12)

By rewriting the primal-dual potential function as

ψ(x, λ) = l ln(cTλ− bTx) + n ln(n)−n∑i=1

ln(λi(ci − aix)

(cTλ− bTx)/n)

≥ l ln(cTλ− bTx) + n ln(n)

it is clear that if ψ(x, λ) ≤ −2lL ln(2), then cTλ − bTx ≤ 2−2L, inwhich case x can be purified to an optimal solution of (D) in onlyO(n3) operations. This proves the following lemma.


Lemma 9 Let δ > 0 be a fixed quantity. Suppose Assumptions 1and 2 are satisfied, and an initial interior point x0 ∈ F0 and aninitial upper bound z0 are available satisfying the property that thereexist λ0 > 0 with Aλ0 = b, cTλ0 = z0 and ψ(x0, λ0) = O(lL). Ifan algorithm reduces the primal–dual potential function ψ(x, λ) by atleast δ at each iteration, then the algorithm can be used to solve (D)in O( l

δ L) iterations.

Proof:See Ye [16] or Freund [3].

2

By using Lemma 9, Ye [16] was the first to obtain an O(√nL)-

iteration bound for a potential reduction algorithm.It is easy to see that

∇xψ(x, λ) = ∇φ(x, cTλ)

and∇2xψ(x, λ) = ∇2φ(x, cTλ).

Therefore, if h2 = ∇φ(x, cTλ)T∇2Φ(x)−1∇Φ(x, cTλ) ≥ 19 it follows

from Lemma 7 that the primal–dual potential function can be reducedby an amount of at least 0.007 by doing a line-search in F0 along dwhere d := −∇2Φ(x)−1∇Φ(x, cTλ). We proceed by showing that ifh < 1

3 then it is possible to construct new primal variables λ′, in sucha way that the primal–dual potential function ψ is again reduced byat least a constant amount, for some special choice of l.

In the derivation of the primal updating formula, we will divide bythe quantity 1−h2 + b, which is valid for small h due to the followinglemma.

Lemma 10 If h2 < 1 then

1− h2 + b > 0.

Proof:Suppose that we have h2 < 1 and 1−h2 + b ≤ 0. This implies that

b < 0. We then get the following impossible inequalities (cf. Lemma 5)

0 ≤ s2 = −lb2 + (1 + 2b)h2 − h4 = h2(1− h2 + b) + bh2 − lb2 < 0.


The lemma is proved by contradiction.2

From

∇φ(x, z) = −∇2Φ(x)d

Φ(x)

= −(∇φ(x)Td)∇φ(x)−n∑i=1

aiai

ci − aTi x

+ qbb

z − bTx

where −∇φ(x)Td = h2 and∇φ(x) = −q bz−bTx+

∑ni=1

aici−aTi x

, it follows

that

b =z − bTx

q(1− h2 + b)

n∑i=1

(1− h2 + ai)ai

ci − aTi x

= Aλ′ (13)

where we define λ′1, λ′2, · · · , λ′n as

λ′i :=z − bTx

q(ci − aTi x)

1− h2 + ai1− h2 + b

. (14)

Therefore

cTλ′ − bTx = (cT − xTA)λ′

=n(1− h2) +

∑ni=1 ai

q(1− h2 + b)(z − bTx)

=n(1− h2 + a)

q(1− h2 + b)(z − bTx) (15)

where we define a :=∑ni=1 ai/n to simplify notations.

Using relation (15), it follows from (14), for i = 1, 2, · · · , n, that

λ′i(ci − aTi x)

(cTλ′ − bTx)/n=

1− h2 + ai1− h2 + a

, (16)

which shows that when h is small, the mean-inequality (12) will betight after updating λ.


By noting that qb− h2 = na, we rewrite (15) as follows

cTλ′ − bTx =n(1− h2) + qb− h2

q(1− h2 + b)(z − bTx)

= (1− l − (l − 1)h2

q(1− h2 + b))(z − bTx), (17)

which shows that cTλ′ < z, i.e. λ′ can be used to reduce the upperbound z if λ′ is primal feasible. The following lemma gives a sufficientcondition for λ′ to be feasible.

Lemma 11 If h2 < 12 , then λ′ > 0.

Proof:Since λ′i := z−bTx

q(ci−aTi x)

(1+ ai−b1−h2+b

), we have λ′ > 0 iff (1+ ai−b1−h2+b

) >

0. Moreover,

1 +ai − b

1− h2 + b≥ 1− | ai − b |

1− h2 + b≥ 1− s

1− h2 + b.

Because s2 = h2 + 2h2b− h4 − lb2 we have

(1− h2 + b)2 − s2 = 1− 3h2 + 2h4 + (2− 4h2)b+ (l + 1)b2

≥ 1− 3h2 + 2h4 − (1− 2h2)2

l + 1.

If h2 < 1/2, it follows from the inequality above that

(1− h2 + b)2 − s2 > 0,

which implies

1− s

1− h2 + b> 0.

The lemma is proved.2

As a corollary, we have a restatement of Lemma 8.

Corollary 1 If z = z∗ := maxx∈F bTx, then h2 ≥ 1

2 .


Proof:By the duality theorem, we have for any primal feasible λ and dual

feasible x that cTλ ≥ bTx. Lemma 11 however, shows that if h2 < 12 ,

there exists a primal feasible λ′ for which (17) holds, i.e. cTλ′ < z.This contradicts the duality theorem, and proves the corollary andtherefore also Lemma 8.2

Consider now the case h2 < 1/2. The reduction in potential func-tion value caused by the primal update, is given by

ψ(x, λ)− ψ(x, λ′) = q ln(cTλ− bTxcTλ′ − bTx

)−n∑i=1

ln(λi(ci − aTi x))

+n∑i=1

ln(λ′(ci − aTi x))

= −l ln(cTλ′ − bTxz − bTx

)

−n∑i=1

ln(λi(ci − aT

i x)

(z − bTx)/n)

+n∑i=1

ln(λ′i(ci − aT

i x)

(cTλ′ − bTx)/n). (18)

The components in (18) can be further estimated. Using (17) weobtain

l ln(cTλ′ − bTxz − bTx

) = l ln(1− l − (l − 1)h2

q(1− h2 + b)) (19)

and using (12),n∑i=1

ln(λi(ci − aT

i x)

(z − bTx)/n) ≤ 0 (20)

and finally using (16),

n∑i=1

ln(λ′i(ci − aT

i x)

(cTλ′ − bTx)/n) =

n∑i=1

ln(1− h2 + ai1− h2 + a

)

=n∑i=1

ln(1 +ai − a

1− h2 + a). (21)


Now using (19), (20) and (21) together with (18) it follows that

ψ(x, λ)− ψ(x, λ′) ≥ −l ln(1− l − (l − 1)h2

q(1− h2 + b)) +

n∑i=1

ln(1 +ai − a

1− h2 + a).

(22)In order to further estimate the reduction, we need the following lem-ma (see also Karmarkar [12]).

Lemma 12 Let | y |< 1. Then,

−y ≥ ln(1− y) ≥ −y − y2

2(1− | y |)

If additionally, −1 < y ≤ 0, then

−y =| y |≥ ln(1− y) ≥ −y − y2

2.

Proof:The first part is proven in Karmarkar [12]. The second part of the

lemma follows by observing

ln(1− y) = −y − y2

2− y3

∞∑k=0

yk

k + 3

= −y − y2

2− y3

∞∑k=0

(y2k

2k + 3+

y2k+1

2k + 4)

and, if −1 < y ≤ 0, we have 12k+3 + y

2k+4 > 0.2

Now we will use Lemma 12 to prove that inequality (22) implies areduction in primal-dual potential function value by at least a constantamount, when l =

√n+ 3 and h < 1/3.

Lemma 13 (Reduction by the primal step) Let l =√n+ 3 and

h < 1/3 and let λ be an interior point of (P) such that cTλ = z. Thenthere exists an interior point λ′ of (P) such that

cTλ′ = z − l − (l − 1)h2

q(1− h2 + b)(z − bTx)

andψ(x, λ)− ψ(x, λ′) > 0.25.


Proof:Let λ′ := z−bTx

q(ci−aTi x)

1−h2+ai1−h2+b

. From (13) and Lemma 11 it follows

that λ′ is an interior point of (P), i.e. Aλ′ = b and λ′ > 0. From (17)we know that

cTλ′ = z − l − (l − 1)h2

q(1− h2 + b)(z − bTx),

and from (22) we have

ψ(x, λ)− ψ(x, λ′) ≥ −l ln(1− l − (l − 1)h2

q(1− h2 + b)) +

n∑i=1

ln(1 +ai − a

1− h2 + a).

Clearly, we have

0 <cTλ′ − bTxz − bTx

= 1− l − (l − 1)h2

q(1− h2 + b)< 1

which enables us to apply Lemma 12 and obtain

−l ln(1− l − (l − 1)h2

q(1− h2 + b)) ≥ l2

q

1− l−1l h

2

1− h2 + b.

As we know from Lemma 5,

b =h2

l±

√1

l(h2 − l − 1

lh4 − s2)

and therefore,

−l ln(1− l − (l − 1)h2

q(1− h2 + b)) ≥ l2

q

1− l−1l h

2

1− l−1l h

2 +√

1l (h

2 − l−1l h

4)

=l2

q

1

1 +√

h2

l−(l−1)h2

≥ 1

2

1

1 +√

1/16= 0.4. (23)

We will complete the proof by estimating the second term in (22),namely

∑ni=1 ln(1 + ai−a

1−h2+a), from below.


As we already know from (3),

ai ≥ −

√l

l − 1h

for i = 1, 2, · · · , n. This inequality holds also for the mean value, i.e.

a ≥ −√

ll−1h. Using l =

√n+ 3 ≥ 2 and h < 1/3 we obtain

(1− h2 −√l/(l − 1)h)2 ≥ (1− 1

9−√

2

3)2 > 0.4. (24)

We know that the arithmetic mean minimizes the sum of squareddeviations, therefore

n∑i=1

(ai − a)2 ≤n∑i=1

(ai − b)2 = s2.

Using Lemma 5 we have s2 ≤ h2 − l−1l h

4, and this gives

n∑i=1

(ai − a)2 ≤ h2 − l − 1

lh4 ≤ 1

9− 1

162< 0.11. (25)

This shows that ∣∣∣∣ ai − a1− h2 + a

∣∣∣∣ <√

0.11

0.4< 1,

which enables us to write the logarithmic function as a power series.In particular, it follows using Lemma 12 that if a− ai ≥ 0,

ln(1− a− ai1− h2 + a

) ≥ ai − a1− h2 + a

− 1

2

(ai − a)2

(1− h2 + a)(1− h2 + ai)

and if a− ai < 0,

ln(1− a− ai1− h2 + a

) ≥ ai − a1− h2 + a

− 1

2

(ai − a)2

(1− h2 + a)2.

Summarizing, we have derived the following inequality

n∑i=1

ln(1 +ai − a

1− h2 + a) ≥

n∑i=1

ai − a1− h2 + a

−1

2

n∑i=1

(ai − a)2

(1− h2 −√l/(l − 1)h)2

= −1

2

∑ni=1(ai − a)2

(1− h2 −√l/(l − 1)h)2

.


Therefore, using the estimations (24) and (25),

n∑i=1

ln(1 +ai − a

1− h2 + a) > −0.5

0.11

0.4> −0.15. (26)

Combining (23) and (26) with (22) shows that the reduction willbe at least 0.4− 0.15 = 0.25, which proves the lemma.2

Combining Lemma 13 and Lemma 7 with Lemma 9, it follows thatthe following algorithm can be used to solve (D) in O(

√nL) iterations.

Algorithm 2 (Primal–dual Iri and Imai’s Algorithm. Input data:A,b,c,x0,z0)

Step 0 Set l =√n+ 3, q = n+ l, i = 0.

Step 1 If zi − bTxi ≤ 2−2L, stop.

Step 2 Set d = −∇2Φ(xi, zi)−1∇Φ(xi, zi) and h =√−∇φ(xi, zi)Td.

Step 3 If h < 13 go to Step 6. Otherwise go to Step 4.

Step 4 Compute xi+1 by minimizing φ(x, zi) along d. Set zi+1 = zi.


Step 6 Set zi+1 = zi - l−(l−1)h2

q(1−h2+b)(zi-bTxi).


Theorem 2 Algorithm 2 solves (D) in O(√nL) main iterations.

Proof:Combining Lemma 13 and Lemma 7 with Lemma 9, the theorem

follows.2


5 Conclusions

To the best knowledge of the authors, the O(√nL)- iteration bound is

currently the best one for any interior point algorithm available. Thebenefit of the algorithm presented in this paper is that it uses Newtondirections for minimizing the dual part of the potential function. Afast convergence can therefore be expected.

The quantity h2 is a good proximity measure for the current dualsolution x to the reference point on the central path determined bythe current primal objective value z. The choice of the parameter lplays an important role in the complexity analysis of the algorithm.Once h2 gets small, larger l will result in a better primal updatingformula in terms of reducing the duality gap. However, when l islarge it is more difficult to get close to the reference point on thecentral path for the dual iterative points. A compromise yields thetheoretically best choice of l: l = O(

√n). Since the dual iterates are

expected to converge fast due to the Newton method being used, itis advisable in practice to let l = O(n). The difference in the choiceof l resembles the difference between the short-step and the large-steppath following methods. For the original Iri-Imai algorithm withoutincorporating the primal variables, it is then better to use small l(e.g. let l be a constant). Because in this case the reduction of thepotential function will be more directly linked to the reduction in thegap between the optimal value and the current objective value, whichis of primary interest for solving the problem.

Finally, as a remark we mention that using a primal–dual reductionmethod so as the one presented in this paper, one can conclude that ifan LP problem would have nonempty interior feasible region and anoptimal solution then the dual problem must have nonempty interiorfeasible region as well.


References

[1] Anstreicher, K.M., A combined phase I-phase II scaled potentialalgorithm for linear programming, Mathematical Programming52, pp. 429-439, 1991.

[2] Dikin, I.I., Iterative solution of problems of linear and quadraticprogramming, Soviet Mathematics Doklady 8, pp. 674-675, 1967.

[3] Freund, R.M., Polynomial-time algorithms for linear program-ming based only on primal scaling and projected gradients of apotential function, Mathematical Programming 51, pp. 203-222,1991.

[4] Freund, R.M., A potential-function reduction algorithm for solv-ing a linear program directly from an infeasible “warm start”,Mathematical Programming 52, pp. 441-466, 1991.

[5] Gonzaga, C.C., Polynomial Affine Algorithms for Linear Pro-gramming, Mathematical Programming 49, pp. 7-21, 1990.

[6] Gonzaga, C.C., Interior Point Algorithms for Linear Program-ming Problems with Inequality Constraints, Mathematical Pro-gramming 52, pp. 209-226, 1991.

[7] Gonzaga, C.C., Path-following methods for Linear Programming,SIAM Review, Vol. 34, No. 2, pp. 167-224, 1992.

[8] Huard, P., Resolution of mathematical programming with nonlin-ear constraints by the method of centres, Nonlinear Programming,ed. J. Abadie, North Holland Publishing Company, Amsterdam,pp. 207-219, 1967.

[9] Imai, H., On the polynomiality of the multiplicative penalty func-tion method for linear programming and related inscribed ellip-soids, IEICE Transactions, Vol. E74, No. 4, pp. 669-671, 1991.

[10] Iri, M., Imai, H., A multiplicative Barrier Function Method forLinear Programming, Algorithmica 1, pp. 455-482, 1986.

[11] Iri, M., A Proof of the Polynomiality of the Iri-Imai Method,

Presented at the 14th International Symposium on MathematicalProgramming, Amsterdam, 1991.

[12] Karmarkar, N., A New Polynomial-Time Algorithm for LinearProgramming, Combinatorica, Vol. 4, pp. 373-395, 1984.


[13] Renegar, J., A polynomial-time algorithm, based on Newton’smethod, for Linear Programming, Mathematical Programming40, pp. 59-93, 1988.

[14] Sonnevend, Gy., An “analytical centre” for polyhedrons and newclasses of global algorithms for linear (smooth, convex) program-ming, Lecture Notes in Control and Information Sciences, Vol. 84,ed. Prekopa, A., Szelezsan, J. and Strazicky, B., Springer Verlag,pp. 866-875, 1986.

[15] Tsuchiya, T. and Muramutsu, M., Global convergence of the long-step affine scaling algorithm for degenerate linear programmingproblems, Research Memorandum 423, The Institute of Statis-tical Mathematics, 4-6-7 Minami-Azabu, Minato-ku, Tokyo 106,Japan, 1992.

[16] Ye, Y, An O(n3L) potential reduction algorithm for linear pro-gramming, Mathematical Programming 50, pp. 239-258, 1991.

[17] Zhang, S. and Shi, M., On the polynomiality of Iri and Imai’s newalgorithm for linear programming, Journal of Qinhua University28, pp. 121-126, 1988 (in Chinese).

[18] Zhang, S., The convergence property of Iri-Imai’s Algorithm forsome Smooth Convex Programming Problems, Research Mem-orandum Nr. 485, Department of Econometrics, University ofGroningen, the Netherlands, 1992 (to appear in Journal of Opti-mization Theory and Applications).

new complexity results for the iri-imai methodzhangs/reports/1996_sz-iri-imai.pdfnew complexity...

Documents