an adaptive accelerated first-order method for convex optimization … · an adaptive accelerated...

An adaptive accelerated first-order method for convex optimization

Renato DC Monteirolowast Camilo Ortizdagger Benar F SvaiterDagger

July 30 2012 (Revised May 14 2014)

Abstract

This paper presents a new accelerated variant of Nesterovrsquos method for solving compositeconvex optimization problems in which certain acceleration parameters are adaptively (and ag-gressively) chosen so as to substantially improve its practical performance compared to existingaccelerated variants while at the same time preserve the optimal iteration-complexity shared bythese methods Computational results are presented to demonstrate that the proposed adaptiveaccelerated method endowed with a restarting scheme outperforms other existing acceleratedvariants

1 Introduction

The methods of choice for solving large-scale convex optimization problems specially when highaccuracy is not needed are first-order methods due to their cheap iteration cost (time and mem-ory) In [11] (see also [13]) Nesterov presents a scheme for accelerating first-order methods morespecifically the steepest descent method for unconstrained convex optimization and more gener-ally the projected gradient method for constrained convex optimization He shows that the rateof convergence of these methods namely O(1k) where k denotes the iteration count can be im-proved to O(1k2) by considering their corresponding accelerated variants Due to its wide use forsolving large-scale convex optimization problems arising in several applications other acceleratedvariants of Nesterovrsquos method for the aforementioned problems and more generally the compositeconvex optimization problem have been proposed and studied in the literature (see for example[1 3 5 6 8 10 11 12 14 15 18]) Moreover there has been an increasing effort towards improvingthe practical performance of these methods (see for example [6 16]) in the solution of large-scaleconvex optimization problems

All of the accelerated variants mentioned above use certain accelerated parameters which areobtained by using well-known update formulas This paper presents a new accelerated variantfor composite convex optimization in which the acceleration parameters are adaptively (and ag-gressively) chosen so as to substantially improve its practical performance in comparison to theseaforementioned variants while at the same time preserve the optimal iteration-complexity shared bythese methods (with the exemption of [16]) More specifically the new accelerated variant chooses

lowastSchool of Industrial and Systems Engineering Georgia Institute of Technology Atlanta GA 30332-0205(emailmonteiroisyegatechedu) The work of this author was partially supported by NSF Grants CMMI-0900094and CMMI-1300221 and ONR Grant ONR N00014-11-1-0062daggerSchool of Industrial and Systems Engineering Georgia Institute of Technology Atlanta GA 30332-0205 (email

camiortgatechedu)DaggerIMPA Estrada Dona Castorina 110 22460-320 Rio de Janeiro Brazil (emailbenarimpabr) The work of this

author was partially supported by CNPq grants no 3035832008-8 3029622011-5 4801012008-6 4749442010-7FAPERJ grants E-261028212008 E-261029402011

1

the acceleration parameters at every iteration by solving a simple two-variable convex quadrati-cally constrained linear program Moreover its iteration cost is comparable to or better than thatof the aforementioned accelerated methods since it only computes a single resolvent (or proximalsubproblem) at every iteration

For the purpose of our computational experiments we have implemented the new acceleratedvariant endowed with two restarting schemes in order to improve its practical performance Thefirst restarting scheme is an aggressive one which restarts the method from the last iterate wheneverthe objective function increases (also proposed in [16]) The second restarting scheme is a moreconservative (and more robust) one which restarts the method whenever the objective functionincreases and the number of iterations at that point is sufficiently large Finally computationalresults are presented on several conic quadratic programming instances showing that the new ac-celerated variant endowed with either one of the restarting schemes is faster and more robust thanother existing Nesterovrsquos variants

Our paper is organized as follows Section 2 presents a class of accelerated first-order methodsand corresponding iteration-complexity results for solving composite convex optimization problemsMoreover this section proposes a specific instance of the latter class which adaptively chooses thesequence of acceleration parameters so as to greedily minimize an upper bound on the primal gapSection 3 presents computational results comparing the latter instance (endowed with one of theaforementioned restart schemes) with other existing variants of Nesterovrsquos method Finally Section4 presents some final remarks

2 An accelerated first-order method

This section introduces an accelerated first-order method presented in [10] for solving compositeconvex optimization problems First we propose a method with general acceleration parameterssatisfying certain conditions which still guarantee an optimal iteration-complexity Next for thepurpose of improving the practical performance of the method we propose a specific way for choosingthese parameters that greedily minimizes an upper bound on the primal gap

Let R denote the set of real numbers and define R = R cup plusmninfin Also let X denote a finitedimensional real inner product space with inner product and induced norm denoted by 〈middot middot〉 and middot respectively Our problem of interest is the composite convex optimization problem

φlowast = minxisinX

φ(x) = f(x) + h(x) (1)

where

C1 h X rarr R is a proper closed convex function

C2 f is a real-valued function which is differentiable and convex on a closed convex set

Ω supe domh = x isin X h(x) ltinfin

C3 nablaf is L-Lipschitz continuous on Ω for some L gt 0 ie

nablaf(xprime)minusnablaf(x) le Lxprime minus x forallx xprime isin Ω

C4 the set Xlowast of optimal solutions of (1) is non-empty

There are many accelerated variants of Nesterovrsquos method for solving (1) under assumptions C1ndashC4 In this paper we are interested in studying the properties of the following accelerated variantof Nesterovrsquos method whose statement uses the projection operator ΠΩ X rarr Ω onto Ω defined as

ΠΩ(x) = arg minxisinΩxminus x forallx isin X

2

Accelerated Class A class of accelerated algorithms for (1)

0) Let λ isin (0 1L] and x0 isin X be given and set A0 = 0 y0 = x0 the affine function Γ0 X rarr Ras Γ0 equiv 0 and k = 1

1) compute ak xk xkΩ yk and the affine function γk X rarr R as

ak =λ+

radicλ2 + 4λAkminus1

2 (2)

xk =Akminus1

Akminus1 + akykminus1 +

akAkminus1 + ak

xkminus1 xkΩ = ΠΩ(xk) (3)

yk = arg minxisinX

λh(x) +

1

2xminus xk + λnablaf(xkΩ)2

(4)

γk(x) = f(xkΩ) + 〈nablaf(xkΩ) yk minus xkΩ〉+ h(yk) +1

λ〈xk minus yk xminus yk〉 forallx isin X (5)

2) choose a pair (Aprimekminus1 aprimek) = (Aprime aprime) isin Rtimes R such that

Aprime ge 0 aprime ge 0 Aprime + aprime ge Akminus1 + ak (6)

(Aprime + aprime)φ(yk) le infxisinX

(AprimeΓkminus1 + aprimeγk)(x) +

1

2xminus x02

(7)

with the safeguard that Aprime0 = A0 = 0 when k = 1

3) compute

Ak = Aprimekminus1 + aprimek (8)

Γk =Aprimekminus1

Aprimekminus1 + aprimekΓkminus1 +

aprimekAprimekminus1 + aprimek

γk (9)

xk = x0 minusAknablaΓk (10)

4) set k larr k + 1 and go to step 1

Observe that (7) (8) and (9) imply

Akφ(yk) le infxisinX

AkΓk(x) +

1

2xminus x02

(11)

Also note that the Accelerated Class does not specify how to choose (Aprimekminus1 aprimek) isin RtimesR satisfying

(6) and (7) Different choices of the pair (Aprimekminus1 aprimek) determines different instances of the class For

example if the pair (Aprimekminus1 aprimek) is chosen as (Aprimekminus1 a

primek) = (Akminus1 ak) then using equation (30) of

Lemma A3 it can be shown that the Accelerated Class reduces to Algorithm I of [10] Moreoverif (Aprimekminus1 a

primek) = (Akminus1 ak) and Ω = X (and hence the gradient of f is defined and is L-Lipschitz

continuous on the whole X ) then the Accelerated Class reduces to a well-known Nesterovrsquos accel-erated variant namely FISTA [1] On the other hand this paper is concerned with the instanceof the above class which chooses the pair (Aprimekminus1 a

primek) which maximizes Ak subject to conditions (8)

and (11)

3

The following result not only shows that the pair (Aprimekminus1 aprimek) = (Akminus1 ak) satisfies (6) and (7)

but that any instance of the above class has optimal iteration-complexity

Proposition 21 The following statements hold for every k ge 1

a) (Aprimekminus1 aprimek) = (Akminus1 ak) satisfies (6) and (7) and hence the Accelerated Class is well-defined

b) Γk is affine and Γk le φ

c) Ak ge λk24

d) there holds

φ(yk)minus φlowast le d20

2Akle 2d2

0

λk2(12)

where d0 is the distance of x0 to Xlowast

Proof Statements a) b) and c) follow from Lemma A3 Letting xlowast denote the projection of x0

onto Xlowast d) follows from c) and the fact that (11) and b) imply that

Akφ(yk) le AkΓk(xlowast) +1

2xlowast minus x02 le Akφlowast +

1

2d2

0

Observe that Proposition 21(d) implies that any instance of the Accelerated Class with λ = 1Lhas the optimal iteration-complexity O(Ld2

0k2) Moreover based on the fact that (8) and (12)

imply that

φ(yk)minus φlowast = O

(1

Aprimekminus1 + aprimek

)

our new accelerated variant chooses the pair (Aprimekminus1 aprimek) as

(Aprimekminus1 aprimek) isin arg max

AprimeaprimeAprime + aprime Aprime ge 0 aprime ge 0 and (Aprime aprime) satisfies (7) (13)

so as to greedily reduce the primal gap φ(yk)minus φlowast as much as possibleWe will now establish that (7) is equivalent to a convex quadratic constraint and hence that

(13) is a simple two-variable convex quadratically constrained linear program We first state thefollowing simple technical result

Lemma 22 Let χ isin R x0 y isin X and an affine function Γ X rarr R be given Then the followingconditions are equivalent

a) minΓ(x) + xminus x022 x isin X ge χ

b) χminus Γ(x0) + nablaΓ22 le 0

Proof The result follows from the optimality conditions of the unconstrained optimization problemin a)

The following result gives the aforementioned characterization for condition (7)

Proposition 23 For every k ge 1 condition (7) holds if and only if

Aprime[φ(yk)minus Γkminus1(x0)] + aprime[φ(yk)minus γk(x0)] +1

2AprimenablaΓkminus1 + aprimenablaγk2 le 0 (14)

4

Proof This result follows from Lemma 22 with Γ = AprimeΓkminus1 + aprimeγk χ = (Aprime + aprime)φ(yk) andy = yk

Note that (14) is a single convex quadratic constraint on the scalars Aprime and aprime Hence the newaccelerated method based on (13) chooses the pair (Aprimekminus1 a

primek) as an optimal solution of the simple

two-variable convex quadratically constrained linear program

max Aprime + aprime

st Aprime[φ(yk+1)minus Γk(x0)] + aprime[φ(yk+1)minus γk+1(x0)] + 1

2AprimenablaΓk + aprimenablaγk+12 le 0

Aprime aprime ge 0

(15)

The remaining part of this section discusses the case where problem (15) is unbounded It willbe seen that this case implies that yk isin Xlowast in which case the new accelerated method may besuccessfully stopped The following result whose proof is given in Appendix B considers a slightlymore general problem which contains (15) as a special case

Proposition 24 Suppose that x0 y isin X and γ1 γm X rarr R are affine functions minorizingφ and consider the problem

maxsumm

i=1 αi

st infxisinXsumm

i=1 αiγi(x) + 12xminus x

02gesumm

i=1 αiφ(y)

α1 ge 0 αm ge 0

(16)

Then the following conditions are equivalent

a) problem (16) is unbounded

b) there exist not all zero nonnegative scalars α1 αm such that

msumi=1

αinablaγi = 0 αi[φ(y)minus γi(y)] = 0 i = 1 m (17)

In both cases y isin Xlowast

Proposition 24 deals with the case where (15) is unbounded On the other hand when (15) isbounded its feasible set is compact and hence (15) has an optimal solution

In our computational experiments we will refer to the instance of the Accelerated Class whichchooses the pair (Aprimekminus1 a

primek) as an optimal solution of (15) as the adaptive accelerated (AA) method

3 Numerical results

In this section we describe two restarting variants of the AA method and report numerical resultscomparing them to the following variants of Nesterovrsquos method

i) FISTA (fast iterative shrinkage-thresholding algorithm) of [1] (see also Algorithm 2 of [18])

ii) FISTA-R restarting variant of FISTA

iii) GKR-2 Algorithm 2 of [7]


5

More specifically we compare the performance of these methods using three classes of conic quadraticprogramming instances namely

a) random convex quadratic programs (CQPs) (see Subsection 31)

b) semidefinite least squares (SDLSs) (see Subsection 32) and

c) random nonnegative least squares (NNLSs) (see Subsection 33)

We observe that we have implemented our own code for FISTA-R Although we have recently learnedthat a similar variant was implemented in [16] we have not had the opportunity to include theircode in our computational benchmark Nevertheless we believe that our implementation shouldbe very similar to the method in [16] and hence should reflect the actual performance of the latteralgorithm

We stop all methods whenever an iterate yk is found such that∥∥∥∥yk minus (I + parth)minus1

(yk minus 1

Lnablaf(yk)

)∥∥∥∥ le 10minus6 (18)

or when 2000 iterations have been performed Note that for the case where h is an indicator functionof a closed convex set the left hand side of (18) is exactly the norm of the projected gradient withstepsize 1L

We now briefly describe the two restarting versions of the AA method In the first versionif the function value at the end of the kth iteration increases ie φ(ykminus1) lt φ(yk) then themethod is restarted at step 0 with x0 = ykminus1 and y0 = ykminus1 FISTA-R refers to the variant ofFISTA which incorporates this restarting scheme Moreover FISTA-R is equivalent to one of therestarting variants of FISTA discussed in [16]

In contrast to the first restarting version above the second one allows some iterations to increasethe function value and hence is more conservative than the first one More specifically we restartthis variant at the kth iteration whenever φ(ykminus1) lt φ(yk) and no more than dlog2(k minus lk)e restartshave been performed so far where lk is the iteration where the last restart was performed

We will refer to the first and second restarting variants of the AA method as AA-R1 and AA-R2respectively

The codes for all the benchmarked variants tested are written in MATLAB All the computa-tional results were obtained on a single core of a server with 2 Xeon X5520 processors at 227GHzand 48GB RAM

We now make some general remarks about how the results are reported on the tables givenbelow Tables 1 3 and 5 report the times and Tables 2 4 and 6 report the number of iterationsfor all instances of the three problem classes Each problem class is associated with two tablesone reporting the times and the other one the number of iterations required by each benchmarkedmethod to solve all instances of the class We display the time or number of iterations that a varianttakes on an instance in red and also with an asterisk () whenever it cannot solve the instance tothe required accuracy In such a case the accuracy obtained at the last iteration of the variant isalso displayed in parentheses

Figures 1 3 and 5 plot the time performance profiles (see [4]) and Figures 2 4 and 6 plotthe iteration performance profiles for each of the three problem classes We recall the followingdefinition of a performance profile For a given instance a method A is said to be at most x timesslower than method B if the time taken (resp number of iterations performed) by method A is atmost x times the time taken (resp number of iterations performed) by method B A point (x y)is in the performance profile curve of a method if it can solve exactly (100y) of all the testedinstances x times slower than any other competing method

6

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1

Performance Profiles (60 CQP instances) tol=10-6

(time)

at most x times (the time of the other methods)

o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 1 Time performance profiles for solving CQP instances with accuracy ε = 10minus6 On the left we include all the

methods and on the right we include the three fastest variants namely AA-R1 AA-R2 and FISTA-R

31 Numerical results for random CQPs

This subsection compares the performance of our methods AA-R1 and AA-R2 with the variants ofNesterovrsquos method listed at the beginning of this section on a class of randomly generated sparseCQPinstances These instances were also used to report the performance of GKR-2 and GKR-3 in [7]

Let Rn denote the n-dimensional Euclidean space Sn denote the set of all n times n symmetricmatrices and Sn+ denote the cone of ntimesn symmetric positive semidefinite matrices Given Q isin Sn+b isin Rn l isin R cup minusinfinn and u isin R cup infinn such that l le u the box constrained convexquadratic programming problem is defined as

minxisinRn

xTQx+ bTx l le x le u

(19)

Letting f and h be defined as

f(x) = xTQx+ bTx h(x) = δB(x) forallx isin Rn

where B = x isin Rn l le x le u we can easily see that (19) is a special case of (1) with Ω = X = RnFigures 1 and 2 plot time and iteration performance profiles of all variants of Nesterovrsquos method

for solving this collection of random sparse CQP instances respectively Tables 1 and 2 report thetime and number of iterations taken by each method respectively

Note that AA-R1 and AA-R2 outperform the other methods on most of the random sparse CQPinstances From Figures 1 and 2 we can see that the aggressive restart scheme of AA-R1 performsslight better than the conservative one of AA-R2 on these instances

32 Numerical results for SDLSs

This subsection compares the performance of our methods AA-R1 and AA-R2 with the variants ofNesterovrsquos method listed at the beginning of this section on a class of SDLS instances

7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)

at most x times (the of iterations of the other methods)

o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 2 Iteration performance profiles for solving CQP instances with accuracy ε = 10minus6 On the left we include all the


Given a linear map A isin Sn rarr Rm and b isin Rm the semidefinite programming (SDP) feasibilityproblem consists of finding x such that

Ax = b x isin Sn+

We can solve the above problem by considering the SDLS reformulation

minxisinSn

1

2Axminus b2 x isin Sn+

(20)


f(x) =1

2Axminus b2 h(x) = δSn

+(x) forallx isin Sn

where middot denotes the Euclidean norm we can easily see that (20) is a special case of (1) withΩ = X = Sn

The SDLS instances included in this comparison are obtained via the above construction fromthe feasibility sets (after bringing them into standard form) of four classes of SDPs namely i)randomly generated SDPs as in [17] ii) SDP relaxations of frequency assignment problems (seefor example Subsection 24 in [2]) iii) SDP relaxations of binary integer quadratic problems (seefor example Section 7 in [19]) and iv) SDP relaxations of quadratic assignment problems (see forexample Section 7 in [19])

Figures 3 and 4 plot time and iteration performance profiles of all variants of Nesterovrsquos methodfor solving this collection of SDLS instances respectively Tables 3 and 4 report the time andnumber of iterations taken by each method respectively

Note that AA-R1 and AA-R2 outperform the other methods on most of the SDLS instancesFrom Figures 3 and 4 we can see that the aggressive restart scheme of AA-R1 performs slight betterthan the conservative one of AA-R2 on these instances

8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1

Performance Profiles (68 SDLS instances) tol=10-6

(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 3 Time performance profiles for solving SDLS instances with accuracy ε = 10minus6 On the left we include all the


1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 4 Number of iterations performance profiles for solving SDLS instances with accuracy ε = 10minus6 On the left we

include all the methods and on the right we include the three fastest variants namely AA-R1 AA-R2 and FISTA-R

9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1

Performance Profiles (58 NNLS instances) tol=10-6

(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 5 Time performance profiles for solving NNLS instances with accuracy ε = 10minus6 On the left we include all the


33 Numerical results for NNLSs

This subsection compares the performance of our methods AA-R1 and AA-R2 with the variants ofNesterovrsquos method listed at the beginning of this section on a class of NNLS instances randomlygenerated as in [9]

Given a matrix A isin Rmtimesn and a vector b isin Rm the NNLS problem is defined as

minxisinRn

1

2Axminus b2 x ge 0

(21)

where middot denotes the Euclidean normLetting f and h be defined as

f(x) =1

2Axminus b2 h(x) = δRn

+(x) forallx isin Rn

where Rn+ is the cone of nonnegative vectors in Rn we can easily see that (21) is a special case of(1) with Ω = X = Rn

Figures 5 and 6 plot the time and iteration performance profiles of all variants of Nesterovrsquosmethod for solving this collection of random NNLS instances respectively Tables 5 and 6 reportthe time and number of iterations taken by each method respectively

Note that AA-R2 outperforms the other methods on most of the NNLS instances where a solutionwith the required accuracy was found From Figures 5 and 6 we can see that the conservative restartscheme of AA-R2 performs better and is more robust than the aggressive one of AA-R1 on theseinstances Also we can see that the methods that do not incorporate a restart scheme are only ableto solve less than 10 of the NNLS instances while the ones that do incorporate a restart schemesolve at least 70 of them

10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 6 Number of iterations performance profiles for solving NNLS instances with accuracy ε = 10minus6 On the left we


4 Concluding remarks

We have observed in our computational experiments that AA-R2 quickly stops performing restartswhile AA-R1 periodically continues to perform restarts

Figures 7 and 8 plot the time and iteration performance profiles of all variants of Nesterovrsquosmethod on the collection of instances obtained by combining all the three instance classes describedin the previous sections From these plots we can see that the overall performances of AA-R1 andAA-R2 are very close to one another with AA-R2 turning out to be more robust as it solves moreproblems to the specified accuracy 10minus6 than any other of the variants tested We can see thatAA-R1 and AA-R2 are the two fastest variants among the six ones used in this benchmark in termsof both time and number of iterations

Finally our implementation of the AA method and its restarting variants can be found athttpwwwisyegatechedu~cod3COrtizsoftware

Acknowledgements

We want to thank Elizabeth W Karas for generously providing us with the code of the algorithmsin [7]

References

[1] A Beck and M Teboulle A fast iterative shrinkage-thresholding algorithm for linear inverseproblems SIAM J Img Sci 2(1)183ndash202 March 2009 URL httpdxdoiorg101137080716542 doi101137080716542

[2] S Burer R D C Monteiro and Y Zhang A computational study of a gradient-based log-barrier algorithm for a class of large-scale SDPs Mathematical Programming 95359ndash379 2003101007s10107-002-0353-7 URL httpdxdoiorg101007s10107-002-0353-7

11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1

Performance Profiles (186 instances) tol=10-6

(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 7 Time performance profiles for solving all the three problem classes with accuracy ε = 10minus6 On the left we include

all the methods and on the right we include the three fastest variants namely AA-R1 AA-R2 and FISTA-R

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R

Figure 8 Iterations performance profiles for solving all the three problem classes with accuracy ε = 10minus6 On the left we


12

[3] O Devolder F Glineur and Y Nesterov First-order methods of smooth convex optimiza-tion with inexact oracle Core discussion paper 201102 Center for Operations Research andEconometrics (CORE) Catholic University of Louvain December 2010

[4] E D Dolan and J J More Benchmarking optimization software with performanceprofiles January 2002 URL httpdxdoiorghttpdxdoiorgdoi101007

s101070100263 doihttpdxdoiorgdoi101007s101070100263

[5] D Goldfarb and K Scheinberg Fast first-order methods for composite convex opti-mization with line search Optimization-online preprint 3004 2011 URL httpwww

optimization-onlineorgDB_HTML2011043004html

[6] C C Gonzaga and E W Karas Fine tuning Nesterovrsquos steepest descent algorithm fordifferentiable convex programming Mathematical Programming pages 1ndash26 2012 URLhttpdxdoiorg101007s10107-012-0541-z doi101007s10107-012-0541-z

[7] Clovis C Gonzaga Elizabeth W Karas and Diane R Rossetto An optimal algorithm forconstrained differentiable convex optimization Optimization-online preprint 3053 pages 1ndash162011 URL httpwwwoptimization-onlineorgDB_HTML2011063053html

[8] G Lan Z Lu and RDC Monteiro Primal-dual first-order methods with iteration-complexityfor cone programming Mathematical Programming 126(1)1ndash29 2011

[9] M Merritt and Y Zhang Interior-point gradient method for large-scale totally nonnegativeleast squares problems Journal of Optimization Theory and Applications 126191ndash202 2005101007s10957-005-2668-z URL httpdxdoiorg101007s10957-005-2668-z

[10] R D C Monteiro and B F Svaiter An accelerated hybrid proximal extragradient method forconvex optimization and its implications to second-order methods SIAM Journal on Optimiza-tion 23(2)1092ndash1125 2013 URL httpepubssiamorgdoiabs101137110833786arXivhttpepubssiamorgdoipdf101137110833786 doi101137110833786

[11] Y Nesterov A method for unconstrained convex minimization problem with the rate of con-vergence O (1k2) Doklady AN SSSR 269(3)543ndash547 1983

[12] Y Nesterov Introductory Lectures on Convex Optimization A Basic Course volume 87 ofApplied Optimization Kluwer Academic Publishers 2004

[13] Y Nesterov Smooth minimization of non-smooth functions Mathematical programming103(1)127ndash152 2005

[14] Y Nesterov Dual extrapolation and its applications to solving variational inequalities andrelated problems Math Program 109(2-3 Ser B)319ndash344 2007

[15] Y Nesterov Gradient methods for minimizing composite objective function CORE 2007

[16] Brendan OrsquoDonoghue and Emmanuel Candes Adaptive restart for accelerated gradientschemes Foundations of Computational Mathematics pages 1ndash18 2013 URL http

dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3

[17] J Povh F Rendl and A Wiegele A boundary point method to solve semidefinite programsComputing 78277ndash286 2006 101007s00607-006-0182-2 URL httpdxdoiorg10

1007s00607-006-0182-2

13

[18] P Tseng On accelerated proximal gradient methods for convex-concave optimization submit-ted to SIAM Journal on Optimization 2008

[19] X-Y Zhao D Sun and K-C Toh A Newton-CG augmented lagrangian method forsemidefinite programming SIAM Journal on Optimization 20(4)1737ndash1765 2010 URLhttplinkaiporglinkSJE2017371 doi101137080718206

A Technical results use in Section 2

This section presents some technical results used in Section 2 In particular Lemma A3 shows thatthe Accelerated Class satisfies statements a)ndashc) of Proposition 21 at every iteration

Assume that the functions φ f and h and the set Ω satisfy C1ndashC4 of Section 2 in the followingresults We start by showing two technical lemmas that will be used to show the main result of thisappendix in Lemma A3

Lemma A1 Let A ge 0 λ gt 0 x0 y isin X and a closed convex function Γ X rarr R be given andassume that the triple (yAΓ) satisfies

Aφ(y) le minuisinXAΓ(u) +

1

2uminus x02 AΓ le Aφ (22)

Also define x isin X a gt 0 and x isin X as

x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)

a =λ+radicλ2 + 4λA

2 x =

A

A+ ay +

a

A+ ax (24)

Then for any proper closed convex function γ X rarr R minorizing φ we have

minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)

Proof Let an arbitrary u isin X be given Define

u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)

Note also that the definition of a in (24) implies that

λ =a2

A+ a

14

In view of (23) 22 the fact that γ le φ the functions Γ and γ are convex and the last threerelations we conclude that

aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2

The result now follows from (25) and the fact that the above inequality holds for any u isin X

Lemma A2 Let λ gt 0 ξ isin X and a proper closed convex function g X rarr (minusinfininfin] be givenand denote the optimal solution and optimal value of

minxisinX

λg(x) +

1

2xminus ξ2

(26)

by x and g respectively Then the affine function γ X rarr R defined as

γ(x) = g(x) +1

λ〈ξ minus x xminus x〉 forallx isin X

has the property that γ le g and the optimal solution and optimal value of

minxisinX

λγ(x) +

1

2xminus ξ2

(27)

is x and g respectively

Proof The optimality conditions for (26) imply that 0 isin λpartg(x) + xminus ξ or equivalently

1

λ(ξ minus x) isin partg(x)

and hence that γ le g by the definition of γ and the subgradient of a function Moreover x clearlysatisfies the optimality condition of (27) Hence the last claim of the proposition follows

The main result of this appendix is as follows

Lemma A3 Let A ge 0 0 lt λ le 1L x0 y isin X and an affine function Γ X rarr R be given andassume that the triple (yAΓ) satisfies (22) Compute x as in (23) a and x as in (24)

xΩ = ΠΩ(x) A+ = A+ a

and

y+ = (I + λparth)minus1(xminus λnablaf(xΩ)) = arg minuisinX

λh(u) +

1

2uminus x+ λnablaf(xΩ)2

15

and define the affine functions γΓ+ X rarr R as

γ(u) = f(xΩ) + 〈nablaf(xΩ) y+ minus xΩ〉+ h(y+) +1

λ〈xminus y+ uminus y+〉 forallu isin X

Γ+ =A

A+ aΓ +

a

A+ aγ

Then the triple (y+ A+Γ+) satisfies

A+φ(y+) le minuisinXA+Γ+(u) +

1

2uminus x02 A+Γ+ le A+φ (28)

radicA+ ge

radicA+

1

2

radicλ (29)

Moreover

x0 minusA+nablaΓ+ = xminus a

λ(xminus y+) (30)

Proof We first claim that γ le φ and

φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)

Define the function g X rarr R as

g(u) = f(xΩ) + 〈nablaf(xΩ) uminus xΩ〉+ h(u) forallu isin X

and note that assumption C3 implies

g(u) le φ(u) le g(u) +L

2uminus xΩ2 le g(u) +

1

2λuminus xΩ2 forallu isin X (32)

Then in view of the definition of y+ and λ and the above relation with u = y+ we have

minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2

2nablaf(xΩ)2 + λf(xΩ)

= λh(y+) +1

2y+ minus x+ λnablaf(xΩ)2 minus λ2


ge λh(y+) +1

2y+ minus xΩ + λnablaf(xΩ)2 minus λ2


= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)

where the first inequality is due to the non-expansiveness property of ΠΩ The claim follows fromthe above relation the definitions of γ and g and Lemma A2 with ξ = x and noting that in thiscase x = y+

Using the first inequality of (22) Lemma A1 and (31) we conclude that

minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16

Hence (28) follows from the previous relation the definitions of A+ and Γ+ the second inequalityin (22) and the previous claim Relation (29) can be shown by noting that the definition of aimplies

a ge λ

2+radicλA

which together with the definition of A+ implies that

A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2

Finally (30) follows immediately from the definition of γ A+ and Γ+ and the fact that x = x0 minusAnablaΓ

B Proof of Proposition 24

First note that the equivalence between a) and c) of Lemma 22 with Γ =summ

i=1 αiγi and χ =summi=1 αiφ(y) imply that the hard constraint in problem (16) is equivalent to∥∥∥∥∥

msumi=1

αinablaγi + y minus x0

∥∥∥∥∥2

+ 2

msumi=1

αi[φ(y)minus γi(y)] le y minus x02 (33)

Now assume that b) holds Then in view of (17) the point α(t) = (tα1 tαm) clearly satisfies(33) and hence is feasible for (16) for every t gt 0 Hence for a fixed xlowast isin Xlowast this conclusionimplies that

msumi=1

(tαi)φ(y) lemsumi=1

(tαi)γi(xlowast) +

1

2xlowast minus x02 le t

(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02

where the last inequality follows from the assumption that γi le φ and the non-negativity oftα1 tαm Dividing this expression by t(

summi=1 αi) gt 0 and letting t rarr infin we then conclude

that φ(y) minus φ(xlowast) le 0 and hence that y isin Xlowast Also the objective function of (16) at the pointα(t) converges to infinity as trarrinfin showing that b) implies a)

Now assume that a) holds Then there exists a sequence αk = (αk1 αkm) k ge 0 of

feasible solutions of problem (16) along which its objective function converges to infinity Considerthe sequence αk = αk(eTαk) k ge 0 where e is the vector of all ones Clearly αk has anaccumulation point α = (α1 αm) where 0 6= α ge 0 Moreover using the fact that αk satisfies(33) and the fact that γi(y) le φ(y) we easily see that (17) holds

17

C Tables

Table 1 Time comparison of the methods on CQP instances

Problem Time in seconds (accuracy less that 10minus6)

Instance n AA-R1 AA-R2 FISTA FISTA-R GKR-2 GKR-3QP n5 p2 5 01 01 01 01 99 05QP n5 p3 5 01 01 01 01 04 03QP n5 p4 5 01 01 03 02 163 24QP n5 p5 5 02 01 03 02 75 09QP n10 p2 10 01 01 01 01 47 05QP n10 p3 10 01 01 02 01 02 05QP n10 p4 10 02 02 03 02 995(12 -6) 13QP n10 p5 10 05 06 04 07 201 79QP n20 p2 20 01 01 01 01 248 03QP n20 p3 20 01 01 01 01 18 04QP n20 p4 20 02 01 03 02 801 42QP n20 p5 20 04 05 04 06 766 129QP n50 p2 50 01 01 01 01 448 03QP n50 p3 50 01 02 04 02 295 11QP n50 p4 50 02 02 02 01 58 3QP n50 p5 50 02 02 04 03 274(13 -6) 61QP n100 p2 100 01 01 02 01 462 05QP n100 p3 100 02 02 04 02 75 17QP n100 p4 100 02 02 08 02 1215 25QP n100 p5 100 03 03 07 03 1382(21 -6) 85QP n200 p2 200 01 01 02 01 282(16 -6) 08QP n200 p3 200 02 02 05 02 403 11QP n200 p4 200 02 02 07 03 33 25QP n200 p5 200 03 04 07 04 479 74QP n500 p2 500 01 02 04 01 1454 09QP n500 p3 500 03 03 12 03 108 24QP n500 p4 500 03 04 14 06 147 43QP n500 p5 500 04 05 21 1 237 128QP n700 p2 700 02 02 06 02 1199 11QP n700 p3 700 03 04 19 06 1159 31QP n700 p4 700 04 04 31 11 183 76QP n700 p5 700 14 11 54 11 191 196QP n900 p2 900 03 03 08 07 1200(22 -6) 17QP n900 p3 900 06 06 29 09 285 4QP n900 p4 900 09 1 38 14 264 97QP n900 p5 900 16 19 69 28 313 249QP n1K p2 1000 04 04 13 06 843(40 -6) 43QP n1K p3 1000 07 07 31 12 315 5QP n1K p4 1000 11 15 56 16 20 84QP n1K p5 1000 25 18 87 3 638 631QP n2K p2 2000 1 12 43 28 1120(31 -6) 39QP n2K p3 2000 18 29 103 37 886 112QP n2K p4 2000 3 31 178 6 1208 234QP n2K p5 2000 95 96 236 104 114 689QP n5K p2 5000 87 71 179 141 8630(32 -6) 202QP n5K p3 5000 203 19 485 291 4007 426QP n5K p4 5000 236 378 1291 467 4711 1614QP n5K p5 5000 478 403 1466 91 5976 5139QP n7K p2 7000 189 251 705 46 15448(44 -6) 652QP n7K p3 7000 314 378 1281 459 9239 855QP n7K p4 7000 259 292 167 637 10823(12 -6) 2642QP n7K p5 7000 644 1026 3385 1158 13071 734QP n9K p2 9000 254 403 589 683 30713(55 -6) 1006QP n9K p3 9000 641 542 219 120 20465 2873QP n9K p4 9000 95 86 4831 1292 1992 6695QP n9K p5 9000 1288 1633 5307 2927 21653 12039QP n10K p2 10000 1079 811 842 2404 35197(55 -6) 1761QP n10K p3 10000 84 643 2518 113 19116 2501QP n10K p4 10000 1738 1069 3654 2369 20897 7335QP n10K p5 10000 1418 2026 6885 4513 21909 1784

18

Table 2 Number of iterations comparison of the methods on CQP instances

Problem of iterations (accuracy less that 10minus6)Instance n AA-R1 AA-R2 FISTA FISTA-R GKR-2 GKR-3QP n5 p2 5 26 25 59 35 913 48QP n5 p3 5 41 38 53 35 18 20QP n5 p4 5 89 92 314 104 1491 195QP n5 p5 5 176 94 215 152 397 75QP n10 p2 10 39 44 94 40 315 37QP n10 p3 10 41 46 140 58 16 44QP n10 p4 10 130 95 254 146 2000(12 -6) 114QP n10 p5 10 462 483 425 536 579 907QP n20 p2 20 23 32 65 33 483 18QP n20 p3 20 47 40 61 46 49 36QP n20 p4 20 118 89 289 114 1132 407QP n20 p5 20 374 360 423 459 1041 1422QP n50 p2 50 27 36 58 33 525 29QP n50 p3 50 64 93 289 90 334 95QP n50 p4 50 126 132 353 164 873 416QP n50 p5 50 183 165 308 247 2000(13 -6) 398QP n100 p2 100 37 48 124 49 618 46QP n100 p3 100 88 93 338 118 948 176QP n100 p4 100 116 130 576 186 1726 256QP n100 p5 100 232 235 659 321 2000(21 -6) 861QP n200 p2 200 41 52 140 61 2000(16 -6) 62QP n200 p3 200 69 80 326 96 525 104QP n200 p4 200 134 139 584 183 391 219QP n200 p5 200 232 259 645 298 628 803QP n500 p2 500 57 74 151 77 1926 58QP n500 p3 500 110 134 406 125 939 155QP n500 p4 500 164 163 710 241 1384 307QP n500 p5 500 293 294 1017 377 1374 1063QP n700 p2 700 51 72 163 111 1598 63QP n700 p3 700 95 134 421 130 1567 178QP n700 p4 700 169 189 686 238 801 346QP n700 p5 700 336 346 929 462 1168 1008QP n900 p2 900 52 72 179 120 2000(22 -6) 62QP n900 p3 900 105 131 398 143 1148 137QP n900 p4 900 155 168 641 225 889 350QP n900 p5 900 320 372 998 431 1443 1086QP n1K p2 1000 58 68 197 83 2000(40 -6) 76QP n1K p3 1000 111 125 433 169 1206 170QP n1K p4 1000 164 192 721 258 857 379QP n1K p5 1000 329 377 1274 432 996 1289QP n2K p2 2000 63 76 187 85 2000(31 -6) 70QP n2K p3 2000 133 159 428 179 1113 172QP n2K p4 2000 160 185 769 277 1540 414QP n2K p5 2000 313 341 1102 438 1021 1076QP n5K p2 5000 83 87 215 103 2000(32 -6) 79QP n5K p3 5000 194 199 453 263 1240 169QP n5K p4 5000 290 299 825 388 1066 412QP n5K p5 5000 356 447 1265 522 1391 1161QP n7K p2 7000 85 94 219 102 2000(44 -6) 77QP n7K p3 7000 136 138 473 195 1341 176QP n7K p4 7000 190 226 871 275 2000(12 -6) 388QP n7K p5 7000 369 539 1385 611 1392 1264QP n9K p2 9000 85 93 225 221 2000(55 -6) 78QP n9K p3 9000 152 225 463 204 1378 176QP n9K p4 9000 299 343 916 337 1203 523QP n9K p5 9000 397 537 1379 673 1333 1259QP n10K p2 10000 204 210 225 225 2000(55 -6) 80QP n10K p3 10000 183 198 487 264 1422 186QP n10K p4 10000 236 258 918 393 1338 408QP n10K p5 10000 459 541 1473 727 1318 1316

19

Table 3 Time comparison of the methods on SDLS instances


Instance n AA-R1 AA-R2 FISTA FISTA-R GKR-2 GKR-3BIQ n21 m252 21 1 07 25 1 4029(14 -6) 468BIQ n31 m527 31 1 08 4 12 2486(14 -6) 476BIQ n41 m902 41 1 08 27 1 4816(14 -6) 478BIQ n51 m1377 51 1 1 28 11 2853(14 -6) 444BIQ n61 m1952 61 09 11 37 12 5372(14 -6) 421BIQ n71 m2627 71 08 12 31 12 5480(14 -6) 528BIQ n81 m3402 81 1 11 42 15 2663(14 -6) 404BIQ n91 m4277 91 1 12 32 15 2790(14 -6) 487BIQ n101 m5252 101 09 15 32 14 3604(14 -6) 567BIQ n121 m7502 121 11 11 63 17 6954(14 -6) 446BIQ n151 m11627 151 13 15 45 18 9399(14 -6) 645BIQ n201 m20502 201 27 21 65 31 7075(14 -6) 1579BIQ n251 m31877 251 27 31 159 4 10738(14 -6) 1448BIQ n501 m126252 501 163 139 523 177 63868(14 -6) 706FAP n52 m1378 52 04 05 05 04 56 37FAP n61 m1866 61 04 05 05 05 65 35FAP n65 m2145 65 05 06 06 05 112 39FAP n81 m3321 81 05 05 07 06 54 47FAP n84 m3570 84 05 06 06 05 59 44FAP n93 m4371 93 05 07 08 07 153 43FAP n98 m4851 98 05 07 08 06 8 44FAP n120 m7260 120 07 09 1 08 17 56FAP n174 m15225 174 09 12 23 11 211 87FAP n183 m14479 183 1 14 19 15 389 104FAP n252 m24292 252 21 26 32 23 398 209FAP n369 m26462 369 37 43 61 4 786 448FAP n2118 m322924 2118 3543 4669 1169 5505 9788 57222FAP n4110 m1154467 4110 25185 30392 51516 3187 66187 294406QAP n144 m10672 144 68 75 442(11 -6) 56 8073(25 -6) 5441QAP n196 m19619 196 12 107 241 111 11506(33 -6) 12998(23 -6)QAP n225 m25783 225 257 13 186 14 6892 1868QAP n256 m33302 256 215 217 248 157 22655(17 -6) 32892(12 -6)QAP n289 m42362 289 28 259 537 229 24962(40 -6) 17275QAP n324 m53161 324 34 361 985 276 32206(19 -6) 31244QAP n400 m80828 400 585 534 4408 526 48329(24 -6) 55714(13 -6)QAP n441 m98152 441 682 763 3399 59 42074 128517(18 -6)QAP n484 m118127 484 953 1139 4344 871 5387 11336QAP n625 m196598 625 2829 2107 14203 1594 141798(28 -6) 2809QAP n676 m229877 676 2123 2633 5431 2476 167150(27 -6) 35665QAP n729 m267217 729 243 2705 3766 2369 208523(18 -6) 200979QAP n784 m308936 784 333 3287 13844 2954 241971(20 -6) 230494QAP n900 m406843 900 4807 5167 16783 4641 235209 65589QAP n1225 m752813 1225 1258 11702 28739 1121 995560(13 -6) 150909QAP n1600 m1283258 1600 25127 28118 36101 30143 2077780(23 -6) 1715000(11 -6)RAND n300 m10K p4 300 13 123 319 161 8103 4903RAND n300 m20K p3 300 435 467 1396 681 29181(11 -6) 29874RAND n300 m25K p3 300 416 489 1577 1433 29811(15 -6) 26822(10 -6)RAND n400 m15K p4 400 157 272 968 277 8518 7944RAND n400 m30K p3 400 823 944 3622 168 49514 89622RAND n400 m40K p3 400 801 847 2732 178 97058(15 -6) 54652(13 -6)RAND n500 m20K p4 500 384 583 1179 472 22744 22929RAND n500 m30K p3 500 1001 1041 3064 1413 65945 5263RAND n500 m40K p3 500 1137 1196 5764 2477 8313 79793RAND n500 m50K p3 500 1929 1483 4119 6477 108546(11 -6) 100969(13 -6)RAND n600 m20K p4 600 368 376 1208 736 21758 23493RAND n600 m40K p3 600 2331 1995 8162 3213 12218 120374RAND n600 m50K p3 600 1689 1669 5718 445 135658 110125RAND n600 m60K p3 600 2188 265 7928 4116 174284(16 -6) 153569(12 -6)RAND n700 m50K p3 700 236 2395 11244 4021 206155 178222RAND n700 m70K p3 700 2588 241 1383 5922 196601 164151RAND n700 m90K p3 700 3527 3177 10751 9482 219676(13 -6) 225538(11 -6)RAND n800 m100K p3 800 4223 6411 15147 1024 303629 251201RAND n800 m110K p3 800 403 4051 22064 11525 306095(12 -6) 304263(11 -6)RAND n800 m70K p3 800 2899 3108 1803 7284 455688 218514RAND n900 m100K p3 900 5893 5875 16669 12789 486509 394385RAND n900 m140K p3 900 5525 577 27674 14655 468009(13 -6) 391168RAND n1K m100K p3 1000 568 5324 20743 16839 577752 43090RAND n1K m150K p3 1000 7684 7777 28102 18095 633968(12 -6) 617195

20

Table 4 Number of iterations comparison of the methods on SDLS instances

Problem of iterations (accuracy less that 10minus6)Instance n AA-R1 AA-R2 FISTA FISTA-R GKR-2 GKR-3BIQ n21 m252 21 79 67 251 90 2000(14 -6) 285BIQ n31 m527 31 79 67 251 90 2000(14 -6) 285BIQ n41 m902 41 79 67 251 90 2000(14 -6) 285BIQ n51 m1377 51 79 67 251 90 2000(14 -6) 285BIQ n61 m1952 61 68 73 251 90 2000(14 -6) 285BIQ n71 m2627 71 65 68 251 90 2000(14 -6) 285BIQ n81 m3402 81 68 73 251 90 2000(14 -6) 285BIQ n91 m4277 91 65 68 251 90 2000(14 -6) 285BIQ n101 m5252 101 67 72 251 90 2000(14 -6) 285BIQ n121 m7502 121 65 67 251 90 2000(14 -6) 285BIQ n151 m11627 151 68 73 251 90 2000(14 -6) 285BIQ n201 m20502 201 79 67 251 90 2000(14 -6) 285BIQ n251 m31877 251 64 66 251 90 2000(14 -6) 285BIQ n501 m126252 501 65 67 251 90 2000(14 -6) 285FAP n52 m1378 52 25 27 37 26 29 18FAP n61 m1866 61 21 25 38 26 32 18FAP n65 m2145 65 25 27 38 26 46 18FAP n81 m3321 81 23 23 37 26 23 19FAP n84 m3570 84 21 26 38 26 22 19FAP n93 m4371 93 20 29 39 26 43 18FAP n98 m4851 98 21 27 43 26 36 18FAP n120 m7260 120 24 28 43 26 38 18FAP n174 m15225 174 23 29 46 26 43 18FAP n183 m14479 183 21 26 44 26 41 19FAP n252 m24292 252 26 31 45 26 35 20FAP n369 m26462 369 24 27 47 26 40 20FAP n2118 m322924 2118 25 30 54 27 43 22FAP n4110 m1154467 4110 26 30 54 29 47 22QAP n144 m10672 144 202 223 2000(11 -6) 233 2000(25 -6) 1688QAP n196 m19619 196 207 220 674 244 2000(33 -6) 2000(23 -6)QAP n225 m25783 225 375 228 385 251 968 227QAP n256 m33302 256 236 253 382 259 2000(17 -6) 2000(12 -6)QAP n289 m42362 289 231 248 609 266 2000(40 -6) 1357QAP n324 m53161 324 240 262 1001 275 2000(19 -6) 1939QAP n400 m80828 400 263 258 1744 286 2000(24 -6) 2000(13 -6)QAP n441 m98152 441 254 269 1611 293 1455 2000(18 -6)QAP n484 m118127 484 271 290 1763 299 1462 284QAP n625 m196598 625 274 304 1740 316 2000(28 -6) 387QAP n676 m229877 676 283 297 885 323 2000(27 -6) 342QAP n729 m267217 729 283 305 527 327 2000(18 -6) 1646QAP n784 m308936 784 279 314 1626 333 2000(20 -6) 1587QAP n900 m406843 900 302 323 1055 343 1393 330QAP n1225 m752813 1225 321 333 931 368 2000(13 -6) 303QAP n1600 m1283258 1600 333 346 583 388 2000(23 -6) 2000(11 -6)RAND n300 m10K p4 300 102 118 337 167 387 323RAND n300 m20K p3 300 405 407 1411 690 2000(11 -6) 2000RAND n300 m25K p3 300 409 411 1579 1244 2000(15 -6) 2000(10 -6)RAND n400 m15K p4 400 86 151 319 160 315 304RAND n400 m30K p3 400 440 442 1176 939 1841 1866RAND n400 m40K p3 400 405 407 1515 1090 2000(15 -6) 2000(13 -6)RAND n500 m20K p4 500 109 149 372 153 485 459RAND n500 m30K p3 500 313 262 944 449 1379 1104RAND n500 m40K p3 500 353 355 1184 810 1747 1618RAND n500 m50K p3 500 442 423 1297 1179 2000(11 -6) 2000(13 -6)RAND n600 m20K p4 600 65 74 243 150 274 187RAND n600 m40K p3 600 366 368 1011 522 1508 1542RAND n600 m50K p3 600 323 325 1150 533 1755 1476RAND n600 m60K p3 600 401 403 1327 851 2000(16 -6) 2000(12 -6)RAND n700 m50K p3 700 327 330 989 558 1830 1570RAND n700 m70K p3 700 356 358 1187 788 1843 1521RAND n700 m90K p3 700 392 394 1420 1420 2000(13 -6) 2000(11 -6)RAND n800 m100K p3 800 383 385 1285 971 1995 1688RAND n800 m110K p3 800 378 380 1378 1182 2000(12 -6) 2000(11 -6)RAND n800 m70K p3 800 274 276 1073 739 1911 1453RAND n900 m100K p3 900 381 383 1190 849 1885 1831RAND n900 m140K p3 900 368 379 1429 893 2000(13 -6) 1712RAND n1K m100K p3 1000 280 282 1070 840 1657 1572RAND n1K m150K p3 1000 368 370 1351 895 2000(12 -6) 1972

21

Table 5 Time comparison of the methods on NNLS instances


Instance n AA-R1 AA-R2 FISTA FISTA-R GKR-2 GKR-3NNLS n200 m10K 200 08 09 72(23 -5) 14 257(38 -5) 219(33 -5)NNLS n200 m1K 200 1 06 22(12 -5) 07 233(19 -6) 168(17 -5)NNLS n200 m20K 200 147(22 -6) 08 62 11 464(32 -6) 590(48 -6)NNLS n200 m2K 200 02 02 19 03 147 147NNLS n200 m400 200 02 02 28 03 311(67 -6) 12NNLS n200 m40K 200 128 107 523(51 -6) 113 1009(13 -5) 875(32 -4)NNLS n200 m4K 200 17 14 30(14 -4) 18 330(11 -5) 405(33 -4)NNLS n200 m600 200 09 08 13(17 -5) 08 172(62 -6) 211(56 -5)NNLS n200 m6K 200 05 05 33(17 -5) 06 196(13 -5) 208(50 -6)NNLS n200 m800 200 08 08 24(77 -5) 1 201(80 -6) 237(54 -5)NNLS n200 m8K 200 18 54(11 -6) 50(31 -4) 32 365(16 -5) 317(41 -4)NNLS n400 m10K 400 39 26 137(30 -4) 48 358(11 -4) 597(19 -4)NNLS n400 m1K 400 04 04 27(21 -6) 05 264 230(70 -6)NNLS n400 m20K 400 85 77 293(37 -4) 76 1183(15 -5) 1056(30 -4)NNLS n400 m2K 400 14 15 40(14 -4) 15 412(65 -6) 410(11 -4)NNLS n400 m40K 400 617(11 -6) 119 557(79 -5) 232 2847(20 -5) 1017(19 -4)NNLS n400 m4K 400 09 08 71(79 -6) 08 442(19 -6) 443(55 -6)NNLS n400 m6K 400 87(10 -6) 12 66(45 -5) 19 247(20 -6) 352(41 -5)NNLS n400 m800 400 03 03 41(12 -6) 05 308 314(46 -6)NNLS n400 m8K 400 28 46 123(26 -4) 84(11 -6) 432(12 -4) 601(29 -4)NNLS n600 m10K 600 31 36 176(84 -5) 172(19 -6) 479(68 -6) 455(56 -5)NNLS n600 m20K 600 286(22 -6) 32 456 408(15 -6) 1228(18 -5) 930(49 -6)NNLS n600 m2K 600 11 11 33(61 -6) 18 322(26 -6) 287(50 -5)NNLS n600 m40K 600 795(46 -6) 651(26 -6) 882(58 -5) 105 2074(45 -5) 2636(33 -5)NNLS n600 m4K 600 18 12 54(66 -6) 13 410(20 -6) 310(35 -5)NNLS n600 m6K 600 12 11 135(75 -5) 17 395(46 -5) 555(60 -5)NNLS n600 m8K 600 16 18 180(30 -5) 25 448(28 -5) 512(18 -5)NNLS n800 m10K 800 24 32 339(22 -5) 35 1058(24 -6) 1011(13 -5)NNLS n800 m20K 800 95 68 388(66 -5) 129 1048(18 -5) 919(21 -4)NNLS n800 m2K 800 12 12 43(12 -4) 17 291(42 -5) 285(78 -5)NNLS n800 m40K 800 70 331 937(14 -4) 720(56 -6) 3600(34 -5) 2226(39 -4)NNLS n800 m4K 800 31 23 122(83 -5) 59 290(53 -6) 631(71 -5)NNLS n800 m6K 800 15 17 112(12 -5) 25 491(40 -6) 443(14 -5)NNLS n800 m8K 800 27 54 297(31 -6) 224(11 -6) 671 919(13 -5)NNLS n1K m10K 1000 233(18 -6) 35 312(47 -5) 51 1320(40 -5) 971(39 -5)NNLS n1K m20K 1000 114 112 597(75 -5) 500(11 -6) 2445(63 -5) 1668(71 -5)NNLS n1K m2K 1000 17 12 43(20 -4) 21 451(18 -5) 262(13 -4)NNLS n1K m40K 1000 1036(35 -6) 953(11 -6) 1435(31 -4) 1375(18 -6) 3319(31 -5) 2769(40 -4)NNLS n1K m4K 1000 19 16 76(25 -5) 22 594(47 -6) 355(35 -5)NNLS n1K m6K 1000 19 28 175(11 -5) 21 566(46 -6) 509(53 -6)NNLS n1K m8K 1000 72 52 187(52 -5) 65 823(13 -5) 742(14 -4)NNLS n2K m10K 2000 142 115 718(51 -5) 181 1397(78 -6) 1333(10 -4)NNLS n2K m20K 2000 892(12 -6) 247 1084(23 -5) 547 3712(23 -5) 2364(24 -5)NNLS n2K m40K 2000 2097(55 -6) 297 2217(22 -5) 60 5732(12 -5) 8008(12 -5)NNLS n2K m4K 2000 75 69 239(31 -5) 64 792(24 -5) 915(26 -5)NNLS n2K m6K 2000 197 128 272(84 -5) 198 1246(37 -5) 1314(62 -5)NNLS n2K m8K 2000 36 49 317(13 -5) 373(16 -6) 1278(82 -6) 826(11 -5)NNLS n4K m10K 4000 948(16 -6) 237 881(49 -5) 315 4684(41 -5) 3511(36 -5)NNLS n4K m20K 4000 536 1573(12 -6) 1808(81 -5) 1859(13 -6) 5613(47 -5) 3813(58 -5)NNLS n4K m40K 4000 4720(16 -6) 4522(16 -6) 4359(83 -5) 5333(29 -6) 15585(13 -5) 12214(14 -4)NNLS n4K m8K 4000 219 165 673(88 -5) 289 2098(21 -5) 1453(38 -5)NNLS n6K m20K 6000 3119(25 -6) 797 3705(76 -5) 2679(12 -6) 10957(99 -6) 10641(65 -5)NNLS n6K m40K 6000 6434(30 -6) 222 6036(12 -4) 6201(12 -6) 22547(55 -5) 25213(92 -5)NNLS n8K m20K 8000 1745 1001 3567(66 -5) 1989 14776(37 -5) 11065(73 -5)NNLS n8K m40K 8000 1096 5922(23 -6) 10685(10 -5) 6377(14 -6) 32092(62 -6) 25564(20 -5)NNLS n10K m20K 10000 4692(21 -6) 1622 5307(11 -4) 4783(30 -6) 16233(59 -5) 14105(82 -5)NNLS n10K m40K 10000 7087(26 -6) 7127(24 -6) 10394(70 -5) 7856(28 -6) 32316(26 -5) 26041(13 -4)NNLS n20K m40K 20000 19521(12 -6) 21940(18 -6) 17922(75 -5) 20442(34 -6) 66707(39 -5) 46488(69 -5)

22

Table 6 Number of iterations comparison of the methods on NNLS instances

Problem of iterations (accuracy less that 10minus6)Instance n AA-R1 AA-R2 FISTA FISTA-R GKR-2 GKR-3NNLS n200 m10K 200 325 330 2000(23 -5) 553 2000(38 -5) 2000(33 -5)NNLS n200 m1K 200 836 447 2000(12 -5) 689 2000(19 -6) 2000(17 -5)NNLS n200 m20K 200 2000(22 -6) 138 1246 181 2000(32 -6) 2000(48 -6)NNLS n200 m2K 200 132 141 1240 199 1046 885NNLS n200 m400 200 168 171 1866 251 2000(67 -6) 1360NNLS n200 m40K 200 936 738 2000(51 -6) 824 2000(13 -5) 2000(32 -4)NNLS n200 m4K 200 979 758 2000(14 -4) 1147 2000(11 -5) 2000(33 -4)NNLS n200 m600 200 809 641 2000(17 -5) 800 2000(62 -6) 2000(56 -5)NNLS n200 m6K 200 230 246 2000(17 -5) 323 2000(13 -5) 2000(50 -6)NNLS n200 m800 200 623 627 2000(77 -5) 808 2000(80 -6) 2000(54 -5)NNLS n200 m8K 200 720 2000(11 -6) 2000(31 -4) 1320 2000(16 -5) 2000(41 -4)NNLS n400 m10K 400 647 669 2000(30 -4) 943 2000(11 -4) 2000(19 -4)NNLS n400 m1K 400 291 275 2000(21 -6) 370 1702 2000(70 -6)NNLS n400 m20K 400 775 775 2000(37 -4) 866 2000(15 -5) 2000(30 -4)NNLS n400 m2K 400 860 865 2000(14 -4) 1049 2000(65 -6) 2000(11 -4)NNLS n400 m40K 400 2000(11 -6) 391 2000(79 -5) 556 2000(20 -5) 2000(19 -4)NNLS n400 m4K 400 344 363 2000(79 -6) 404 2000(19 -6) 2000(55 -6)NNLS n400 m6K 400 2000(10 -6) 387 2000(45 -5) 673 2000(20 -6) 2000(41 -5)NNLS n400 m800 400 224 217 2000(12 -6) 321 1908 2000(46 -6)NNLS n400 m8K 400 744 706 2000(26 -4) 2000(11 -6) 2000(12 -4) 2000(29 -4)NNLS n600 m10K 600 432 444 2000(84 -5) 2000(19 -6) 2000(68 -6) 2000(56 -5)NNLS n600 m20K 600 2000(22 -6) 194 1855 2000(15 -6) 2000(18 -5) 2000(49 -6)NNLS n600 m2K 600 612 555 2000(61 -6) 718 2000(26 -6) 2000(50 -5)NNLS n600 m40K 600 2000(46 -6) 2000(26 -6) 2000(58 -5) 392 2000(45 -5) 2000(33 -5)NNLS n600 m4K 600 379 387 2000(66 -6) 466 2000(20 -6) 2000(35 -5)NNLS n600 m6K 600 304 308 2000(75 -5) 461 2000(46 -5) 2000(60 -5)NNLS n600 m8K 600 288 336 2000(30 -5) 486 2000(28 -5) 2000(18 -5)NNLS n800 m10K 800 250 260 2000(22 -5) 395 2000(24 -6) 2000(13 -5)NNLS n800 m20K 800 518 436 2000(66 -5) 626 2000(18 -5) 2000(21 -4)NNLS n800 m2K 800 572 577 2000(12 -4) 830 2000(42 -5) 2000(78 -5)NNLS n800 m40K 800 1431 768 2000(14 -4) 2000(56 -6) 2000(34 -5) 2000(39 -4)NNLS n800 m4K 800 769 774 2000(83 -5) 1133 2000(53 -6) 2000(71 -5)NNLS n800 m6K 800 348 349 2000(12 -5) 499 2000(40 -6) 2000(14 -5)NNLS n800 m8K 800 417 444 2000(31 -6) 2000(11 -6) 1760 2000(13 -5)NNLS n1K m10K 1000 2000(18 -6) 352 2000(47 -5) 372 2000(40 -5) 2000(39 -5)NNLS n1K m20K 1000 411 433 2000(75 -5) 2000(11 -6) 2000(63 -5) 2000(71 -5)NNLS n1K m2K 1000 749 500 2000(20 -4) 958 2000(18 -5) 2000(13 -4)NNLS n1K m40K 1000 2000(35 -6) 2000(11 -6) 2000(31 -4) 2000(18 -6) 2000(31 -5) 2000(40 -4)NNLS n1K m4K 1000 345 320 2000(25 -5) 533 2000(47 -6) 2000(35 -5)NNLS n1K m6K 1000 263 278 2000(11 -5) 356 2000(46 -6) 2000(53 -6)NNLS n1K m8K 1000 718 547 2000(52 -5) 870 2000(13 -5) 2000(14 -4)NNLS n2K m10K 2000 705 580 2000(51 -5) 889 2000(78 -6) 2000(10 -4)NNLS n2K m20K 2000 2000(12 -6) 414 2000(23 -5) 521 2000(23 -5) 2000(24 -5)NNLS n2K m40K 2000 2000(55 -6) 286 2000(22 -5) 449 2000(12 -5) 2000(12 -5)NNLS n2K m4K 2000 804 715 2000(31 -5) 897 2000(24 -5) 2000(26 -5)NNLS n2K m6K 2000 828 850 2000(84 -5) 1193 2000(37 -5) 2000(62 -5)NNLS n2K m8K 2000 262 313 2000(13 -5) 2000(16 -6) 2000(82 -6) 2000(11 -5)NNLS n4K m10K 4000 2000(16 -6) 543 2000(49 -5) 826 2000(41 -5) 2000(36 -5)NNLS n4K m20K 4000 668 2000(12 -6) 2000(81 -5) 2000(13 -6) 2000(47 -5) 2000(58 -5)NNLS n4K m40K 4000 2000(16 -6) 2000(16 -6) 2000(83 -5) 2000(29 -6) 2000(13 -5) 2000(14 -4)NNLS n4K m8K 4000 580 591 2000(88 -5) 929 2000(21 -5) 2000(38 -5)NNLS n6K m20K 6000 2000(25 -6) 490 2000(76 -5) 2000(12 -6) 2000(99 -6) 2000(65 -5)NNLS n6K m40K 6000 2000(30 -6) 695 2000(12 -4) 2000(12 -6) 2000(55 -5) 2000(92 -5)NNLS n8K m20K 8000 964 723 2000(66 -5) 953 2000(37 -5) 2000(73 -5)NNLS n8K m40K 8000 334 2000(23 -6) 2000(10 -5) 2000(14 -6) 2000(62 -6) 2000(20 -5)NNLS n10K m20K 10000 2000(21 -6) 744 2000(11 -4) 2000(30 -6) 2000(59 -5) 2000(82 -5)NNLS n10K m40K 10000 2000(26 -6) 2000(24 -6) 2000(70 -5) 2000(28 -6) 2000(26 -5) 2000(13 -4)NNLS n20K m40K 20000 2000(12 -6) 2000(18 -6) 2000(75 -5) 2000(34 -6) 2000(39 -5) 2000(69 -5)

23

Introduction

An accelerated first-order method

Numerical results

Numerical results for random CQPs

Numerical results for SDLSs

Numerical results for NNLSs

Concluding remarks

Technical results use in Section 2

Proof of Proposition 24

Tables

the acceleration parameters at every iteration by solving a simple two-variable convex quadrati-cally constrained linear program Moreover its iteration cost is comparable to or better than thatof the aforementioned accelerated methods since it only computes a single resolvent (or proximalsubproblem) at every iteration

For the purpose of our computational experiments we have implemented the new acceleratedvariant endowed with two restarting schemes in order to improve its practical performance Thefirst restarting scheme is an aggressive one which restarts the method from the last iterate wheneverthe objective function increases (also proposed in [16]) The second restarting scheme is a moreconservative (and more robust) one which restarts the method whenever the objective functionincreases and the number of iterations at that point is sufficiently large Finally computationalresults are presented on several conic quadratic programming instances showing that the new ac-celerated variant endowed with either one of the restarting schemes is faster and more robust thanother existing Nesterovrsquos variants

Our paper is organized as follows Section 2 presents a class of accelerated first-order methodsand corresponding iteration-complexity results for solving composite convex optimization problemsMoreover this section proposes a specific instance of the latter class which adaptively chooses thesequence of acceleration parameters so as to greedily minimize an upper bound on the primal gapSection 3 presents computational results comparing the latter instance (endowed with one of theaforementioned restart schemes) with other existing variants of Nesterovrsquos method Finally Section4 presents some final remarks

2 An accelerated first-order method

This section introduces an accelerated first-order method presented in [10] for solving compositeconvex optimization problems First we propose a method with general acceleration parameterssatisfying certain conditions which still guarantee an optimal iteration-complexity Next for thepurpose of improving the practical performance of the method we propose a specific way for choosingthese parameters that greedily minimizes an upper bound on the primal gap

Let R denote the set of real numbers and define R = R cup plusmninfin Also let X denote a finitedimensional real inner product space with inner product and induced norm denoted by 〈middot middot〉 and middot respectively Our problem of interest is the composite convex optimization problem

φlowast = minxisinX

φ(x) = f(x) + h(x) (1)

where

C1 h X rarr R is a proper closed convex function

C2 f is a real-valued function which is differentiable and convex on a closed convex set

Ω supe domh = x isin X h(x) ltinfin

C3 nablaf is L-Lipschitz continuous on Ω for some L gt 0 ie

nablaf(xprime)minusnablaf(x) le Lxprime minus x forallx xprime isin Ω

C4 the set Xlowast of optimal solutions of (1) is non-empty

There are many accelerated variants of Nesterovrsquos method for solving (1) under assumptions C1ndashC4 In this paper we are interested in studying the properties of the following accelerated variantof Nesterovrsquos method whose statement uses the projection operator ΠΩ X rarr Ω onto Ω defined as

ΠΩ(x) = arg minxisinΩxminus x forallx isin X

2




ak =λ+


2 (2)

xk =Akminus1


akAkminus1 + ak


yk = arg minxisinX

λh(x) +

1


(4)







1

2xminus x02

(7)


3) compute


Γk =Aprimekminus1



γk (9)





AkΓk(x) +

1

2xminus x02

(11)









and (11)

3






c) Ak ge λk24

d) there holds


2Akle 2d2

0

λk2(12)






1

2d2

0



imply that


(1


)














4





max Aprime + aprime



Aprime aprime ge 0

(15)



maxsumm

i=1 αi

st infxisinXsumm


02gesumm

i=1 αiφ(y)

α1 ge 0 αm ge 0

(16)




msumi=1






3 Numerical results






5







(yk minus 1

Lnablaf(yk)

)∥∥∥∥ le 10minus6 (18)








6

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn


(19)








7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables




ak =λ+


2 (2)

xk =Akminus1


akAkminus1 + ak


yk = arg minxisinX

λh(x) +

1


(4)







1

2xminus x02

(7)


3) compute


Γk =Aprimekminus1



γk (9)





AkΓk(x) +

1

2xminus x02

(11)









and (11)

3






c) Ak ge λk24

d) there holds


2Akle 2d2

0

λk2(12)






1

2d2

0



imply that


(1


)














4





max Aprime + aprime



Aprime aprime ge 0

(15)



maxsumm

i=1 αi

st infxisinXsumm


02gesumm

i=1 αiφ(y)

α1 ge 0 αm ge 0

(16)




msumi=1






3 Numerical results






5







(yk minus 1

Lnablaf(yk)

)∥∥∥∥ le 10minus6 (18)








6

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn


(19)








7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables






c) Ak ge λk24

d) there holds


2Akle 2d2

0

λk2(12)






1

2d2

0



imply that


(1


)














4





max Aprime + aprime



Aprime aprime ge 0

(15)



maxsumm

i=1 αi

st infxisinXsumm


02gesumm

i=1 αiφ(y)

α1 ge 0 αm ge 0

(16)




msumi=1






3 Numerical results






5







(yk minus 1

Lnablaf(yk)

)∥∥∥∥ le 10minus6 (18)








6

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn


(19)








7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables





max Aprime + aprime



Aprime aprime ge 0

(15)



maxsumm

i=1 αi

st infxisinXsumm


02gesumm

i=1 αiφ(y)

α1 ge 0 αm ge 0

(16)




msumi=1






3 Numerical results






5







(yk minus 1

Lnablaf(yk)

)∥∥∥∥ le 10minus6 (18)








6

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn


(19)








7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables







(yk minus 1

Lnablaf(yk)

)∥∥∥∥ le 10minus6 (18)








6

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn


(19)








7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 30

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn


(19)








7

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R




Ax = b x isin Sn+


minxisinSn

1


(20)


f(x) =1







8

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

1 2 4 8 16 32 64 1280

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



9

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

1 2 4 8 16 32 640

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 250

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R






minxisinRn

1

2Axminus b2 x ge 0

(21)


f(x) =1






10

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

1 2 4 80

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 20

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R







Acknowledgements


References



11

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

1 2 4 8 16 32 64 128 256 512 10240

01

02

03

04

05

06

07

08

09

1


(time)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 2 40

01

02

03

04

05

06

07

08

09

1


(time)


of pro

ble

ms

AA-R1

AA-R2

FISTA-R



1 2 4 8 16 320

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA

FISTA-R

GKR-2

GKR-3

1 15 2 25 3 350

01

02

03

04

05

06

07

08

09

1


(iterations)


o

f pro

ble

ms

AA-R1

AA-R2

FISTA-R



12

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

















dxdoiorg101007s10208-013-9150-3 doi101007s10208-013-9150-3


1007s00607-006-0182-2

13








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables








1



x = arg minuisinX

AΓ(u) +

1

2uminus x02

(23)


2 x =

A

A+ ay +

a

A+ ax (24)


minuisinX

aγ(u) +AΓ(u) +

1

2uminus u02

ge (A+ a)

θ

λ

where

θ = minuisinX

λγ(u) +

1

2uminus x2

(25)


u =Ay + au

A+ a

and note that

uminus x =

(a

A+ a

)2

(uminus x)


λ =a2

A+ a

14


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables


aγ(u) +AΓ(u) +1

2uminus x02

ge aγ(u) +1

2uminus x2 + min

uisinX

AΓ(u) +

1

2uminus x02

ge aγ(u) +

1

2uminus x2 +Aγ(y)

ge (A+ a)γ

(Ay + au

A+ a

)+

1

2uminus x2

= (A+ a)

[γ(u) +

(A+ a)

2a2uminus x2

]ge (A+ a) min

uprimeisinX

γ(uprime) +

1

2λuprime minus x2



minxisinX

λg(x) +

1

2xminus ξ2

(26)


γ(x) = g(x) +1



minxisinX

λγ(x) +

1

2xminus ξ2

(27)



1






and


λh(u) +

1


15




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables




Γ+ =A

A+ aΓ +

a

A+ aγ



1


radicA+ ge

radicA+

1

2

radicλ (29)

Moreover


λ(xminus y+) (30)


φ(y+) le minuisinX

γ(u) +

1

2λuminus x2

(31)






1



minuisinX

λg(u) +

1

2uminus x2

= min

uisinX

λh(u) +

1


minus λ2


= λh(y+) +1



ge λh(y+) +1



= λ

[g(y+) +

1

2λy+ minus xΩ2

]ge λφ(y+)



minuisinX

aγ(u) +AΓ(u) +

1

2uminus x02

ge A+ a

λminuisinX

λγ(u) +

1

2uminus x2

ge (A+ a)φ(y+)

16


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables


a ge λ

2+radicλA


A+ ge A+

(λ

2+radicλA

)ge(radic

A+1

2

radicλ

)2





msumi=1


∥∥∥∥∥2

+ 2

msumi=1



msumi=1



1


(msumi=1

αi

)φ(xlowast) +

1

2xlowast minus x02






17

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables

C Tables




18



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables



19




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables




20



21




22



23

Introduction


Numerical results




Concluding remarks



Tables



21




22



23

Introduction


Numerical results




Concluding remarks



Tables




22



23

Introduction


Numerical results




Concluding remarks



Tables



23

Introduction


Numerical results




Concluding remarks



Tables

an adaptive accelerated first-order method for convex optimization … · an adaptive accelerated...

Documents