automated inference with adaptive batchesproceedings.mlr.press/v54/de17a/de17a.pdf · automated...

Automated Inference with Adaptive Batches

Soham De, Abhay Yadav, David Jacobs and Tom Goldstein

Department of Computer Science, University of Maryland, College Park, MD, USA 20742.{sohamde, jaiabhay, djacobs, tomg}@cs.umd.edu

Abstract

Classical stochastic gradient methods for op-timization rely on noisy gradient approxima-tions that become progressively less accurateas iterates approach a solution. The largenoise and small signal in the resulting gradi-ents makes it di�cult to use them for adap-tive stepsize selection and automatic stop-ping. We propose alternative “big batch”SGD schemes that adaptively grow the batchsize over time to maintain a nearly constantsignal-to-noise ratio in the gradient approx-imation. The resulting methods have simi-lar convergence rates to classical SGD, anddo not require convexity of the objective.The high fidelity gradients enable automatedlearning rate selection and do not requirestepsize decay. Big batch methods are thuseasily automated and can run with little orno oversight.

1 Introduction

We are interested in problems of the form

minx2X

`(x) :=

(Ez⇠p[f(x; z)],

1N

PN

i=1 f(x; zi),(1)

where {zi

} is a collection of data drawn from a proba-bility distribution p. We assume that ` and f are dif-ferentiable, but possibly non-convex, and domain Xis convex. In typical applications, each term f(x; z)measures how well a model with parameters x fits oneparticular data observation z. The expectation over zmeasures how well the model fits the entire corpus ofdata on average.

Proceedings of the 20th International Conference on Artifi-cial Intelligence and Statistics (AISTATS) 2017, Fort Laud-erdale, Florida, USA. JMLR: W&CP volume 54. Copy-right 2017 by the author(s).

When N is large (or even infinite), it becomes in-tractable to exactly evaluate `(x) or its gradientr`(x),which makes classical gradient methods impossible. Insuch situations, the method of choice for minimizing(1) is the stochastic gradient descent (SGD) algorithm(Robbins and Monro, 1951). On iteration t, SGD se-lects a batch B ⇢ {z

i

} of data uniformly at random,and then computes

xt+1 = xt � ↵trx`B(xt), (2)

where `B(x) =1

|B|X

z2Bf(x; z),

where ↵t

denotes the stepsize used on the t-th itera-tion. Note that EB[rx`B(xt)] = rx`(xt), and so thecalculated gradient r

x

`B(xt) can be interpreted as a“noisy” approximation to the true gradient.

Because the gradient approximations are noisy, thestepsize ↵

t

must vanish as t ! 1 to guarantee con-vergence of the method. Typical stepsize rules requirethe user to find the optimal decay rate schedule, whichusually requires an expensive grid search over di↵erentpossible parameter values.

In this paper, we consider a “big batch” strategy forSGD. Rather than letting the stepsize vanish over timeas the iterates approach a minimizer, we let the mini-batch B adaptively grow in size to maintain a constantsignal-to-noise ratio of the gradient approximation.This prevents the algorithm from getting overwhelmedwith noise, and guarantees convergence with an ap-propriate constant stepsize. Recent results (Keskaret al., 2016) have shown that large fixed batch sizesfail to find good minimizers for non-convex problemslike deep neural networks. Adaptively increasing thebatch size over time overcomes this limitation: intu-itively, in the initial iterations, the increased stochas-ticity (corresponding to smaller batches) can help landthe iterates near a good minimizer, and larger batcheslater on can increase the speed of convergence towardsthis minimizer.

Using this batching strategy, we show that we can keep

An extended version of this work is in De et al. (2016).


the stepsize constant, or let it adapt using a simpleArmijo backtracking line search, making the methodcompletely adaptive with no user-defined parameters.We also derive an adaptive stepsize method based onthe Barzilai and Borwein (1988) curvature estimatethat fully automates the big batch method, while em-pirically enjoying a faster convergence rate than theArmijo backtracking line search.

Big batch methods that adaptively grow the batch sizeover time have several potential advantages over con-ventional small-batch SGD:

• Big batch methods don’t require the user tochoose stepsize decay parameters. Larger batchsizes with less noise enable easy estimation of theaccuracy of the approximate gradient, making itstraightforward to adaptively scale up the batchsize and maintain fast convergence.

• Backtracking line search tends to work very wellwhen combined with big batches, making themethods completely adaptive with no parameters.A nearly constant signal-to-noise ratio also en-ables us to define an adaptive stepsize methodbased on the Barzilai-Borwein curvature esti-mate, that performs better empirically on a rangeof convex problems than the backtracking linesearch.

• Higher order methods like stochastic L-BFGS typ-ically require more work per iteration than sim-ple SGD. When using big batches, the overhead ofmore complex methods like L-BFGS can be amor-tized over more costly gradient approximations.Furthermore, better Hessian approximations canbe computed using less noisy gradient terms.

• For a restricted class of non-convex problems(functions satisfying the Polyak- Lojasiewicz In-equality), the per-iteration complexity of bigbatch SGD is linear and the approximate gradi-ents vanish as the method approaches a solution,which makes it easy to define automated stoppingconditions. In contrast, small batch SGD exhibitssub-linear convergence, and the noisy gradientsare not usable as a stopping criterion.

• Big batch methods are much more e�cientthan conventional SGD in massively paral-lel/distributed settings. Bigger batches performmore computation between parameter updates,and thus allow a much higher ratio of computa-tion to communication.

For the reasons above, big batch SGD is potentiallymuch easier to automate and requires much less useroversight than classical small batch SGD.

1.1 Related work

In this paper, we focus on automating stochastic opti-mization methods by reducing the noise in SGD. Wedo this by adaptively growing the batch size to controlthe variance in the gradient estimates, maintaining anapproximately constant signal-to-noise ratio, leadingto automated methods that do not require vanishingstepsize parameters. While there has been some workon adaptive stepsize methods for stochastic optimiza-tion (Mahsereci and Hennig, 2015; Schaul et al., 2013;Tan et al., 2016; Kingma and Ba, 2014; Zeiler, 2012),the methods are largely heuristic without any kind oftheoretical guarantees or convergence rates. The workin Tan et al. (2016) was a first step towards provableautomated stochastic methods, and we explore in thisdirection to show provable convergence rates for theautomated big batch method.

While there has been relatively little work in prov-able automated stochastic methods, there has beenrecent interest in methods that control gradient noise.These methods mitigate the e↵ects of vanishing step-sizes, though choosing the (constant) stepsize still re-quires tuning and oversight. There have been a fewpapers in this direction that use dynamically increas-ing batch sizes. In Friedlander and Schmidt (2012),the authors propose to increase the size of the batchby a constant factor on every iteration, and prove lin-ear convergence in terms of the iterates of the algo-rithm. In Byrd et al. (2012), the authors propose anadaptive strategy for growing the batch size; however,the authors do not present a theoretical guarantee forthis method, and instead prove linear convergence fora continuously growing batch, similar to Friedlanderand Schmidt (2012).

Variance reduction (VR) SGD methods use an errorcorrection term to reduce the noise in stochastic gra-dient estimates. The methods enjoy a provably fasterconvergence rate than SGD and have been shown tooutperform SGD on convex problems (Defazio et al.,2014a; Johnson and Zhang, 2013; Schmidt et al., 2013;Defazio et al., 2014b), as well as in parallel (Reddiet al., 2015) and distributed settings (De and Gold-stein, 2016). A caveat, however, is that these methodsrequire either extra storage or full gradient computa-tions, both limiting factors when the dataset is verylarge. In a recent paper (Harikandeh et al., 2015), theauthors propose a growing batch strategy for a VRmethod that enjoys the same convergence guarantees.However, as mentioned above, choosing the constantstepsize still requires tuning. Another conceptually re-lated approach is importance sampling, i.e., choosingtraining points such that the variance in the gradientestimates is reduced (Bouchard et al., 2015; Csiba andRichtárik, 2016; Needell et al., 2014).


2 Big Batch SGD

2.1 Preliminaries and motivation

Classical stochastic gradient methods thrive when thecurrent iterate is far from optimal. In this case, asmall amount of data is necessary to find a descentdirection, and optimization progresses e�ciently. Asxt

starts approaching the true solution x?, however,noisy gradient estimates frequently fail to produce de-scent directions and do not reliably decrease the ob-jective. By choosing larger batches with less noise, wemay be able to maintain descent directions on each it-eration and uphold fast convergence. This observationmotivates the proposed “big batch” method. We nowexplore this idea more rigorously.

To simplify notation, we hereon use r` to denote rx

`.We wish to show that a noisy gradient approximationproduces a descent direction when the noise is compa-rable in magnitude to the true gradient.

Lemma 1. A su�cient condition for �r`B(x) to bea descent direction is

kr`B(x)�r`(x)k2 < kr`B(x)k2.

This is a standard result in stochastic optimiza-tion (see the supplemental). In words, if the errorkr`B(x) � r`(x)k2 is small relative to the gradientkr`B(x)k2, the stochastic approximation is a descentdirection. But how big is this error and how large doesa batch need to be to guarantee this condition? By theweak law of large numbers1

E[kr`B(x)�r`(x)k2] =1

|B|E[krf(x; z)�r`(x)k2]

=1

|B| TrVarzrf(x; z),

and so we can estimate the error of a stochastic gra-dient if we have some knowledge of the variance ofrf(x; z). In practice, this variance could be estimatedusing the sample variance of a batch {rf(x; z)}

z2B.However, we would like some bounds on the magni-tude of this gradient to show that it is well-behaved,and also to analyze worst-case convergence behavior.To this end, we make the following assumption.

Assumption 1. We assume f has Lz

-Lipschitz de-

pendence on data z, i.e., given two data points z1, z2 ⇠p(z), we have: krf(x; z1)�rf(x; z2)k Lzkz1�z2k.

Under this assumption, we can bound the error of thestochastic gradient. The bound is uniform with re-spect to x, which makes it rather useful in analyzingthe convergence rate for big batch methods.

1We assume the random variablerf(x; z) is measurableand has bounded second moment. These conditions will beguaranteed by the hypothesis of Theorem 1.

Theorem 1. Given the current iterate x, suppose As-sumption 1 holds and that the data distribution p hasbounded second moment. Then the estimated gradient

r`B(x) has variance bounded by

EBkr`B(x)�r`(x)k2 := TrVarB(r`B(x))

4L2z

TrVarz

(z)

|B| ,

where z ⇠ p(z). Note the bound is uniform in x.

The proof is in the supplemental. Note that, using afinite number of samples, one can easily approximatethe quantity Var

z

(z) that appears in our bound.

2.2 A template for big batch SGD

Theorem 1 and Lemma 1 together suggest that weshould expect d = �r`B to be a descent directionreasonably often provided

✓2kr`B(x)k2 �1

|B| [TrVarz(rf(x; zi))], (3)

or ✓2kr`B(x)k2 �4L2

z

TrVarz

(z)

|B| ,

for some ✓ < 1. Big batch methods capitalize on thisobservation.

On each iteration t, starting from a point xt

, the bigbatch method performs the following steps:

1. Estimate the variance TrVarz

[rf(xt

; z)], and abatch size K large enough that

✓2Ekr`Bt(xt)k2 � Ekr`Bt(xt)�r`(xt)k2

=1

KTrVar

z

f(xt

; z), (4)

where ✓ 2 (0, 1) and Bt

denotes the selected batchon the t-th iteration with |B| = K.

2. Choose a stepsize ↵t

.

3. Perform the update xt+1 = xt � ↵tr`Bt(xt).

Clearly, we have a lot of latitude in how to implementthese steps using di↵erent variance estimators and dif-ferent stepsize strategies. In the following section, weshow that, if condition (4) holds, then linear conver-gence can be achieved using an appropriate constantstepsize. In subsequent sections, we address the issueof how to build practical big batch implementationsusing automated variance and stepsize estimators thatrequire no user oversight.


3 Convergence Analysis

We now present convergence bounds for big batch SGDmethods (5). We study stochastic gradient updates ofthe form

xt+1 = xt � ↵r`Bt(xt) = xt � ↵(r`(xt) + et), (5)

where et

= r`Bt(xt)�r`(xt), and EB[et] = 0. Let usalso define g

t

= r`(xt

) + et

.

Before we present our results, we first state two as-sumptions about the loss function `(x).

Assumption 2. We assume that the objective func-

tion ` has L-Lipschitz gradients:

`(x) `(y) +r`(y)T (x� y) + L2kx� yk2.

This is a standard smoothness assumption used widelyin the optimization literature. Note that a conse-quence of Assumption 2 is the property:

kr`(x)�r`(y)k Lkx� yk.

Assumption 3. We also assume that the objective

function ` satisfies the Polyak- Lojasiewicz Inequality:

kr`(x)k2 � 2µ(`(x)� `(x?)).

Note that this inequality does not require ` to be con-vex, and is, in fact, a weaker assumption than whatis usually used. It does, however, imply that everystationary point is a global minimizer (Karimi et al.,2016; Polyak, 1963).

We now present a result that establishes an upperbound on the objective value in terms of the error inthe gradient of the sampled batch. We present all theproofs in the Supplementary Material.

Lemma 2. Suppose we apply an update of the form

(5) where the batch Bt

is uniformly sampled from the

distribution p on each iteration t. If the objective `satisfies Assumption 2, then we have

E[`(xt+1)� `(x?)] E

⇥`(x

t

)� `(x?)

��↵� L↵

2

2

�kr`(x

t

)k2 + L↵2

2ke

t

k2⇤.

Further, if the objective ` satisfies the PL Inequality(Assumption 3), we have:

E[`(xt+1)� `(x?)]

⇣1� 2µ

�↵� L↵

2

2

�⌘E[`(x

t

)� `(x?)] + L↵2

2Eke

t

k2.

Using Lemma 2, we now provide convergence rates forbig batch SGD.

Theorem 2. Suppose ` satisfies Assumptions 2 and3. Suppose further that on each iteration the batch

size is large enough to satisfy (4) for ✓ 2 (0, 1). If0 ↵ < 2

L�

, where � = ✓2+(1�✓)2(1�✓)2 , then we get the

following linear convergence bound for big batch SGD

using updates of the form 5:

E[`(xt+1)� `(x?)] � · E[`(xt)� `(x?)],

where � =�1 � 2µ(↵ � L↵

2�

2 )�. Choosing the optimal

stepsize of ↵ = 1�L

, we get

E[`(xt+1)� `(x?)]

�1� µ

�L

�· E[`(x

t

)� `(x?)].

Note that the above linear convergence rate boundholds without requiring convexity. Comparing it withthe convergence rate of deterministic gradient descentunder similar assumptions, we see that big batch SGDsu↵ers a slowdown by a factor �, due to the noise inthe estimation of the gradients.

3.1 Comparison to classical SGD

Conventional small batch SGD methods can attainonly O(1/t) convergence for strongly convex problems,thus requiring O(1/✏) gradient evaluations to achievean optimality gap less than ✏, and this has been shownto be optimal in the online setting (i.e., the infinitedata setting) (Rakhlin et al., 2011). In the previoussection, however, we have shown that big batch SGDmethods converge linearly in the number of iterations,under a weaker assumption than strong convexity, inthe online setting. Unfortunately, per-iteration con-vergence rates are not a fair comparison between thesemethods because the cost of a big batch iteration growswith the iteration count, unlike classical SGD. For thisreason, it is interesting to study the convergence rateof big batch SGD as a function of gradient evaluations.

From Lemma 2, we see that we should not expect toachieve an optimality gap less than ✏ until we have:L↵

2

2 EBtketk2 < ✏. In the worst case, by Theorem 1,

this requires L↵2

24L2z TrVarz(z)

|B| < ✏, or |B| � O(1/✏)gradient evaluations. Note that in the online or infinitedata case, this is an optimal bound, and matches thatof other SGD methods.

We choose to study the infinite sample case since thefinite sample case is fairly trivial with a growing batchsize: asymptotically, the batch size becomes the wholedataset, at which point we get the same asymptoticbehavior as deterministic gradient descent, achievinglinear convergence rates.


Algorithm 1 Big batch SGD: fixed stepsize

1: initialize starting pt. x0, stepsize ↵, initial batchsize K > 1, batch size increment �

k

2: while not converged do3: Draw random batch with size |B| = K4: Calculate VB and r`B(xt) using (6)5: while kr`B(xt)k2 VB/K do6: Increase batch size K K + �

K

7: Sample more gradients8: Update VB and r`B(xt)9: end while

10: xt+1 = xt � ↵r`B(xt)

11: end while

4 Practical Implementation withBacktracking Line Search

While one could implement a big batch method us-ing analytical bounds on the gradient and its variance(such as that provided by Theorem 1), the purposeof big batch methods is to enable automated adaptiveestimation of algorithm parameters. Furthermore, thestepsize bounds provided by our convergence analysis,like the stepsize bounds for classical SGD, are fairlyconservative and more aggressive stepsize choices arelikely to be more e↵ective.

The framework outlined in Section 2.2 requires two in-gredients: estimating the batch size and estimating thestepsize. Estimating the batch size needed to achieve(4) is fairly straightforward. We start with an ini-tial batch size K, and draw a random batch B with|B| = K. We then compute the stochastic gradient es-timate r`B(xt) and the sample variance

VB :=1

|B|� 1X

z2Bkrf(x

t

; z)�r`B(xt)k2

⇡ TrVarz2B(rf(xt; z)). (6)

We then test whether kr`B(xt)k2 > VB/|B| as a proxyfor (4). If this condition holds, we proceed with a gra-dient step, else we increase the batch sizeK K+�

K

,and check our condition again. We fix �

K

= 0.1K forall our experiments. Our aggressive implementationalso simply chooses ✓ = 1. The fixed stepsize big batchmethod is listed in Algorithm 1.

We also consider a backtracking variant of SGD thatadaptively tunes the stepsize. This method selectsbatch sizes using the same criterion (6) as in the con-stant stepsize case. However, after a batch has beenselected, a backtracking Armijo line search is used toselect a stepsize. In the Armijo line search, we keepdecreasing the stepsize by a constant factor (in ourcase, by a factor of 2) until the following condition is

satisfied on each iteration:

`B(xt+1) `B(xt)� c↵tkr`B(xt)k2, (7)

where c is a parameter of the line search usually set to0 < c 0.5. We now present a convergence result ofbig batch SGD using the Armijo line search.

Theorem 3. Suppose that ` satisfies Assumptions 2and 3 and on each iteration, and the batch size is large

enough to satisfy (4) for ✓ 2 (0, 1). If an Armijo linesearch, given by (7), is used, and the stepsize is de-creased by a factor of 2 failing (7), then we get thefollowing linear convergence bound for big batch SGD

using updates of the form 5:

E[`(xt+1)� `(x?)] � · E[`(xt)� `(x?)],

where � =⇣1 � 2cµmin

�↵0,

12�L

�⌘and 0 < c 0.5.

If the initial stepsize ↵0 is set large enough such that↵0 � 12�L , then we get:

E[`(xt+1)� `(x?)]

⇣1� cµ

�L

⌘E[`(x

t

)� `(x?)].

In practice, on iterations where the batch size in-creases, we double the stepsize before running linesearch to prevent the stepsizes from decreasing mono-tonically (algorithm is listed in the supplemental).

5 Adaptive Step Sizes using theBarzilai-Borwein Estimate

While the Armijo backtracking line search leads to anautomated big batch method, the stepsize sequence ismonotonic (neglecting the heuristic mentioned in theprevious section). In this section, we derive a non-monotonic stepsize scheme that uses curvature esti-mates to propose new stepsize choices.

Our derivation follows the classical adaptive Barzilaiand Borwein (1988) (BB) method. The BB meth-ods fits a quadratic model to the objective on eachiteration, and a stepsize is proposed that is optimalfor the local quadratic model (Goldstein et al., 2014).To derive the analog of the BB method for stochasticproblems, we consider quadratic approximations of theform `(x) = E

�

f(x;�), where f(x;�) = ⌫2kx��k2 and

� ⇠ N (x?,�2I). We now derive the optimal stepsizeon each iteration for this quadratic approximation (fordetails see the supplemental). We have:

`(x) = E�

f(x;�) =⌫

2

�kx� x?k2 + d�2

�,

Now, we can rewrite the big batch SGD update as:

xt+1 = xt � ↵t

1

|B|X

i2B⌫(x

t

� �i

)


= (1� ⌫↵t

)xt

+ ⌫↵t

x? +⌫�↵

t

|B|X

i2B⇠i

,

where we write �i

= x?+�⇠i

with ⇠i

⇠ N (0, 1). Thus,minimizing E[`(x

t+1)] w.r.t. ↵t we get:

↵t

=1

⌫·✓1�

1|Bt| TrVar[rf(xt)]

E��r`Bt(xt)

��2

◆. (8)

Here ⌫ denotes the curvature of the quadratic approx-imation. Note that, in the case of deterministic gradi-ent descent, the optimal stepsize is simply 1/⌫ (Gold-stein et al., 2014).

We estimate the curvature ⌫t

on each iteration usingthe BB least-squares rule (Barzilai and Borwein, 1988):

⌫t

=hx

t

� xt�1,r`Bt(xt)�r`Bt(xt�1)ikx

t

� xt�1k2

. (9)

Thus, each time we sample a batch Bt

on the t-th itera-tion, we calculate the gradient on that batch in the pre-vious iterate, i.e., we calculate r`Bt(xt�1). This givesus an approximate curvature estimate, with which wederive the stepsize ↵

t

using (8).

5.1 Convergence Proof

Here we prove convergence for the adaptive stepsizemethod described above. For the convergence proof,we first state two assumptions:

Assumption 4. Each f has L-Lipschitz gradients:f(x) f(y) +rf(y)T (x� y) + L2 kx� yk

2.

Assumption 5. Each f is µ-strongly convex:hrf(x)�rf(y), x� yi � µkx� yk2.

Note that both assumptions are stronger than As-sumptions 2 and 3, i.e., Assumption 4 implies 2 andAssumption 5 implies 3 (Karimi et al., 2016). Bothare very standard assumptions frequently used in theconvex optimization literature.

Also note that from (8), we can lower bound the step-size as: ↵

t

� (1 � ✓2)/⌫. Thus, the stepsize for bigbatch SGD is scaled down by at most 1� ✓2. For sim-plicity, we assume that the stepsize is set to this lowerbound:↵

t

= (1 � ✓2)/⌫t

. Thus, from Assumptions 4and 5, we can bound ⌫

t

, and also ↵t

, as follows:

µ ⌫t

L =) 1� ✓2

L ↵

t

1� ✓2

µ.

From Theorem 2, we see that we have linear conver-gence with the adaptive stepsize method when:

1� 2µ�↵� L↵

2�

2

� 1� 2(1� ✓

2)

+ �(1� ✓2)2 < 1,

=) 2 < 2�(1� ✓2) ,

where = L/µ is the condition number. We see thatthe adaptive stepsize method enjoys a linear conver-gence rate when the problem is well-conditioned. Inthe next section, we talk about ways to deal withpoorly-conditioned problems.

5.2 Practical Implementation

To achieve robustness of the algorithm for poorlyconditioned problems, we include a backtracking linesearch after calculating (8), to ensure that the stepsizesdo not blow up. Further, instead of calculating twogradients on each iteration (r`Bt(xt) andr`Bt(xt�1)),our implementation uses the same batch (and stepsize)on two consecutive iterations. Thus, one parameterupdate takes place for each gradient calculation.

We found the stepsize calculated from (8) to be noisywhen the batch is small. While this did not a↵ect long-term performance, we perform a smoothing operationto even out the stepsizes and make performance morepredictable. Let ↵̃

t

denote the stepsize calculated from(8). Then, the stepsize on each iteration is given by

↵t

=⇣1� |B|

N

⌘↵t�1 +

|B|N

↵̃t

.

This ensures that the update is proportional to howaccurate the estimate on each iteration is. This sim-ple smoothing operation seemed to work very well inpractice as shown in the experimental section. Notethat when |B

t

| = N , we just use ↵t

= 1/⌫t

. Sincethere is no noise in the algorithm in this case, we usethe optimal stepsize for a deterministic algorithm (seesupplemental for the complete algorithm).

6 Experiments

In this section, we present our experimental results.We explore big batch methods with both convex andnon-convex (neural network) experiments on large andhigh-dimensional datasets.

6.1 Convex Experiments

For the convex experiments, we test big batch SGDon a binary classification problem with logistic regres-sion: min

x

1n

Pn

i=1 log(1 + exp(�biaTi x)), and a linearregression problem: min

x

1n

Pn

i=1(aT

i

x� bi

)2.

Figure 1 presents the results of our convex experi-ments on three standard real world datasets: IJCNN1(Prokhorov, 2001) and COVERTYPE (Blackard andDean, 1999) for logistic regression, and MILLION-SONG (Bertin-Mahieux et al., 2011) for linear re-gression. As a preprocessing step, we normalize the


Figure 1: Convex experiments. Left to right: Ridge regression on MILLIONSONG; Logistic regression on COVERTYPE;Logistic regression on IJCNN1. The top row shows how the norm of the true gradient decreases with the number of epochs,the middle and bottom rows show the batch sizes and stepsizes used on each iteration by the big batch methods. Here‘passes through the data’ indicates number of epochs, while ‘iterations’ refers to the number of parameter updates usedby the method (there may be multiple iterations during one epoch).

features for each dataset. We compare deterministicgradient descent (GD) and SGD with stepsize decay(↵

t

= a/(b+ t)) to big batch SGD using a fixed step-size (BBS+Fixed LR), with backtracking line search(BBS+Armijo) and with the adaptive stepsize (8)(BBS+BB), as well as the growing batch method de-scribed in Friedlander and Schmidt (2012) (denoted asSF; while the authors propose a quasi-Newton method,we adapt their algorithm to a first-order method).We selected stepsize parameters using a comprehen-sive grid search for all algorithms, except BBS+Armijoand BBS+BB, which require no parameter tuning.

We see that across all three problems, the big batchmethods outperform the other algorithms. We also seethat both fully automated methods are always compa-rable to or better than fixed stepsize methods. The au-tomated methods increase the batch size more slowlythan BBS+Fixed LR and SF, and thus, these methodscan take more steps with smaller batches, leveraging

its advantages longer. Further, note that the stepsizesderived by the automated methods are very close tothe optimal fixed stepsize rate.

6.2 Neural Network Experiments

To demonstrate the versatility of the big batch SGDframework, we also present results on neural networkexperiments. We compare big batch SGD against SGDwith finely tuned stepsize schedules and fixed step-sizes. We also compare with Adadelta (Zeiler, 2012),and combine the big batch method with AdaDelta(BB+AdaDelta) to show that more complex SGD vari-ants can benefit from growing batch sizes. In addi-tion, we had also compared big batch methods withL-BFGS. However, we found L-BFGS to consistentlyyield poorer generalization error on neural networks,and thus we omitted these results.

We train a convolutional neural network (LeCun et al.,


0 10 20 30 40Number of epochs

50

60

70

80

90

100

Accuracy

Mean class accuracy (train set)

AdadeltaBB+AdadeltaSGD+Mom (Fine Tuned)SGD+Mom (Fixed LR)BBS+Mom (Fixed LR)BBS+Mom+Armijo


86

88

90

92

94

96

98

100

Accuracy




98.4

98.6

98.8

99

99.2

99.4

99.6

99.8

100

Accuracy



0 10 20 30 40

Number of epochs

50

55

60

65

70

75

80

Accuracy

Mean class accuracy (test set)

0 10 20 30 40

Number of epochs

83

84

85

86

87

88

89

90

Accuracy


0 10 20 30 40

Number of epochs

98

98.5

99

Accuracy


Figure 2: Neural Network Experiments. The three columns from left to right correspond to results for CIFAR-10, SVHN,and MNIST, respectively. The top row presents classification accuracies on the training set, while the bottom row presentsclassification accuracies on the test set.

1998) (ConvNet) to classify three benchmark imagedatasets: CIFAR-10 (Krizhevsky and Hinton, 2009),SVHN (Netzer et al., 2011), and MNIST (LeCun et al.,1998). Our ConvNet is composed of 4 layers, excludingthe input layer, with over 4.3 million weights. To com-pare against fine-tuned SGD, we used a comprehensivegrid search on the stepsize schedule to identify the op-timal schedule. Fixed stepsize methods use the defaultdecay rule of the Torch library: ↵

t

= ↵0/(1 + 10�7t),where ↵0 was chosen to be the stepsize used in the finetuned experiments. We also tune the hyper-parameter⇢ in the Adadelta algorithm. Details of the ConvNetand exact hyper-parameters used for training are pre-sented in the supplemental.

We plot the accuracy on the train and test set vs thenumber of epochs (full passes through the dataset) inFigure 2. We notice that the big batch SGD with back-tracking performs better than both Adadelta and SGD(Fixed LR) in terms of both train and test error. Bigbatch SGD even performs comparably to fine tunedSGD but without the trouble of fine tuning. This is in-teresting because most state-of-the-art deep networks(like AlexNet (Krizhevsky et al., 2012), VGG Net (Si-monyan and Zisserman, 2014), ResNets (He et al.,2016)) were trained by their creators using standardSGD with momentum, and training parameters weretuned over long periods of time (sometimes months).Finally, we note that the big batch AdaDelta performsconsistently better than plain AdaDelta on both largescale problems (SVHN and CIFAR-10), and perfor-mance is nearly identical on the small-scale MNISTproblem.

7 Conclusion

We analyzed and studied the behavior of alternativeSGD methods in which the batch size increases overtime. Unlike classical SGD methods, in which stochas-tic gradients quickly become swamped with noise,these “big batch” methods maintain a nearly constantsignal to noise ratio of the approximate gradient. As aresult, big batch methods are able to adaptively adjustbatch sizes without user oversight. The proposed au-tomated methods are shown to be empirically compa-rable or better performing than other standard meth-ods, but without requiring an expert user to chooselearning rates and decay parameters.

Acknowledgements

This work was supported by the US O�ce of Naval Re-search (N00014-17-1-2078), and the National ScienceFoundation (CCF-1535902 and IIS-1526234). A. Ya-dav and D. Jacobs were supported by the O�ce of theDirector of National Intelligence (ODNI), IntelligenceAdvanced Research Projects Activity (IARPA), viaIARPA R&D Contract No. 2014-14071600012. Theviews and conclusions contained herein are those ofthe authors and should not be interpreted as necessar-ily representing the o�cial policies or endorsements,either expressed or implied, of the ODNI, IARPA, orthe U.S. Government. The U.S. Government is autho-rized to reproduce and distribute reprints for Govern-mental purposes notwithstanding any copyright anno-tation thereon.


References

Jonathan Barzilai and Jonathan M Borwein. Two-point step size gradient methods. IMA Journal ofNumerical Analysis, 8(1):141–148, 1988.

Thierry Bertin-Mahieux, Daniel P.W. Ellis, BrianWhitman, and Paul Lamere. The million songdataset. In Proceedings of the 12th InternationalConference on Music Information Retrieval (ISMIR

2011), 2011.

Jock A Blackard and Denis J Dean. Comparative accu-racies of artificial neural networks and discriminantanalysis in predicting forest cover types from car-tographic variables. Computers and electronics inagriculture, 24(3):131–151, 1999.

Guillaume Bouchard, Théo Trouillon, Julien Perez,and Adrien Gaidon. Accelerating stochastic gradi-ent descent via online learning to sample. arXivpreprint arXiv:1506.09016, 2015.

Richard H Byrd, Gillian M Chin, Jorge Nocedal, andYuchen Wu. Sample size selection in optimizationmethods for machine learning. Mathematical pro-gramming, 134(1):127–155, 2012.

Dominik Csiba and Peter Richtárik. Impor-tance sampling for minibatches. arXiv preprintarXiv:1602.02283, 2016.

Soham De and Tom Goldstein. E�cient distributedSGD with variance reduction. In 2016 IEEE Inter-national Conference on Data Mining. IEEE, 2016.

Soham De, Abhay Yadav, David Jacobs, and TomGoldstein. Big batch SGD: Automated infer-ence using adaptive batch sizes. arXiv preprintarXiv:1610.05792, 2016.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient methodwith support for non-strongly convex composite ob-jectives. In Advances in Neural Information Pro-cessing Systems, pages 1646–1654, 2014a.

Aaron J Defazio, Tibério S Caetano, and JustinDomke. Finito: A faster, permutable incremen-tal gradient method for big data problems. arXivpreprint arXiv:1407.2710, 2014b.

Michael P Friedlander and Mark Schmidt. Hy-brid deterministic-stochastic methods for data fit-ting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.

Tom Goldstein, Christoph Studer, and RichardBaraniuk. A field guide to forward-backwardsplitting with a FASTA implementation.arXiv eprint, abs/1411.3406, 2014. URLhttp://arxiv.org/abs/1411.3406.

Reza Harikandeh, Mohamed Osama Ahmed, Alim Vi-rani, Mark Schmidt, Jakub Konečnỳ, and ScottSallinen. Stopwasting my gradients: Practical svrg.In Advances in Neural Information Processing Sys-tems, pages 2251–2259, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. Deep residual learning for image recognition.In Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition, pages 770–778, 2016.

Rie Johnson and Tong Zhang. Accelerating stochasticgradient descent using predictive variance reduction.In Advances in Neural Information Processing Sys-tems, pages 315–323, 2013.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Lin-ear convergence of gradient and proximal-gradientmethods under the polyak- lojasiewicz condition. InJoint European Conference on Machine Learning

and Knowledge Discovery in Databases, pages 795–811. Springer, 2016.

Nitish Shirish Keskar, Dheevatsa Mudigere, JorgeNocedal, Mikhail Smelyanskiy, and Ping Tak Pe-ter Tang. On large-batch training for deep learn-ing: Generalization gap and sharp minima. arXivpreprint arXiv:1609.04836, 2016.

Diederik Kingma and Jimmy Ba. Adam: Amethod for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014.

Alex Krizhevsky and Geo↵rey Hinton. Learning mul-tiple layers of features from tiny images, 2009.

Alex Krizhevsky, Ilya Sutskever, and Geo↵rey E Hin-ton. Imagenet classification with deep convolutionalneural networks. In Advances in neural informationprocessing systems, pages 1097–1105, 2012.

Yann LeCun, Léon Bottou, Yoshua Bengio, andPatrick Ha↵ner. Gradient-based learning applied todocument recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

Maren Mahsereci and Philipp Hennig. Probabilisticline searches for stochastic optimization. In Ad-vances In Neural Information Processing Systems,pages 181–189, 2015.

Deanna Needell, Rachel Ward, and Nati Srebro.Stochastic gradient descent, weighted sampling, andthe randomized kaczmarz algorithm. In Advancesin Neural Information Processing Systems, pages1017–1025, 2014.

Yuval Netzer, Tao Wang, Adam Coates, AlessandroBissacco, Bo Wu, and Andrew Y Ng. Reading digitsin natural images with unsupervised feature learn-ing. In NIPS workshop on deep learning and un-supervised feature learning, volume 2011, page 4.Granada, Spain, 2011.


Boris Teodorovich Polyak. Gradient methods for min-imizing functionals. Zhurnal Vychislitel’noi Matem-atiki i Matematicheskoi Fiziki, 3(4):643–653, 1963.

Danil Prokhorov. Ijcnn 2001 neural network competi-tion. Slide presentation in IJCNN, 1, 2001.

Alexander Rakhlin, Ohad Shamir, and Karthik Srid-haran. Making gradient descent optimal for stronglyconvex stochastic optimization. arXiv preprintarXiv:1109.5647, 2011.

Marc Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau,and Yann LeCun. Unsupervised learning of invari-ant feature hierarchies with applications to objectrecognition. In Computer Vision and Pattern Recog-nition, 2007. CVPR’07. IEEE Conference on, pages1–8. IEEE, 2007.

Sashank J Reddi, Ahmed Hefny, Suvrit Sra, BarnabasPoczos, and Alex J Smola. On variance reductionin stochastic gradient descent and its asynchronousvariants. In Advances in Neural Information Pro-cessing Systems, pages 2647–2655, 2015.

Herbert Robbins and Sutton Monro. A stochastic ap-proximation method. The annals of mathematicalstatistics, pages 400–407, 1951.

Tom Schaul, Sixin Zhang, and Yann LeCun. No morepesky learning rates. In Proceedings of The 30th In-ternational Conference on Machine Learning, pages343–351, 2013.

Mark Schmidt, Nicolas Le Roux, and Francis Bach.Minimizing finite sums with the stochastic averagegradient. arXiv preprint arXiv:1309.2388, 2013.

Karen Simonyan and Andrew Zisserman. Very deepconvolutional networks for large-scale image recog-nition. arXiv preprint arXiv:1409.1556, 2014.

Conghui Tan, Shiqian Ma, Yu-Hong Dai, and YuqiuQian. Barzilai-borwein step size for stochastic gradi-ent descent. arXiv preprint arXiv:1605.04131, 2016.

Matthew D Zeiler. Adadelta: an adaptive learningrate method. arXiv preprint arXiv:1212.5701, 2012.

automated inference with adaptive batchesproceedings.mlr.press/v54/de17a/de17a.pdf · automated...

Documents