balancing computation and communication in distributed...

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Balancing Computation and Communication inDistributed Optimization

Ermin Wei

Department of Electrical Engineering and Computer ScienceNorthwestern University

DIMACSRutgers Univeristy

Aug 23, 2017

Balancing Computation & Communication 1/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Collaborators

Albert S. Berahas Raghu Bollapragada Nitish Shirish Keskar

Balancing Computation & Communication 2/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Overview

1 Introduction and Motivation

2 Distributed Gradient Descent Variant

3 Communication Computation Decoupled DGD Variants

4 Conclusions & Future Work

Balancing Computation & Communication 3/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Overview

1 Introduction and Motivation

2 Distributed Gradient Descent Variant

3 Communication Computation Decoupled DGD Variants

4 Conclusions & Future Work

Balancing Computation & Communication 4/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Problem Formulation

minx∈Rp

f (x) =n∑

i=1

fi (x)

Applications: Sensor Networks, Robotic Teams, Machine Learning.

Parameter estimation insensor networks.Communication

Multi-agent cooperativecontrol and coordination.Battery

Large scale computation.Computation

Balancing Computation & Communication 5/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Algorithm Evaluation

Typical numerical results (measured in iterations or time orcommunication rounds).

0 50 100 150 2000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of iterations

Err

or=

|F-F

*|/F

*

Existing AExisting B Our method

Evaluation framework should reflect features of different applications.

Balancing Computation & Communication 6/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Algorithm Evaluation

Typical numerical results (measured in iterations or time orcommunication rounds).

0 50 100 150 2000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of iterations

Err

or=

|F-F

*|/F

*

Existing AExisting B Our method

Evaluation framework should reflect features of different applications.

Balancing Computation & Communication 6/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Problem Formulation

minx∈Rp

f (x) =n∑

i=1

fi (x)

Applications: Sensor Networks, Robotic Teams, Machine Learning.

Distributed Setting: Consensus Optimization problem

minxi∈Rp

f (x) =n∑

i=1

fi (xi )

s.t. xi = xj , ∀i , j ∈ Ni

each node i has a local copy of the parameter vector xi

@ optimality consensus is achieved among all the nodes in the network

Balancing Computation & Communication 7/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Problem Formulation

minx∈Rp

f (x) =n∑

i=1

fi (x)

Applications: Sensor Networks, Robotic Teams, Machine Learning.

Distributed Setting:

Consensus Optimization problem

minxi∈Rp

f (x) =n∑

i=1

fi (xi )

s.t. xi = xj , ∀i , j ∈ Ni

each node i has a local copy of the parameter vector xi

@ optimality consensus is achieved among all the nodes in the network

Balancing Computation & Communication 7/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Problem Formulation

minx∈Rp

f (x) =n∑

i=1

fi (x)

Applications: Sensor Networks, Robotic Teams, Machine Learning.

Distributed Setting: Consensus Optimization problem

minxi∈Rp

f (x) =n∑

i=1

fi (xi )

s.t. xi = xj , ∀i , j ∈ Ni

each node i has a local copy of the parameter vector xi

@ optimality consensus is achieved among all the nodes in the network

Balancing Computation & Communication 7/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Consensus Optimization Problem

minxi∈Rp

f (x) =n∑

i=1

fi (xi )

s.t. xi = xj , ∀i , j ∈ Ni

x is a concatenation of all local xi ’s

W is a doubly-stochastic matrix that defines the connections in thenetwork

x =

x1

x2

...xn

∈ Rnp, W =

w11 w12 · · · w1n

w21 w22 · · · w2n

.... . .

...wn1 wn2 · · · wnn

∈ Rn×n, Z = W⊗ Ip ∈ Rnp×np

Balancing Computation & Communication 8/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Consensus Optimization Problem

minxi∈Rp

f (x) =n∑

i=1

fi (xi )

s.t. xi = xj , ∀i , j ∈ Ni

1

23

4

5

6

78

9x1, f1

x2, f2

x3, f3

x4, f4

x5, f5

x6, f6

x7, f7

x8, f8

x9, f9

x is a concatenation of all local xi ’s

W is a doubly-stochastic matrix that defines the connections in thenetwork

x =

x1

x2

...xn

∈ Rnp, W =

w11 w12 · · · w1n

w21 w22 · · · w2n

.... . .

...wn1 wn2 · · · wnn

∈ Rn×n, Z = W⊗ Ip ∈ Rnp×np

Balancing Computation & Communication 8/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Consensus Optimization Problem

minxi∈Rp

f (x) =n∑

i=1

fi (xi )

s.t. Zx = x

x is a concatenation of all local xi ’s

W is a doubly-stochastic matrix that defines the connections in thenetwork

x =

x1

x2

...xn

∈ Rnp, W =

w11 w12 · · · w1n

w21 w22 · · · w2n

.... . .

...wn1 wn2 · · · wnn

∈ Rn×n, Z = W⊗ Ip ∈ Rnp×np

Balancing Computation & Communication 8/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Literature Review

1 Sublinearly Converging Methods

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram et.

al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016] ...

2 Linearly and Superlinearly Converging Methods

EXTRA [Shi et. al., 2015], DIGing [Nedic et. al., 2017], NEXT [Lorenzo and

Scutari, 2015], Aug-DGM [Xu et. al., 2015], NN-EXTRA [Mokhtari et. al., 2016],

[Qu and Li, 2017], DQN [Eisen et. al., 2017], NN [Mokhtari et. al., 2014, 2015]...

3 Communication Efficient Methods

[Chen and Ozdaglar, 2012], [Shamir et. al., 2014], [Chow et. al., 2016], [Lan et.

al., 2017], [Tsianos et. al., 2012], [Zhang and Lin, 2015], ...

4 Asynchronous Methods

[Tsitsiklis and Bertsekas, 1989], [Tsitsiklis et. al., 1986], [Sundhar Ram et. al.,

2009], [Wei and Ozdaglar, 2013], [Mansoori and Wei, 2017], [Zhang and Kwok,

2014], [Wu et. al., 2017], ...

Balancing Computation & Communication 9/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Literature Review

1 Sublinearly Converging Methods

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram et.

al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016] ...

2 Linearly and Superlinearly Converging Methods

EXTRA [Shi et. al., 2015], DIGing [Nedic et. al., 2017], NEXT [Lorenzo and

Scutari, 2015], Aug-DGM [Xu et. al., 2015], NN-EXTRA [Mokhtari et. al., 2016],

[Qu and Li, 2017], DQN [Eisen et. al., 2017], NN [Mokhtari et. al., 2014, 2015]...

3 Communication Efficient Methods

[Chen and Ozdaglar, 2012], [Shamir et. al., 2014], [Chow et. al., 2016], [Lan et.

al., 2017], [Tsianos et. al., 2012], [Zhang and Lin, 2015], ...

4 Asynchronous Methods

[Tsitsiklis and Bertsekas, 1989], [Tsitsiklis et. al., 1986], [Sundhar Ram et. al.,

2009], [Wei and Ozdaglar, 2013], [Mansoori and Wei, 2017], [Zhang and Kwok,

2014], [Wu et. al., 2017], ...

Balancing Computation & Communication 9/38

Cost =#

Communication

s×cc+ #

Computation

s×cg

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Literature Review

1 Sublinearly Converging Methods

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram et.

al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016] ...

2 Linearly and Superlinearly Converging Methods

EXTRA [Shi et. al., 2015], DIGing [Nedic et. al., 2017], NEXT [Lorenzo and

Scutari, 2015], Aug-DGM [Xu et. al., 2015], NN-EXTRA [Mokhtari et. al., 2016],

[Qu and Li, 2017], DQN [Eisen et. al., 2017], NN [Mokhtari et. al., 2014, 2015]...

3 Communication Efficient Methods

[Chen and Ozdaglar, 2012], [Shamir et. al., 2014], [Chow et. al., 2016], [Lan et.

al., 2017], [Tsianos et. al., 2012], [Zhang and Lin, 2015], ...

4 Asynchronous Methods

[Tsitsiklis and Bertsekas, 1989], [Tsitsiklis et. al., 1986], [Sundhar Ram et. al.,

2009], [Wei and Ozdaglar, 2013], [Mansoori and Wei, 2017], [Zhang and Kwok,

2014], [Wu et. al., 2017], ...

Balancing Computation & Communication 9/38

Cost =#Communications

×cc

+ #Computations

×cg

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Literature Review

1 Sublinearly Converging Methods

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram et.

al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016] ...

2 Linearly and Superlinearly Converging Methods

EXTRA [Shi et. al., 2015], DIGing [Nedic et. al., 2017], NEXT [Lorenzo and

Scutari, 2015], Aug-DGM [Xu et. al., 2015], NN-EXTRA [Mokhtari et. al., 2016],

[Qu and Li, 2017], DQN [Eisen et. al., 2017], NN [Mokhtari et. al., 2014, 2015]...

3 Communication Efficient Methods

[Chen and Ozdaglar, 2012], [Shamir et. al., 2014], [Chow et. al., 2016], [Lan et.

al., 2017], [Tsianos et. al., 2012], [Zhang and Lin, 2015], ...

4 Asynchronous Methods

[Tsitsiklis and Bertsekas, 1989], [Tsitsiklis et. al., 1986], [Sundhar Ram et. al.,

2009], [Wei and Ozdaglar, 2013], [Mansoori and Wei, 2017], [Zhang and Kwok,

2014], [Wu et. al., 2017], ...

Balancing Computation & Communication 9/38

Cost =#Communications×cc

+ #Computations×cg

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Goal of the Project

Develop an algorithmic framework that is independent of the methodused to balance computation and communication in distributedoptimization

Prove convergence for methods that use the framework

Show that the framework can be applied to a many consensusoptimization problems (with different communication andcomputation costs)

Illustrate empirically that methods that utilize the frameworkoutperform their base algorithms for specific applications

Balancing Computation & Communication 10/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Goal of the Project

Develop an algorithmic framework that is independent of the methodused to balance computation and communication in distributedoptimization

Prove convergence for methods that use the framework

Show that the framework can be applied to a many consensusoptimization problems (with different communication andcomputation costs)

Illustrate empirically that methods that utilize the frameworkoutperform their base algorithms for specific applications

Balancing Computation & Communication 10/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Goal of the Project

Develop an algorithmic framework that is independent of the methodused to balance computation and communication in distributedoptimization

Prove convergence for methods that use the framework

Show that the framework can be applied to a many consensusoptimization problems (with different communication andcomputation costs)

Illustrate empirically that methods that utilize the frameworkoutperform their base algorithms for specific applications

Balancing Computation & Communication 10/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Goal of the Project

Develop an algorithmic framework that is independent of the methodused to balance computation and communication in distributedoptimization

Prove convergence for methods that use the framework

Show that the framework can be applied to a many consensusoptimization problems (with different communication andcomputation costs)

Illustrate empirically that methods that utilize the frameworkoutperform their base algorithms for specific applications

Balancing Computation & Communication 10/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

This talk

First stage of the project

Multiple consensus in DGD (theoretically and in practice)

Design a flexible first-order algorithm that decouples the twooperations

Investigate the method theoretically and empirically

By-product: variants of DGD with exact convergence

Not in this talk (ongoing work)

Multiple gradients

Extend framework to different algorithms (e.g., EXTRA, NN) orasynchronous methods

Balancing Computation & Communication 11/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

This talk

First stage of the project

Multiple consensus in DGD (theoretically and in practice)

Design a flexible first-order algorithm that decouples the twooperations

Investigate the method theoretically and empirically

By-product: variants of DGD with exact convergence

Not in this talk (ongoing work)

Multiple gradients

Extend framework to different algorithms (e.g., EXTRA, NN) orasynchronous methods

Balancing Computation & Communication 11/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Overview

1 Introduction and Motivation

2 Distributed Gradient Descent Variant

3 Communication Computation Decoupled DGD Variants

4 Conclusions & Future Work

Balancing Computation & Communication 12/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Distributed Gradient Descent (DGD)

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram

et. al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016]

xi ,k+1 =∑

j∈Ni∪iwijxj ,k − α∇fi (xi ,k), ∀i = 1, ..., n

xk+1 = Zxk − α∇f(xk)

Balancing Computation & Communication 13/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Distributed Gradient Descent (DGD)

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram

et. al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016]

xi ,k+1 =∑

j∈Ni∪iwijxj ,k − α∇fi (xi ,k), ∀i = 1, ..., n

xk+1 = Zxk − α∇f(xk)

x =

x1

x2

...xn

∈ Rnp, ∇f(xk) =

∇f1(x1,k)∇f2(x2,k)

...∇fn(xn,k)

∈ Rnp, Z = W⊗ Ip ∈ Rnp×np

Balancing Computation & Communication 13/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Distributed Gradient Descent (DGD)

DGD [Tsitsiklis and Bertsekas, 1989; Nedic and Ozdaglar, 2009; Sundhar Ram

et. al., 2010; Tsianos and Rabbat, 2012; Yuan et. al., 2015; Zeng and Yin, 2016]

xi ,k+1 =∑

j∈Ni∪iwijxj ,k − α∇fi (xi ,k), ∀i = 1, ..., n

xk+1 = Zxk − α∇f(xk)

Diminishing α: Sub-linear convergence to the solution

Constant α: Linear convergence to a neighborhood O(α)

Balancing Computation & Communication 13/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Questions

DGD

xk+1 = Zxk − α∇f(xk)

Is it necessary to do one consensus step and one optimization(gradient) step?

If not, what is the interpretation of methods that do moreconsensus/optimization steps?

What convergence guarantees can be proven for such methods?

How do these variants perform in practice?

DGDt

xk+1 = Ztxk − α∇f(xk), Zt = Wt ⊗ Ip

Balancing Computation & Communication 14/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Questions

DGD

xk+1 = Zxk − α∇f(xk)

Is it necessary to do one consensus step and one optimization(gradient) step?

If not, what is the interpretation of methods that do moreconsensus/optimization steps?

What convergence guarantees can be proven for such methods?

How do these variants perform in practice?

DGDt

xk+1 = Ztxk − α∇f(xk), Zt = Wt ⊗ Ip

Balancing Computation & Communication 14/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Questions

DGD

xk+1 = Zxk − α∇f(xk)

Is it necessary to do one consensus step and one optimization(gradient) step?

If not, what is the interpretation of methods that do moreconsensus/optimization steps?

What convergence guarantees can be proven for such methods?

How do these variants perform in practice?

DGDt

xk+1 = Ztxk − α∇f(xk), Zt = Wt ⊗ Ip

Balancing Computation & Communication 14/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Questions

DGD

xk+1 = Zxk − α∇f(xk)

Is it necessary to do one consensus step and one optimization(gradient) step?

If not, what is the interpretation of methods that do moreconsensus/optimization steps?

What convergence guarantees can be proven for such methods?

How do these variants perform in practice?

DGDt

xk+1 = Ztxk − α∇f(xk), Zt = Wt ⊗ Ip

Balancing Computation & Communication 14/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Assumptions & Definitions

Assumptions

1 Each component function fi is µi > 0 strongly convex and has Li > 0Lipschitz continuous gradients

2 The mixing matrix W is symmetric and doubly stochastic with β < 1(β is the second largest eigenvalue)

Definitions

xk =1

n

n∑i=1

xi,k , ∇f (xk) =n∑

i=1

∇fi (xi,k), ∇f (xk) =n∑

i=1

∇fi (xk)

Balancing Computation & Communication 15/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Bounded distance to minimum

Theorem (Bounded distance to minimum) [Yuan et. al., 2015]

Suppose Assumptions 1 & 2 hold, and let the step length satisfy

α ≤ min

{1 + λn(W)

Lf,

1

µf + Lf

}where µf is the strong convexity parameter of f and Lf is the Lipschitz constant of the gradientof f . Then, for all k = 0, 1, ...

‖xk+1 − x?‖2 ≤ c21‖xk − x?‖2 +

c23

(1− β)2

c21 = 1− αc2 + αδ − α2δc2, c2 =

µf Lf

µf + Lf,

c23 = α3(α+ δ−1)L2D2, D =

√√√√2L(n∑

i=1

fi (0)− f ?),

where x? = arg minx f (x) and δ > 0.

Balancing Computation & Communication 16/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Bounded distance to minimum

Theorem (Bounded distance to minimum) [Yuan et. al., 2015]

Suppose Assumptions 1 & 2 hold, and let the step length satisfy

α ≤ min

{1 + λn(W)

Lf,

1

µf + Lf

}where µf is the strong convexity parameter of f and Lf is the Lipschitz constant of the gradientof f . Then, for all k = 0, 1, ...

‖xk+1 − x?‖2 ≤ c21‖xk − x?‖2 +

c23c23c23

(1− β)2(1− β)2(1− β)2

c21 = 1− αc2 + αδ − α2δc2, c2 =

µf Lf

µf + Lf,

c23 = α3(α+ δ−1)L2D2, D =

√√√√2L(n∑

i=1

fi (0)− f ?),

where x? = arg minx f (x) and δ > 0.

Balancing Computation & Communication 16/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGDt – Theory – Bounded distance to minimum

Theorem (Bounded distance to minimum) [ASB, RB, NSK and EW, 2017]

Suppose Assumptions 1 & 2 hold, and let the step length satisfy

α ≤ min

{1 + λn(Wt)

Lf,

1

µf + Lf

}where µf is the strong convexity parameter of f and Lf is the Lipschitz constant of the gradientof f . Then, for all k = 0, 1, ...

‖xk+1 − x?‖2 ≤ c21‖xk − x?‖2 +

c23c23c23

(1− βt)2(1− βt)2(1− βt)2

c21 = 1− αc2 + αδ − α2δc2, c2 =

µf Lf

µf + Lf,

c23 = α3(α+ δ−1)L2D2, D =

√√√√2L(n∑

i=1

fi (0)− f ?),

where x? = arg minx f (x) and δ > 0.

Balancing Computation & Communication 16/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Can show similar error neighborhood for ‖xi ,k − x∗‖.Theoretical results are similar to DGD in nature (constant α)

Linear convergence to a neighborhood of the solutionImproved neighborhood

O(

1

(1− β)2

)v.s. O

(1

(1− βt)2

)But, cannot kill the neighborhood with increased communication

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Can show similar error neighborhood for ‖xi ,k − x∗‖.Theoretical results are similar to DGD in nature (constant α)

Linear convergence to a neighborhood of the solutionImproved neighborhood

O(

1

(1− β)2

)v.s. O

(1

(1− βt)2

)But, cannot kill the neighborhood with increased communication

Drawback: requires extra communication

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Can show similar error neighborhood for ‖xi ,k − x∗‖.Theoretical results are similar to DGD in nature (constant α)

Linear convergence to a neighborhood of the solutionImproved neighborhood

O(

1

(1− β)2

)v.s. O

(1

(1− βt)2

)But, cannot kill the neighborhood with increased communication

Drawback: requires extra communication

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

,

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

,

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

,

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

,

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

, W 2 =

3/8 1/4 1/4 1

81/4 3/8 1/8 1/41/4 1/8 3/8 1/41/8 1/4 1/4 3/8

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

,

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

, W 10 ≈

1/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/4

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

,

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

, W 10 ≈

1/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/4

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

,

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

, W 10 ≈

1/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/4

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

, W 2 =

5/9 1/3 1/9 01/3 1/3 2/9 1/91/9 2/9 1/3 1/3

0 1/9 1/3 5/9

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Comments

Effectively, DGDt is DGD with a different underlying graph (differentweights in W)

1 2

3 4

1 2 3 4

W =

1/2 1/4 1/4 01/4 1/2 0 1/41/4 0 1/2 1/4

0 1/4 1/4 1/2

, W 10 ≈

1/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/41/4 1/4 1/4 1/4

W =

2/3 1/3 0 01/3 1/3 1/3 00 1/3 1/3 1/3

0 0 1/3 2/3

, W 3 =

0.48 0.33 0.15 0.040.33 0.30 0.22 0.150.15 0.22 0.30 0.330.04 0.15 0.33 0.48

Balancing Computation & Communication 17/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD(t) – Numerical Results

Problem: Quadratic

f (x) =1

2

n∑i=1

xTAix + bTi x

each node i = {1, ..., n} has local data Ai ∈ Rni×p and bi ∈ Rni

Parameters: n = 10, p = 10, ni = 10, κ = 102

Methods: DGD (1,1), DGD (1,2), DGD (1,5), DGD (1,10)

Graph: 4-cyclic graph, wii = 15 , wij =

{15 if j ∈ Ni

0 otherwise

Balancing Computation & Communication 18/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD(t) – Numerical Results

Problem: Quadratic

f (x) =1

2

n∑i=1

xTAix + bTi x

each node i = {1, ..., n} has local data Ai ∈ Rni×p and bi ∈ Rni

Parameters: n = 10, p = 10, ni = 10, κ = 102

Methods: DGD (1,1), DGD (1,2), DGD (1,5), DGD (1,10)

Graph: 4-cyclic graph, wii = 15 , wij =

{15 if j ∈ Ni

0 otherwise

Show the effect of multiple consensus steps per gradient step

Balancing Computation & Communication 18/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD(t) – Numerical Results

0 1000 2000 3000 4000 5000

Iterations

10-2

10-1

100

Relative Error

DGD (1,1)

DGD (1,2)

DGD (1,5)

DGD (1,10)

0 1 2 3 4 5 6

Cost ×104

10-2

10-1

100

Relative Error

DGD (1,1)

DGD (1,2)

DGD (1,5)

DGD (1,10)

Quadratic. n = 10, p = 10, ni = 10, κ = 102.

Balancing Computation & Communication 18/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Overview

1 Introduction and Motivation

2 Distributed Gradient Descent Variant

3 Communication Computation Decoupled DGD Variants

4 Conclusions & Future Work

Balancing Computation & Communication 19/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Operators

DGD

xk+1 = Zxk − α∇f(xk)

Methods

DGD: (T − I +W)[xk ] = Zxk − α∇f(xk)

WT: W[T [xk ]] = Zxk − αZ∇f(xk)

A special case of the algorithm appeared as CTA (Combined then Adapt)and ATC in [Sayed, 13] for quadratic problems

1 What can be proven about T [W[x]] and W[T [x]] variants of DGD?2 How do these methods perform in practice?3 Advantages and limitations of the methods?

Balancing Computation & Communication 20/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Operators

DGD

xk+1 = Zxk − α∇f(xk)

Operators

W[x] = Zx

T [x] = x− α∇f(x)

Methods

DGD: (T − I +W)[xk ] = Zxk − α∇f(xk)

WT: W[T [xk ]] = Zxk − αZ∇f(xk)

A special case of the algorithm appeared as CTA (Combined then Adapt)and ATC in [Sayed, 13] for quadratic problems

1 What can be proven about T [W[x]] and W[T [x]] variants of DGD?2 How do these methods perform in practice?3 Advantages and limitations of the methods?

Balancing Computation & Communication 20/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Operators

DGD

xk+1 = Zxk − α∇f(xk)

Operators

W[x] = Zx

T [x] = x− α∇f(x)

Methods

DGD: (T − I +W)[xk ] = Zxk − α∇f(xk)

WT: W[T [xk ]] = Zxk − αZ∇f(xk)

A special case of the algorithm appeared as CTA (Combined then Adapt)and ATC in [Sayed, 13] for quadratic problems

1 What can be proven about T [W[x]] and W[T [x]] variants of DGD?2 How do these methods perform in practice?3 Advantages and limitations of the methods?

Balancing Computation & Communication 20/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Operators

DGD

xk+1 = Zxk − α∇f(xk)

Operators

W[x] = Zx

T [x] = x− α∇f(x)

Methods

DGD: (T − I +W)[xk ] = Zxk − α∇f(xk)TW: T [W[xk ]] = Zxk − α∇f(Zxk)WT: W[T [xk ]] = Zxk − αZ∇f(xk)

A special case of the algorithm appeared as CTA (Combined then Adapt)and ATC in [Sayed, 13] for quadratic problems

1 What can be proven about T [W[x]] and W[T [x]] variants of DGD?2 How do these methods perform in practice?3 Advantages and limitations of the methods?

Balancing Computation & Communication 20/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Operators

DGD

xk+1 = Zxk − α∇f(xk)

Operators

W[x] = Zx

T [x] = x− α∇f(x)

Methods

DGD: (T − I +W)[xk ] = Zxk − α∇f(xk)TW: T [W[xk ]] = Zxk − α∇f(Zxk)WT: W[T [xk ]] = Zxk − αZ∇f(xk)

A special case of the algorithm appeared as CTA (Combined then Adapt)and ATC in [Sayed, 13] for quadratic problems

1 What can be proven about T [W[x]] and W[T [x]] variants of DGD?2 How do these methods perform in practice?3 Advantages and limitations of the methods?

Balancing Computation & Communication 20/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Operators

TW

yk = Zxk

xk+1 = yk − α∇f(yk)

Methods

DGD: (T − I +W)[xk ] = Zxk − α∇f(xk)TW: T [W[xk ]] = Zxk − α∇f(Zxk)WT: W[T [xk ]] = Zxk − αZ∇f(xk)

A special case of the algorithm appeared as CTA (Combined then Adapt)and ATC in [Sayed, 13] for quadratic problems

1 What can be proven about T [W[x]] and W[T [x]] variants of DGD?2 How do these methods perform in practice?3 Advantages and limitations of the methods?

Balancing Computation & Communication 20/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TW – Methods, Assumptions & Definitions

Methods

TWt : t (predetermined) consensus steps for every gradient step

TW+: increasing number of consensus steps

Assumptions

1 Assumptions 1 & 2, same as before

Definitions

xk =1

n

n∑i=1

xi ,k , ∇f (yk) =n∑

i=1

∇fi (yi ,k), ∇f (xk) =n∑

i=1

∇fi (xk)

Balancing Computation & Communication 21/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TWt – Theory – Bounded distance to minimum

Theorem (Bounded distance to minimum) [ASB, RB, NSK and EW, 2017]

Suppose Assumptions 1 & 2 hold, and let the step length satisfy

α ≤ min

{1 + λn(Wt)

Lf,

1

µf + Lf

}where µf is the strong convexity parameter of f and Lf is the Lipschitz constant of the gradientof f . Then, for all k = 0, 1, ...

‖xi,k − x?‖ ≤ cki ‖x?‖+

c3√1− c2

1

βt + βtαD

where x? = arg minx f (x) and δ > 0.

Balancing Computation & Communication 22/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TW+ Theory (increasing Consensus)

Can we increase the number of consensus steps and converge to thesolution?

Increase t(k) accordingly so that we kill the error term

O(βt(k))

Similar idea appeared in [Chen and Ozdaglar, 2012] for nonsmoothproblems

Resulting in TW t(k) algorithm with exact convergence: As long as wekeep increasing the number of consensus

Balancing Computation & Communication 23/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TW+ Theory (increasing Consensus)

Can we increase the number of consensus steps and converge to thesolution?

Increase t(k) accordingly so that we kill the error term

O(βt(k))

Similar idea appeared in [Chen and Ozdaglar, 2012] for nonsmoothproblems

Resulting in TW t(k) algorithm with exact convergence: As long as wekeep increasing the number of consensus

Balancing Computation & Communication 23/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TW+ Theory (increasing Consensus)

Can we increase the number of consensus steps and converge to thesolution?

Increase t(k) accordingly so that we kill the error term

O(βt(k))

Similar idea appeared in [Chen and Ozdaglar, 2012] for nonsmoothproblems

Resulting in TW t(k) algorithm with exact convergence: As long as wekeep increasing the number of consensus

Balancing Computation & Communication 23/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TW+ Theory (increasing Consensus)

Can we increase the number of consensus steps and converge to thesolution?

Increase t(k) accordingly so that we kill the error term

O(βt(k))

Similar idea appeared in [Chen and Ozdaglar, 2012] for nonsmoothproblems

Resulting in TW t(k) algorithm with exact convergence: As long as wekeep increasing the number of consensus

Balancing Computation & Communication 23/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TW+ – Theory – Bounded distance to minimum

Theorem (Bounded distance to minimum)[ASB, RB, NSK and EW, 2017]

Suppose Assumptions 1 & 2 hold, t(k) = k and let the step length satisfy

α ≤ min

{1

Lf,

1

µf + Lf

}where µf is the strong convexity parameter of f and Lf is the Lipschitz constant of the gradientof f . Then, for all k = 0, 1, ...

‖xi,k − x?‖ ≤ Cρk ,

where x? = arg minx f (x) and some constants C , ρ.

When t(k) = k to reach an ε−accurate solution, we need O(log( 1ε )) number of

gradient evaluation and O((log( 1ε ))2) rounds of communication.

Balancing Computation & Communication 24/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments

Methods: DGD, TW (1,1,-), TW (10,1,-), TW (1,10,-), TW (1,1,k),TW (1,1,500), TW (1,1,1000)

Problem: Quadratic

f (x) =1

2

n∑i=1

xTAix + bTi x

each node i = {1, ..., n} has local data Ai ∈ Rni×p and bi ∈ Rni

Parameters: n = 10, p = 10, ni = 10, κ = Lµ = 104

Graph: 4-cyclic graph

Balancing Computation & Communication 25/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments

Methods: DGD, TW (1,1,-), TW (10,1,-), TW (1,10,-), TW (1,1,k),TW (1,1,500), TW (1,1,1000)

Problem: Quadratic

f (x) =1

2

n∑i=1

xTAix + bTi x

each node i = {1, ..., n} has local data Ai ∈ Rni×p and bi ∈ Rni

Parameters: n = 10, p = 10, ni = 10, κ = Lµ = 104

Graph: 4-cyclic graph

Balancing Computation & Communication 25/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Balancing Computation & Communication 26/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems – Different Costs

0 2 4 6 8 10

Cost ×104

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2 4 6 8 10

Cost ×104

10-6

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2 4 6 8 10

Cost ×104

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.

Left: cg = 10, cc = 1;

Center: cg = 1, cc = 1;

Right: cg = 1, cc = 10.

Balancing Computation & Communication 27/38

Cost = #Communications × cc + #Computations × cg

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems – Different Costs

0 2 4 6 8 10

Cost ×104

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2 4 6 8 10

Cost ×104

10-6

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2 4 6 8 10

Cost ×104

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.Left: cg = 10, cc = 1;

Center: cg = 1, cc = 1;

Right: cg = 1, cc = 10.

Balancing Computation & Communication 27/38

Cost = #Communications × cc + #Computations × cg

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems – Different Costs

0 2 4 6 8 10

Cost ×104

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2 4 6 8 10

Cost ×104

10-6

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2 4 6 8 10

Cost ×104

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

Quadratic. n = 10, p = 10, ni = 10, κ = 104.Left: cg = 10, cc = 1;

Center: cg = 1, cc = 1;Right: cg = 1, cc = 10.

Balancing Computation & Communication 27/38

Cost = #Communications × cc + #Computations × cg

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Logistic Regression

Problem: Logistic Regression - Binary Classification (MushroomDataset)

f (x) =1

n · ni

n∑i=1

ni∑j=1

log(1 + e−(bi )j (xT (Ai )j·))

where A ∈ Rn·ni×p and b ∈ {−1, 1}n·ni , and each node i = 1, ..., nhas a portion of A and b, Ai ∈ Rni×p and bi ∈ Rni

Parameters: n = 10, p = 114, ni = 812, κ = 104

Graph: 4-cyclic graph

Balancing Computation & Communication 28/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Numerical Experiments – Logistic Regression

0 1000 2000 3000 4000 5000

Iterations

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,250)

TW (1,1,500)

0 2000 4000 6000 8000 10000

Cost

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,250)

TW (1,1,500)

Logistic Regression - mushroom. n = 10, p = 114, ni = 812.

Balancing Computation & Communication 28/38

Cost = #Communications × 1 + #Computations × 1

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Overview

1 Introduction and Motivation

2 Distributed Gradient Descent Variant

3 Communication Computation Decoupled DGD Variants

4 Conclusions & Future Work

Balancing Computation & Communication 29/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Final Remarks

Most distributed optimization algorithms do one communication andone computation per iteration

Showed the effect (theoretically and empirically) of doing multipleconsensus in DGD

Proposed a variant of DGD, TW, that decouples the two operations(consensus and computation) and converges to the solution byperforming multiple consensus steps

Important to balance communication and computation in order to getbest performance in terms of cost — right balance depends on theapplication (e.g., cost of communication and cost of computation)

Balancing Computation & Communication 30/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Final Remarks

Most distributed optimization algorithms do one communication andone computation per iteration

Showed the effect (theoretically and empirically) of doing multipleconsensus in DGD

Proposed a variant of DGD, TW, that decouples the two operations(consensus and computation) and converges to the solution byperforming multiple consensus steps

Important to balance communication and computation in order to getbest performance in terms of cost — right balance depends on theapplication (e.g., cost of communication and cost of computation)

Balancing Computation & Communication 30/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Final Remarks

Most distributed optimization algorithms do one communication andone computation per iteration

Showed the effect (theoretically and empirically) of doing multipleconsensus in DGD

Proposed a variant of DGD, TW, that decouples the two operations(consensus and computation) and converges to the solution byperforming multiple consensus steps

Important to balance communication and computation in order to getbest performance in terms of cost — right balance depends on theapplication (e.g., cost of communication and cost of computation)

Balancing Computation & Communication 30/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Final Remarks

Most distributed optimization algorithms do one communication andone computation per iteration

Showed the effect (theoretically and empirically) of doing multipleconsensus in DGD

Proposed a variant of DGD, TW, that decouples the two operations(consensus and computation) and converges to the solution byperforming multiple consensus steps

DGD: xk+1 = Zxk − α∇f (xk)

TW: xk+1 = Zxk − α∇f (Zxk)

Important to balance communication and computation in order to getbest performance in terms of cost — right balance depends on theapplication (e.g., cost of communication and cost of computation)

Balancing Computation & Communication 30/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Final Remarks

Most distributed optimization algorithms do one communication andone computation per iteration

Showed the effect (theoretically and empirically) of doing multipleconsensus in DGD

Proposed a variant of DGD, TW, that decouples the two operations(consensus and computation) and converges to the solution byperforming multiple consensus steps

Important to balance communication and computation in order to getbest performance in terms of cost — right balance depends on theapplication (e.g., cost of communication and cost of computation)

Balancing Computation & Communication 30/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Future Work

Apply framework to other algorithms (exact, second-order,asynchronous, ...)

Construct framework to do multiple gradient steps

Adapt number of gradient and communication steps in algorithmicway

Other considerations: memory access, partial blocks, quantizationeffects, dynamic environment

Balancing Computation & Communication 31/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Backup Slides

Balancing Computation & Communication 32/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Bounded gradients

Lemma (Bounded gradients) [Yuan et. al., 2015]

Suppose Assumption 1 holds, and let the step size satisfy

α ≤ 1 + λn(W)

L

where λn(W) is the smallest eigenvalue of W and L = maxi Li . Then, starting fromxi,0 = 0 (i = 1, 2, ..., n), the sequence xi,k generated by DGD converges. In addition, wealso have

‖∇f(xk)‖ ≤ D =

√√√√2L(1

n

n∑i=1

fi (0)− f ?i ) (1)

for all k = 1, 2, ..., where f ?i = fi (x?i ) and x?i = arg minx fi (x).

Balancing Computation & Communication 33/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGDt – Theory – Bounded gradients

Lemma (Bounded gradients) [ASB, RB, NSK and EW, 2017]

Suppose Assumption 1 holds, and let the step size satisfy

α ≤ 1 + λn(Wt)

L

where λn(Wt) is the smallest eigenvalue of Wt and L = maxi Li . Then, starting fromxi,0 = 0 (i = 1, 2, ..., n), the sequence xi,k generated by DGD converges. In addition, wealso have

‖∇f(xk)‖ ≤ D =

√√√√2L(1

n

n∑i=1

fi (0)− f ?i ) (1)

for all k = 1, 2, ..., where f ?i = fi (x?i ) and x?i = arg minx fi (x).

Balancing Computation & Communication 33/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGD – Theory – Bounded deviation from mean

Lemma (Bounded deviation from mean) [Yuan et. al., 2015]

If (1) and Assumption 2 hold, then the total deviation from the mean is bounded,namely,

‖xi,k − xk‖ ≤αD

1− β

for all k and i . Moreover, if in addition Assumption 1 holds, then

‖∇fi (xi,k)−∇fi (xk)‖ ≤ αDLi

1− β

‖∇f (xk)−∇f (xk)‖ ≤ αDL

1− β

for all k and i .

Balancing Computation & Communication 34/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGDt – Theory – Bounded deviation from mean

Lemma (Bounded deviation from mean) [ASB, RB, NSK and EW, 2017]

If (1) and Assumption 2 hold, then the total deviation from the mean is bounded,namely,

‖xi,k − xk‖ ≤αD

1− βt

for all k and i . Moreover, if in addition Assumption 1 holds, then

‖∇fi (xi,k)−∇fi (xk)‖ ≤ αDLi

1− βt

‖∇f (xk)−∇f (xk)‖ ≤ αDL

1− βt

for all k and i .

Balancing Computation & Communication 34/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

TWt – Theory – Bounded deviation from mean

Lemma (Bounded deviation from mean) [ASB, RB, NSK and EW, 2017]

If Assumption 2 holds, then the total deviation from the mean is bounded, namely,

‖yi,k − xk‖ ≤βtαDk(k + 1)

2

for all k and i . Moreover, if in addition Assumption 1 holds, then

‖∇fi (yi,k)−∇fi (xk)‖ ≤ βtαDLik(k + 1)

2

‖∇f (yk)−∇f (xk)‖ ≤ βtαDLk(k + 1)

2

for all k and i .

Balancing Computation & Communication 35/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

DGDt Numerical Results

0 1000 2000 3000 4000 5000

Iterations

10-2

10-1

100

Relative Error

DGD (1,1)

DGD (1,2)

DGD (1,5)

DGD (1,10)

0 1000 2000 3000 4000 5000

Number of Gradient Evaluations

10-2

10-1

100

Relative Error

DGD (1,1)

DGD (1,2)

DGD (1,5)

DGD (1,10)

0 0.5 1 1.5 2

Number of Communications ×106

10-2

10-1

100

Relative Error

DGD (1,1)

DGD (1,2)

DGD (1,5)

DGD (1,10)

0 1 2 3 4 5 6

Cost ×104

10-2

10-1

100

Relative Error

DGD (1,1)

DGD (1,2)

DGD (1,5)

DGD (1,10)

Quadratic. n = 10, p = 10, ni = 10, κ = 102.

Balancing Computation & Communication 36/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Quadratic Problems

0 2000 4000 6000 8000 10000

Iterations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 2000 4000 6000 8000 10000

Number of Gradient Evaluations

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 0.5 1 1.5 2 2.5 3 3.5 4

Number of Communications ×105

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

0 0.5 1 1.5 2

Cost ×104

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,500)

TW (1,1,1000)

Quadratic. n = 10, d = 10, ni = 10, κ = 104.

Balancing Computation & Communication 37/38

Introduction and MotivationDistributed Gradient Descent Variant

Communication Computation Decoupled DGD VariantsConclusions & Future Work

Experiments – Logistic Regression

0 1000 2000 3000 4000 5000

Iterations

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,250)

TW (1,1,500)

0 1000 2000 3000 4000 5000

Number of Gradient Evaluations

10-5

10-4

10-3

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,250)

TW (1,1,500)

0 0.5 1 1.5 2

Number of Communications ×105

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,250)

TW (1,1,500)

0 2000 4000 6000 8000 10000

Cost

10-2

10-1

100

Relative Error

DGD

TW (1,1,-)

TW (10,1,-)

TW (1,10,-)

TW (1,1,k)

TW (1,1,250)

TW (1,1,500)

Logistic Regression - mushroom. n = 10, d = 114, ni = 812.

Balancing Computation & Communication 38/38