Communication-Efficient Distributed SGD with Error-Feedback, Revisited

Tran Thi Phuong; Le Trieu Phong

doi:10.2991/ijcis.d.210412.001

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Volume 14, Issue 1, 2021, Pages 1373 - 1387

Communication-Efficient Distributed SGD with Error-Feedback, Revisited

Authors

Tran Thi Phuong¹^{, 2}^{, 3}^{, *}^,, Le Trieu Phong³^,

¹Faculty of Mathematics and Statistics, Ton Duc Thang University, No.19 Nguyen Huu Tho Street, Tan Phong Ward, District 7, Ho Chi Minh City, Vietnam

²Meiji University, 1-1-1 Higashi-Mita, Tama-ku, Kawasaki-shi, Kanagawa, 214-8571, Japan

³National Institute of Information and Communications Technology (NICT) 4-2-1, Nukui-Kitamachi, Koganei, Tokyo, 184-8795, Japan

^*Corresponding author. Email: tranthiphuong@tdtu.edu.vn

Corresponding Author

Tran Thi Phuong

Received 15 July 2020, Accepted 31 March 2021, Available Online 20 April 2021.

DOI: 10.2991/ijcis.d.210412.001 How to use a DOI?
Keywords: Optimizer; Distributed learning; SGD; Error-feedback; Deep neural networks
Abstract: We show that the convergence proof of a recent algorithm called dist-EF-SGD for distributed stochastic gradient descent with communication efficiency using error-feedback of Zheng et al., Communication-efficient distributed blockwise momentum SGD with error-feedback, in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS 2019), 2019, pp. 11446–11456, is problematic mathematically. Concretely, the original error bound for arbitrary sequences of learning rate is unfortunately incorrect, leading to an invalidated upper bound in the convergence theorem for the algorithm. As evidences, we explicitly provide several counter-examples, for both convex and nonconvex cases, to show the incorrectness of the error bound. We fix the issue by providing a new error bound and its corresponding proof, leading to a new convergence theorem for the dist-EF-SGD algorithm, and therefore recovering its mathematical analysis.
Copyright: © 2021 The Authors. Published by Atlantis Press B.V.
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

1.1. Background

For training deep neural networks over large-scale and distributed datasets, distributed stochastic gradient descent (distributed SGD) is a vital method. In distributed SGD, a central server updates the model parameters using information transmitted from distributed workers, as illustrated in Figure 1.

Communication between the server and distributed workers can be a bottleneck in distributed SGD. Alleviating the bottleneck is a considerable concern of the community, so that variants of distributed SGD using gradient compression have been proposed to reduce the communication cost between workers and the server.

Recently, Zheng et al. [1] proposed an algorithm named dist- EF-SGD recalled in Algorithm 1, in which gradients are compressed before transmission, and errors between real and compressed gradients in one step of the algorithm are re-used in future steps.

1.2. Our Contributions

In this paper, we point out a flaw in the convergence proof of Algorithm 1 given in Zheng et al. [1]. We then fix the flaw by providing a new convergence theorem with a new proof for Algorithm 1.

Zheng et al. [1] stated the following theorem for any sequence of learning rate {η_t}.

Theorem A

(Theorem 1 of [1], problematic). Suppose that Assumptions 1-3 (given together with related notations in Section 2) hold. Assume that the learning rate 0<ηt<32L for all t ⩾ 0. For sequence x_t generated from Algorithm 1, we have the following upper bound on the expected Euclidean norm of gradients:

E[‖∇f(xo)‖2]⩽4(f(x0)−f⋆)∑k=0T−1ηk(3−2Lηk)+2Lσ2M∑t=0T−1ηt2∑k=0T−1ηk(3−2Lηk)+32L2(1−δ)G2δ2[1+16δ2]∑t=0T−1ηtηt−12∑k=0T−1ηk(3−2Lηk)

where o ∈ {0, …, T − 1} is an index such that the probability

Pr(o=k)=ηk(3−2Lηk)∑t=0T−1ηt(3−2Lηt),∀k=0,…,T−1.

Problem in Theorem A.

Unfortunately, the proof of Theorem A as given in [1] becomes invalidated when the learning rate sequence {η_t} is decreasing. In that proof, a lemma is employed to handle decreasing learning rate sequences. However, in Section 3 we present several counter-examples showing that lemma does not hold. We move on to fix that lemma and finally obtain the following result as our correction for Theorem A.

Theorem 1.

(Our correction for Theorem A) With all notations and assumptions are identical to Theorem A, we have

E[‖∇f(xo)‖2]⩽4(f(x0)−f⋆)∑k=0T−1ηk(3−2Lηk)+2Lσ2M∑t=0T−1ηt2∑k=0T−1ηk(3−2Lηk)+8(1−δ)(2−δ)G2L2δ∑k=0T−1ηk(3−2Lηk)∑t=0T−1ηtηt−12∑k=0t−1ηt−1−k2ηt−12αk+16(1−δ)(2−δ)3G2L2δ2∑k=0T−1ηk(3−2Lηk)× ∑t=0T−1ηtηt−12∑j=0t−1αt−1−j∑k=0jηj−k2ηt−12αk,

where α=1−δ2 and o∈{0,…,T−1} is an index

Algorithm 1 Distributed SGD with Error-Feedback (dist-EF-SGD) [1]

1: Input: Loss function L, learning rate {η_t} with η₋₁ = 0; number of workers M; compressor C(⋅)

2: Initialize: initial parameter x0∈ℝd; error e0,i=0∈ℝd on each worker i; error e~0=0∈ℝd on server

3: for t∈{0,…,T−1} do

4: • on each worker 1 ⩽ i ⩽ M:

5: pick data ξ_t,i from the dataset

6: gt,i=∇L(xt,ξt,i) ▷ stochastic gradient

7: pt,i=gt,i+ηt−1ηtet,i ▷ gradient added with previous error

8: push Δt,i=C(pt,i) to server ▷ gradient compression at worker, and transmission

9: pull Δ~t from server

10: xt+1=xt−ηtΔ~t ▷ local weight update

11: et+1,i=pt,i−Δt,i ▷ local error-feedback to next step

12: • on central parameter server:

13: pull ∆_t,i from each worker i

14: p~t=1M∑i=1MΔt,i+ηt−1ηte~t ▷ gradient average with error

15: push Δ~t=C(p~t) to each worker ▷ gradient compression at server

16: e~t+1=p~t−Δ~t ▷ error on server

17: end for

such that the probability

Pr(o=k)=ηk(3−2Lηk)∑t=0T−1ηt(3−2Lηt),∀k=0,…,T−1.

In addition, we show that the upper bound in Theorem 1 becomes O(1MT) for a proper choice of decreasing sequence {η_t} in Corollary 2. Moreover the upper bound in Theorem 1 matches previous results given in Zheng et al. [1] when {η_t} is nondecreasing (Corollary 1).

1.3. Paper Roadmap

We begin with notations and settings in Section 2. In Section 3, we provide counter-examples to justify the issue in [1] for both nonconvex and convex cases. We then correct the issue in Section 4 and then present a proof for Theorem 1 in Section 5.

1.4. Related Works

The use of gradient compression for reducing the communication cost is widely considered in distributed machine learning recently. One line of research is to compress the gradient only on the worker side before sending the result to the parameter server, namely one-side compression. The parameter server receives and aggregates these results and sends back the aggregated result to all workers. Some recent papers such as [2–5] are in this line of research.

Another line of research uses gradient compression on both workers and server, namely two-side compression. In these two-side compression methods, the workers send the compressed local gradients or some corrected forms of them to the parameter server, and the parameter server compresses the aggregated result before sending it back to all workers. Papers [1,6,7] use two-side compression with an identical method of gradient compression for both workers and the parameter server. Paper [8] considers two-side compression with flexible compression for both workers and the parameter server.

2. PRELIMINARIES

Let 〈⋅,⋅〉 be the inner product of vectors. The Cauchy–Schwarz inequality states that for all vectors u, v it holds that |〈u,v〉|2⩽〈u,u〉×〈v,v〉. The Young inequality γ > 0 (sometimes called the Peter–Paul inequality) states that (a + b)² ⩽ (1 + γ)a² + (1 + 1/γ)b² ∀a, b ∈ ℝ. Let ‖⋅‖ be the Euclidean norm of a vector.

For completeness, we recall the algorithm of Zheng et al. [1] in Algorithm 1 and its explanation as follows. At iteration t, the scale ηt−1ηt of the local accumulated error vector e_t,i is added to the gradient g_t,i (line 7 of Algorithm 1) for the compression step. Each worker i stores these local accumulated error vector e_t,i and local corrected gradient vector p_t,i for the next iteration. The compressed ∆_t,i of p_t,i are pushed to the parameter server. The parameter server aggregates these ∆_t,i and uses the aggregated result to update the global error-corrected vector p~t (line 14 of Algorithm 1), which in turn is used to update the global accumulated error vector e~t+1 (line 16 of Algorithm 1). Each worker receives the compressed Δ~t of p~t from the parameter server and uses it to update the parameter x_t₊₁.

In order to construct Algorithm 1, Zheng et al. [1] used the idea of Karimireddy et al. [9] that combined gradient compression with error correction. The innovative ideas of Zheng et al. [1] were to apply compression on the parameter server and to use the scale ηt−1ηt in line 7 of Algorithm 1. Unfortunately the scale ηt−1ηt caused an issue in the proof of convergence theorem of Algorithm 1. We examine this issue in details in Section 3.

2.1. Compressor and Assumptions

Following [5,9], an operator C:ℝd→ℝd is a δ-compressor for a number δ∈(0,1) if

EC∥C(x)−x∥2⩽(1−δ)∥x∥2(1)

where the expectation EC is taken over the randomness of C.

Given a loss function ℒ, define f(x) = Eξ[ℒ(x,ξ)] where x∈ℝd is the (neural network) model parameters, and ξ is the data batch drawn from some unknown distribution. We consider the following assumptions on f, which are standard and have been used in previous works [1,9].

Assumption 1.

f is lower-bounded, i.e., f^* = infx∈ℝdf(x)<∞, and L-smooth, i.e., f is differentiable and there exists a constant L ⩾ 0 such that

∥∇f(x)−∇f(y)∥⩽ L∥x−y∥, ∀x,y∈ℝd.(2)

By [10], the L-smooth condition in (2) implies that ∀x,y∈ℝd,

f(x) ⩽ f(y)+〈∇f(y),x−y〉+L2‖x−y‖2.(3)

Assumption 2.

Let E_t denote the expectation at iteration t. Then E_t[g_t,i] = ∇f(x_t) and the stochastic gradient g_t,i has bounded gradient, i.e.,

Et[‖gt,i−∇f(xt)‖2]⩽σ2.

Assumption 3.

The full gradient ∇f is uniformly bounded, i.e., ‖∇f(xt)‖2⩽ω2.

Under Assumptions 2 and 3, we have

Et[‖gt,i‖2]⩽G2=σ2+ω2,(4)

because Et[‖gt,i−∇f(xt)‖2]⩽σ2,‖∇f(xt)‖2⩽ω2, and the fact that E[‖X−E[X]‖2]+‖E[X]‖2=E[‖X‖2].

2.2. Supporting Lemmas

We need a few supporting lemmas for proving Theorem 1.

Lemma 1.

Let 0<M∈ℕ and xi∈ℝd. Then

‖1M∑i=1Mxi‖2⩽1M∑i=1M‖xi‖2.

Proof.

Since xi∈ℝd, x_i has the form xi=(xi,1,xi,2,…,xi,d)∈ℝd. We have

‖1M∑i=1Mxi‖2=1M2‖∑i=1Mxi‖2 =1M2‖(∑i=1Mxi,1,∑i=1Mxi,2,…,∑i=1Mxi,d)‖2 =1M2∑j=1d(∑i=1Mxi,j)2.

Applying the Cauchy–Schwarz inequality on (∑i=1Mxi,j)2 gives us

(∑i=1Mxi,j)2⩽M∑i=1Mxi,j2.

Therefore

‖1M∑i=1Mxi‖2⩽1M∑j=1d(∑i=1Mxi,j2) =1M∑i=1M(∑j=1dxi,j2) =1M∑i=1M‖xi‖2

which ends the proof.

Lemma 2.

Let {a_t}, {α_t}, {β_t} be non-negative sequences in ℝ such that a₀ = 0 and, for all t ⩾ 0,

at+1 ⩽ αtat+βt.(5)

Then

at+1⩽βt+∑j=1t∏i=jtαiβj−1.

In particular, if β_t = β for all t, then

at+1 ⩽ β(1+∑j=1t∏i=jtαi).

Proof.

By (5), we have

a1⩽α0a0+β0=β0.

Proving by induction, assume that then we have

at ⩽ βt−1+∑j=1t−1∏i=jt−1αiβj−1,(6)

then we have

at+1⩽αtat+βt (by (5)) ⩽βt+αt(βt−1+∑j=1t−1∏i=jt−1αiβj−1) (by (6)) =βt+αtβt−1+∑j=1t−1∏i=jtαiβj−1 =βt+∑j=1l∏i=jαiβj−1,

which ends the proof.

3. THE ISSUE IN ZHENG ET AL. [1]

In order to prove the convergence theorem for Algorithm 1, Zheng et al. [1] have used the following lemma.

Lemma A

(Lemma 2 of [1], incorrect). For any t ⩾ 0, e~t,et,i,ηt from Algorithm 1, compressor parameter δ at (1), and gradient bound G at (4),

E[‖e~t+1M∑i=1Met,i‖2]⩽8(1−δ)G2δ2(1+16δ2).

Intuitively, Lemma A can become incorrect because its right-hand side only depends on the gradient bound G and compressor parameter δ, and does not capture the scaling factor η_t−₁/η_t of the errors e~t and e_t,i of Algorithm 1. More formally, the following claim states that Lemma A is invalidated when the learning rate sequence {η_t} is decreasing.

Claim 1.

Lemma A (i.e., Lemma 2 of [1]) does not hold. More precisely, referring to Algorithm 1, there exist a sequence of loss functions ℒ(xt,ξ), a decreasing sequence {ηt}t ⩾−1, a number δ with respect to a compressor C, and a step t such that

E[‖e~t+1M∑i=1Met,i‖2]>8(1−δ)G2δ2(1+16δ2).(7)

Claim 1 is justified by the following counter-examples, in which we intentionally utilize the fact that the quotient η_t−₁/η_t as in line 7 of Algorithm 1 can be large with decreasing learning rate sequences.

Counter-example 1.

(Convex case) For t ⩾ 0 and xt,ξ∈ℝ, we consider the sequence of loss functions

ℒ(xt,ξ)=φ(xt)=14xt

in the constraint set [−1, 1], the decreasing sequence of learning rate {ηt}t⩾−1 with

η−1=0,{ηt=148t+2}t⩾0,

the compressor C:ℝ→ℝ such that ∀x∈ℝ

C(x)=x0.77.

Then at t = 1, Claim 1 holds true.

Proof.

(Justification of Counter-example 1) It is trivial that the loss function ℒ satisfies all the Assumptions 1, 2, and 3. The upper bound gradient of f is G=14 because we have gt,i=14∀t,i.

The function C with C(x)=x0.77 is a compressor with respect to δ = 0.9. Indeed, we have

‖C(x)−x‖2⩽(1−δ)‖x‖2⇔‖x0.77−x‖2⩽(1−δ)‖x‖2⇔‖10.77−1‖2⩽1−δ.

The last inequality is equivalent to

δ⩽1−‖10.77−1‖2=0.9107775341541.

Therefore δ = 0.9 suffices.

To continue, let us consider the number of workers is M = 2. Initially e_0,i = 0 on each worker i ∈ {1,2} and e~0=0 on server. Because the stochastic gradients are the same on each worker, the results of computations on each worker are the same. So it is sufficient to consider the computations on worker 1 in details.

At t = 0 we have the computations on the workers and the server as follows.
- –
  On worker 1:
  p0,1=g0,1+η−1η0e0,1=g0,1=14,Δ0,1=C(p0,1)=p0,10.77=0.3246753246753,e1,1=p0,1−Δ0,1=−0.07467532467532.
- –
  On worker 2, p_0,2 = p_0,1, ∆_0,2 = ∆_0,1, and e_1,2 = e_1,1.
- –
  On server:
  p~0=12(Δ0,1+Δ0,2)+η−1η0e~0 =Δ0,1=0.3246753246753,Δ~0=C(p~0) =p~00.77=0.42165626581210,e~1=p~0−Δ~0=−0.09698094113678.
At t = 1 we have the computations on the workers and the server as follows:
- –
  On worker 1:
  p1,1=g1,1+η0η1e1,1=−1.6168831168831,Δ1,1=C(p1,1)=p1,10.77=−2.0998482037443,e2,1=p1,1−Δ1,1=0.48296508686119.
- –
  On worker 2: p_1,2 = p_1,1, ∆_1,2 = ∆_1,1, and e_2,2 = e_2,1.
- –
  On server:
  p~1=12(Δ1,1+Δ1,2)+η0η1e~1 =Δ1,1+η0η1e~1, =−4.52437173216,Δ~1=C(p~1) =p~10.77=−5.8758074443687,e~2=p~1−Δ~1=1.3514357122048.

Now we compute the left- and right-hand sides of (7) with t = 2.

‖e~2+12(e2,1+e2,2)‖2 =‖e~2+e2,1‖2 =3.365026291613992

and

8(1−δ)G2δ2(1+16δ2)=8(0.1)(14)20.92(1+160.92) =1.2810547172687086.

Thus

‖e~2+12(e2,1+e2,2)‖2>8(1−δ)G2δ2(1+16δ2),

and then Claim 1 follows.

Counter-example 2.

(Convex case) For t ⩾ 0 and x_t,ξ ∈ ℝ, we consider the sequence of loss functions

ℒ(xt,ξ)=φ(xt)=xt2

in the constraint set [−1, 1], the decreasing sequence of learning rate {η_t}_t⩾−1 with

η−1=0,{ηt=34(126t+2)}t⩾0,

and the following compressor C:ℝ→ℝ with parameter δ = 0.9 as in counter-example 1,

C(x)=x0.77.

Then at t = 1, Claim 1 holds true.

Proof.

(Justification of Counter-example 2) It is trivial that the loss function ℒ is 2-smooth and ℒ satisfies all the Assumptions 1, 2, and 3. Since ∇f(x) = 2x and gt,i=2xt,∀t,i, we have the upper bound gradient of f in the constraint set [−1, 1] as G = 2. Let us consider the number of workers M = 2. Initially e_0,i = 0 on each worker i∈{1,2} and e~0=0 server. Let us take x₀ = 1. Because the stochastic gradients are the same on each worker, the results of computations on each worker are the same. So it is sufficient to consider the computations on worker 1 in details.

At t = 0 we have the computations on the workers and the server as follows:
- –
  On worker 1:
  p0,1=g0,1+η−1η0e0,1=g0,1=2,Δ0,1=C(p0,1)=p0,10.77=2.5974025974025974,e1,1=pp0,1−Δ0,1=−0.5974025974025974.
- –
  On worker 2, p_0,2 = p_0,1, ∆_0,2 = ∆_0,1, and e_1,2 = e_1,1.
- –
  On server:
  p~0=12(Δ0,1+Δ0,2)+η−1η0e~0 =Δ0,1=2.5974025974025974,Δ~0=C(p~0) =p~00.77=3.3732501264968797,x1=x0−η0Δ~0 =−0.26496879743632995,e~1=p~0−Δ~0=−0.7758475290942823.
At t = 1 we have the computations on the workers and the server as follows:
- –
  On worker 1:
  g1,1=∇f(x1)=−0.5299375948726599,p1,1=g1,1+η0η1e1,1 =−8.893573958509023,Δ1,1=C(p1,1) =p1,10.77=−11.550096050011717,e2,1=p1,1−Δ1,1=2.656522091502694.
- –
  On worker 2, p_1,2 = p_1,1, ∆_1,2 = ∆_1,1, and e_2,2 = e_2,1.
- –
  On server:
  p~1=12(Δ1,1+Δ1,2)+η0η1e~1 =Δ1,1+η0η1e~1 =−22.41196145733167,Δ~1=C(p~1) =p~10.77=−29.106443451080093,e~2=p~1−Δ~1=6.694481993748422.

Now we compute the left- and right-hand sides of (7) with t = 2. We have

‖e~2+12(e2,1+e2,2)‖2 =‖e~2+e2,1‖2 =87.44127740238307

and

8(1−δ)G2δ2(1+16δ2)=8(0.1)220.92(1+160.92) =81.98750190519735.

Thus

‖e~2+12(e2,1+e2,2)‖2>8(1−δ)G2δ2(1+16δ2),

and then Claim 1 follows.

Counter-example 3.

(nonconvex case) For t ⩾ 0,x_t,ξ ∈ ℝ, we consider the sequence of loss functions

L(xt,ξ)=φ(xt)=11+e−xt,

the decreasing sequence of learning rate {η_t}_t_⩾−1 with

η−1=0,{ηt=32(148t+2)}t⩾0,

and the following compressor C:ℝ→ℝ with parameter δ = 0.9 as in counter-example 1,

C(x)=x0.77.

Then at t = 1, Claim 1 holds true.

Proof.

(Justification of Counter-example 3) First, we check that Assumptions 1-3 are satisfied.

The function ϕ is lower-bounded because 0 ⩽ ϕ(x) ⩽ 1, ∀x ∈ ℝ. The upper bound of ∇ϕ(x) is G=14, since

∇φ(x)⩽14⇔φ(x)(1−φ(x))⩽14 ⇔−φ(x)2+φ(x)−14⩽0 ⇔−(φ(x)−12)2⩽0,(8)

which holds true for all x∈ℝ.

The function ϕ is L-smooth, with L = 1. Indeed, for all x,y∈ℝ, we have

|∇φ(x)−∇φ(y)| =|φ(x)(1−φ(x))−φ(y)(1−φ(y))| =|φ(x)−φ(y)+(φ(y)−φ(x))(φ(y)+φ(x))| =|[φ(x)−φ(y)][1−(φ(x)+φ(y))]|.

Since 0 ⩽ φ(ξ) ⩽ 1 (∀ξ), we have 0 ⩽ φ(x)+ φ(y) ⩽ 2. Therefore −1 ⩽ 1−(φ(x)+φ(y)) ⩽ 1, and hence

|[φ(x)−φ(y)][1−(φ(x)+φ(y))]|⩽|φ(x)−φ(y)|.

This means that in order to prove |∇φ(x)−∇φ(y)|⩽ |x−y|, it is sufficient to prove

|φ(x)−φ(y)|⩽|x−y|.(9)

If x ⩾ y, we obtain ϕ(x) ⩾ ϕ(y). Therefore

(9)⇔φ(x)−φ (y) ⩽ x−y ⇔φ(x)−x ⩽ φ(y)−y

Let φ(x) = ϕ(x)−x. Because ∇ϕ(x)=∇φ(x)−1 ⩽ 14−1<0 by (8), we have φ(x) is a decreasing function. Therefore the inequality φ(x)−x ⩽ φ(y)−y holds true, and hence (9) is proven. By the same technique, we obtain (9) for the case x < y.

To continue, let us consider the number of workers M = 2. We initialize e_0;i = 0 on each worker i∈{1,2} and e~0=0 on server. Let us take x₀ = 0. Because the stochastic gradients are the same on each worker, the results of computations on each worker are the same. So it is sufficient to consider the computations on worker 1 in details. We have

g0,1=g0,2=∇φ(x0)=0.25.

At t = 0 we have the computations on the workers and the server as follows:
- –
  On worker 1:
  p0,1=g0,1+η−1η0e0,1 =g0,1=0.25,Δ0,1=C(p0,1) =p0,10.77=0.3246753246753247,e1,1=p0,1−Δ0,1=−0.07467532467532467.
- –
  On worker 2, p_0,2 = p_0,1, ∆_0,2 = ∆_0,1, and e_1,2 = e_1,1.
- –
  On server:
  p~0=12(Δ0,1+Δ0,2)+η−1η0e~0 =Δ0,1=0.3246753246753247,Δ~0=C(p~0) =p~00.77=0.42165626581210996,x1=x0−η0Δ~0 =−0.3162421993590825,e~1=p~0−Δ~0=−0.09698094113678529.
At t = 1 we have the computations on the workers and the server as follows.
- –
  On worker 1:
  g1,1=∇φ(x1)=0.243852158038919p1,1=g1,1+η0η1e1,1 =−1.6230309588441978,Δ1,1=C(p1,1) =p1,10.77=−2.1078324140833735,e2,1=p1,1−Δ1,1=0.4848014552391757.
- –
  On worker 2, p_1,2 = p_1,1, ∆_1,2 = ∆_1,1, and e_2,2 = e_2,1.
- –
  On server:
  p~1=12(Δ1,1+Δ1,2)+η0η1e~1 =Δ1,1+η0η1e~1 =−4.532355942503006,Δ~1=C(p~1) =p~10.77=−5.886176548705203,e~2=p~1−Δ~1=1.3538206062021967.

Now we compute the left- and right-hand sides of (7) with t = 2. We have

‖e~2+12(e2,1+e2,2)‖2=‖e~2+e2,1‖2 =3.3805310848189216

and, with δ = 0.9,

8(1−δ)G2δ2(1+16δ2)=8(0.1)(14)20.92(1+160.92) =1.2810547172687086.

Thus

‖e~2+12(e2,1+e2,2)‖2>8(1−δ)G2δ2(1+16δ2),

and hence Claim 1 follows.

4. CORRECTING THE ERROR BOUND OF ZHENG ET AL. [1]

In general, the error E[‖e~t+1+1M∑i=1Met+1,i‖2] is bounded as follows:

Theorem 2.

(Fix for Lemma 2 of [1]) With e~t,et,i,ηt, and δ from Algorithm 1, for arbitrary {ηt}, we have

E[‖e~t+1+1M∑i=1Met+1,i‖2]2] ⩽ 2(1−δ)(2−δ)G2δ∑k=0tηt−k2ηt2αk +4(1−δ)(2−δ)3G2δ2∑j=0tαt−j∑k=0jηj−k2ηt2αk,

where α=1−δ2 and gradient bound G is at (4).

Remark 1.

[Sanity check of the new upper bound] The right-hand side of Theorem 2 can become large together with decreasing leaning rate sequences. Therefore, the error bounds of the sequences in counter-examples 1-3 do satisfy Theorem 2. Indeed, the upper bound on the error in Theorem 2 at t = 1 is

U=2(2−δ)(1−δ)G2δ(1+η02η12(1−δ2)) +4(1−δ)(2−δ)3G2δ2(1+2η02η12(1−δ2)).

Concretely, at sanity check,

in counter-example 1:
U=33.550763888888895
which is indeed larger than
‖e~2+1M∑i=1Me2,i‖2=3.365026291613992.
in counter-example 2:
U=675.8530370370372
which is indeed larger than
‖e~2+1M∑i=1Me2,i‖2=87.44127740238307.
in counter-example 3:
U=33.550763888888895
which is indeed larger than
‖e~2+1M∑i=1Me2,i‖2=3.3805310848189216.

Proof.

[Proof of Theorem 2] We have

E[‖e~t+1+1M∑i=1Met+1,i‖2] ⩽ 2E[‖e~t+1‖2]+2E[‖1M∑i=1Met+1,i‖2] ⩽ 2E[‖e~t+1‖2]+2M∑iME[‖et+1,i‖2],(10)

where the first inequality is by the fact that (a + b)² ⩽ 2a² + 2b²,∀a,b, and the second inequality is by Lemma 1. We will separately bound the two terms of (10). Firstly, we consider 1M∑i=1ME[‖et+1,i‖2]. We have

1M∑i=1ME[‖et+1,i‖2]=1M∑i=1ME[‖C(pt,i)−pt,i‖2(11)

⩽ 1−δM∑i=1ME[‖pt,i‖2](12)

= 1−δM∑i=1ME[‖gt,i+ηt−1ηtet,i‖2](13)

⩽(1−δ)(1+γ)M∑i=1ME[‖ηt−1ηtet,i‖2] +(1−δ)(1+1/γ)M∑i=1ME[‖gt,i‖2](14)

⩽(1−δ)(1+γ)ηt−12ηt2(1M∑i=1ME[‖et,i‖2]) +(1−δ)(1+1/γ)G2,(15)

where (11) and (13) is by the setting of e_t_+1,i and p_t,i in Algorithm 1, (12) is by the definition of compressor C, (14) is by Young inequality with any γ > 0, and (15) is by (4). Now, for all t ⩾ 0, applying Lemma 2 to the inequality (15) with

at+1=1M∑i=1ME[‖et+1,i‖2],αt=(1−δ)(1+γ)ηt−12ηt2,β=(1−δ)(1+1/γ)G2,

we have

1M∑i=1ME[‖et+1,i‖2]⩽β(1+∑j=1t∏i=jtαi)

Moreover, since

1+∑j=1t∏i=jtαi = 1+∑j=1t∏i=jt(1−δ)(1+γ)ηi−12ηi2 = 1+∑j=1tηj−12ηt2[(1−δ)(1+γ)]t−(j−1) = ∑j=1t+1ηj−12ηt2[(1−δ)(1+γ)]t−(j−1) = ∑k=0tηt−k2ηt2[(1−δ)(1+γ)]k,

we obtain

1M∑i=1ME[‖et+1,i‖2]⩽ (1−δ)(1+1/γ)G2∑k=0tηt−k2ηt2[(1−δ)(1+γ)]k.

By choosing γ=δ2(1−δ), we have

(1−δ)(1+1/γ)=(1−δ)(2−δ)δ

and

(1−δ)(1+γ)=1−δ2.

Therefore

1M∑i=1ME[‖et+1,i‖2] ⩽ (2−δ)(1−δ)G2δ∑k=0tηt−k2ηt2αk,(16)

where α=1−δ2. Next, we consider the term E[‖e~t+1‖2] of (10). By the setting of e_t_+1,i and p_t,i in Algorithm 1 and the definition of compressor C, we have

E[‖e~t+1‖2]=E[‖C(p~t)−p~t‖2]⩽(1−δ)E[‖p~t‖2]=(1−δ)E[‖1M∑i=1MC(pt,i)+ηt−1ηte~t‖2]⩽(1−δ)(1+γ)ηt−12ηt2E[‖e~t‖2]+(1−δ)(1+1/γ)E[‖1M∑i=1MC(pt,i)‖2],

where the last inequality is by Young inequality for any γ>0. Looking at E[‖1M∑i=1MC(pt,i)‖2], we have

E[‖1M∑i=1MC(pt,i)‖2]⩽1M∑i=1ME[‖C(pt,i)‖2](17)

⩽1M∑i=1M(2E[‖C(pt,i)−pt,i‖2]+2E[‖pt,i‖2])(18)

⩽1M∑i=1M(2(1−δ)E[‖pt,i‖2]+2E[‖pt,i‖2])=2(2−δ)1M∑i=1ME[‖pt,i‖2],(19)

where (17) is by Lemma 1, (18) is by the fact that (a + b)² ⩽ 2a² + 2b²,∀a,b, (19) is by the definition of compressor C. Therefore Moreover, (12) and (16) yield. Therefore

E[‖e~t+1‖2]⩽(1−δ)(1+γ)ηt−12ηt2E[‖e~t‖2]+2(2−δ)(1−δ)(1+1/γ)1M∑i=1ME[‖pt,i‖2].

Moreover, (12) and (16) yield

1−δM∑i=1ME[‖pt,i‖2]⩽(2−δ)(1−δ)G2δ∑k=0tηt−k2ηt2αk.

Therefore

E[‖e~t+1‖2]

⩽(1−δ)(1+γ)ηt−12ηt2E[‖e~t‖2] +2(2−δ)2(1−δ)(1+1/γ)G2δ∑k=0tηt−k2ηt2αk.

With γ=δ2(1−δ), since (1−δ)(1+1/γ)=(1−δ)(2−δ)δ and (1−δ)(1+γ)=1−δ2=α, we have

E[‖e~t+1‖2]⩽αηt−12ηt2E[‖e~t‖2] + 2(1−δ)(2−δ)3G2δ2∑k=0tηt−k2ηt2αk.

By applying Lemma 2 with

at=E[‖e~i‖2],αt=αηt−12ηt2,βt=2(1−δ)(2−δ)3G2δ2∑k=0tηt−k2ηt2αk,

we obtain

E[‖e~t+1‖2]⩽βt+∑j=1t(∏i=jtαiβj−1).

Since

∏i=jtαiβj−1=∏i=jtαηi−12ηi2βj−1 =αt−(j−1)ηj−12ηt2βj−1,

we have

E[‖e~t+1‖2]⩽βt+∑j=1tαt−(j−1)ηj−12ηt2βj−1 =∑j=1t+1αt−(j−1)ηj−12ηt2βj−1.

Therefore,

E[‖e~t+1‖2]⩽ ∑j=1t+1αt−(j−1)ηj−12ηt22(1−δ)(2−δ)3G2δ2∑k=0j−1ηj−1−k2ηj−12αk=2(1−δ)(2−δ)3G2δ2∑j=1t+1αt−(j−1)∑k=0j−1ηj−1−k2ηt2αk=2(1−δ)(2−δ)3G2δ2∑j=0tαt−j∑k=0jηj−k2ηt2αk.(20)

Substituting (16) and (20) to (10), we obtain

E[‖e~t+1+1M∑i=1Met+1,i‖2]⩽ 2(2−δ)(1−δ)G2δ∑k=0tηt−k2ηt2αk +4(1−δ)(2−δ)3G2δ2∑j=0tαt−j∑k=0jηj−k2ηt2αk,

as claimed in Theorem 2.

As a sanity check, Theorem 2 matches the results given in [1] when the learning rate is nondecreasing. As a result, Theorems 1 and A agree when the learning rate is nondecreasing.

Corollary 1.

(Sanity check of Theorem 2, cf. Lemma 6 of [1] with μ = 0) In Theorem 2, if {η_t} is a nondecreasing sequence such that ηt>0,∀t ⩾ 0, then

E[‖e~t+1+1M∑i=1Met+1,i‖2]⩽8(1−δ)G2δ2(1+16δ2).

Proof.

Since {η_t} is nondecreasing, we have ηt−k2ηt2⩽1. Moreover, since α=1−δ2∈(0,1), we obtain

∑k=0tηt−k2ηt2αk⩽∑k=0tαk⩽2δ(21)

and

∑j=0tαt−j∑k=0jηj−k2ηt2αk⩽∑j=0tαt−j∑k=0jαk ⩽∑j=0tαt−j(2δ) ⩽4δ2.

Replacing (21) and (22) to Theorem 2, we have the result stated in Corollary 1.

5. CORRECTING THE CONVERGENCE THEOREM OF ZHENG ET AL. [1]

Because the error bound plays a crucial role in the proof of the convergence theorem of dis-EF-SGD, fixing [1, Lemma 2] as in Theorem 2 leads to the consequence that the convergence theorem need to be fixed as well.

Proof.

(Proof of Theorem 1) Following [1], we consider the iteration

x~t=xt−ηt−1(e~t+1M∑i=1Met,i),

where x_t, e~t and e_t,i are generated from Algorithm 1. Then, by [1, Lemma 1],

x~t+1=x~t−ηt1M∑i=1Mgt,i.(22)

Since f is L-smooth, by (3), we have Moreover, we have

Et[f(x~t+1)] ⩽ f(x~t)+〈∇f(x~t),Et[x~t+1−x~t]〉 +L2Et[‖x~t+1−x~t‖2] =f(x~t)−ηt〈∇f(x~t),Et[1M∑i=1Mgt,i]〉 +Lηt22Ef[‖1M∑i=1Mgt,i‖2],(23)

Moreover, we have

Et[‖1M∑i=1Mgt,i‖2]=‖∇f(xt)‖2+Et[‖1M∑i=1Mgt,i−∇f(xt)‖2],(24)

which follows from the fact that E[∥X−E[X]∥2]=E[∥X∥2]−∥E[X]∥2. Substituting (24) to (23), we obtain

Et[f(x~t+1)] ⩽ f(x~t)−ηt〈∇f(x~t),∇f(xt)〉 +Lηt22‖∇f(xt)‖2 +Lηt22Et[‖1M∑i=1Mgt,i−∇f(xt)‖2].(25)

Following [1], we assume that {gt,i−∇f(xt)}1⩽i⩽M are independent random vectors. Then the assumption Et[gt,i]=∇f(xt) of Assumption 2 implies that g_t,i − ∇ f (x_t) are random vectors with 0 means. Therefore

Et[‖∑i=1M(gt,i−∇f(xt))‖2]=∑i=1MEt[‖gt,i−∇f(xt)‖2]

and hence we have

Et[‖1M∑i=1Mgt,i−∇f(xt)‖2] = Et[‖1M∑i=1M(gt,i−∇f(xt))‖2] = 1M2Et[‖∑i=1M(gt,i−∇f(xt))‖2] = 1M2∑i=1MEt[‖gt,i−∇f(xt)‖2] ⩽ σ2MM2=σ2M.

Substituting the above bound to (25) gives us

Et[f(x~t+1)] ⩽f(x~t)−ηt〈∇f(x~t),∇f(xt)〉 +Lηt22‖∇f(xt)‖2+Lηt2σ22M.(26)

Moreover, we have

−ηt〈∇f(x~t),∇f(xt)〉=ηt〈∇f(xt)−∇f(x~t),∇f(xt)〉−ηt〈∇f(xt),∇f(xt)〉=ηt〈∇f(xt)−∇f(x~t),∇f(xt)〉−ηt‖∇f(xt)‖⩽ηtρ2‖∇f(xt)‖2+ηt2ρ‖∇f(xt)−∇f(x~t)‖2 −ηt‖∇f(xt)‖2(27)

⩽−ηt(1−ρ2)‖∇f(xt)‖2+ηtL22ρ‖xt−x~t‖2 ,(28)

where (27) is by the fact that 〈a,b〉 ≤ (ρ/2) ‖a‖ 2 + (ρ⁻¹/2)‖b‖² for all a,b and real number ρ > 0, and (28) is by Assumption 1. Replacing (28) to (26) gives us

Et[f(x~t+1)]⩽f(x~l)−ηt(1−Lηt+ρ2)‖∇f(xl)‖2 +ηtL22ρ‖xt−x~t‖2+Lηt2σ22M.

Taking ρ=12, we have

Et[f(x~t+1)]⩽f(x~t)−ηt(34−Lηt2)‖∇f(xt)‖2 +ηtL2‖xt−x~t‖2+Lηt2σ22M.(29)

Since xt−x~t=ηt−1(e~t+1M∑i=1Met,i) by (22), after rearranging the terms and taking total expectation, we obtain

ηt(34−Lηt2)E[‖∇f(xt)‖2] ⩽ E[f(x~t)−f(x~t+1)]+Lηt2σ22M + ηtηt−12L2E[‖e~t+1M∑i=1Met,i‖2] .

Applying Theorem 2 gives us

ηt(34−Lηt2)E[‖∇f(xt)‖2](30)

⩽E[f(x~t)−f(x~t+1)]+Lηt2σ22M+ηtηt−122(2−δ)(1−δ)G2L2δ∑k=0t−1ηt−1−k2ηt−12αk+ηtηt−124(1−δ)(2−δ)3G2L2δ2∑j=0t−1αt−1−j∑k=0jηj−k2ηt−12αk,

where α=1−δ2. Since ηt<32L,∀t, we have

∑k=0T−1ηk4(3−2Lηk)>0.

Taking summation and dividing by ∑k=0T−1ηk4(3−2Lηk), (30) yields

∑t=0T−1ηt(3−2Lηt)E[‖∇f(xt)‖2]∑k=0T−1ηk(3−2Lηk)⩽4(f(x0)−f⋆)∑k=0T−1ηk(3−2Lηk)+2Lσ2M∑t=0T−1ηt2∑k=0T−1ηk(3−2Lηk)+8(2−δ)(1−δ)G2L2δ∑k=0T−1ηk(3−2Lηk)∑t=0T−1ηtηt−12∑k=0t−1ηt−1−k2ηt−12αk+16(1−δ)(2−δ)3G2L2δ2∑k=0T−1ηk(3−2Lηk)× ∑t=0T−1ηtηt−12∑j=0t−1αt−1−j∑k=0jηj−k2ηt−12αk.

Following Zheng et al. [1], let o ∈ {0, …, T − 1} be an index such that

Pr(o=k)=ηk(3−2Lηk)∑t−0T−1ηt(3−2Lηt)

Then

E[‖∇f(xo)‖2]=1∑k=0T−1ηk(3−2Lηk)∑t=0T−1ηt(3−2Lηt)E[‖∇f(xt)‖2]

and we obtain the result stated in Theorem 1

The following corollary establishes the convergence rate O(1MT) of Algorithm 1 when the learning rate is decreasing.

Corollary 2.

(Convergence rate with decreasing learning rate) Under the assumptions of Theorem 1, if {ηt} is a decreasing sequence such that

ηt=1((t+1)T)1/4M+T1/3

with sufficiently large T. Then

E[‖∇f(xo)‖2] ⩽2(1MT+1T2/3)[f(x0)−f⋆+Lσ2+4(1−δ)(2−δ)G2L2δ2(1+4δ2),

which yields E[‖∇f(xo)‖2]⩽O(1MT).

Proof.

Following [1], assume that T ⩾ 16L4M2, we have

ηt=1((t+1)T)1/4M+T1/3⩽M((t+1)T)1/4 ⩽M(t+1)1/4(16L4M2)1/4 =1(t+1)1/42L⩽12L.

Therefore ηt<32L∀t⩾0, which satisfies the assumption of Theorem 1 on {ηt}. Recall that by Theorem 1, we have

E[‖∇f(xo)‖2]⩽4(f(x0)−f⋆)∑k=0T−1ηk(3−2Lηk)+2Lσ2M∑k=0T−1ηk(3−2Lηk)∑t=0T−1ηt2 +8(1−δ)(2−δ)G2L2δ∑k=0T−1ηk(3−2Lηk)∑t=0T−1ηtηt−12∑k=0t−1ηt−1−k2ηt−12αk +16(1−δ)(2−δ)3G2L2δ2∑k=0T−1ηk(3−2Lηk)× ∑t=0T−1ηtηt−12∑j=0t−1αt−1−j∑k=0jηj−k2ηt−12αk,(31)

where α=1−δ2 and o∈{0,…,T−1} is an index such that

Pr(o=k)=ηk(3−2Lηk)∑k=0T−1ηt(3−2Lηt),∀k=0,…,T−1.

Since

3−2Lηt⩾3−1(t+1)1/4⩾2,

we have

1∑k=0T−1ηt(32Lηt)⩽12∑k=0T−1ηt.(32)

Moreover, we have

ηtηt−12∑k=0t−1ηt−1−k2ηt−12αk=∑k=0t−1ηtηt−1−k2αk(33)

and

ηtηt−12∑j=0t−1αt−1−j∑k=0jηj−k2ηt−12αk =∑j=0t−1αt−1−j∑k=0jηtηj−k2αk.(34)

Substituting (32), (33), and (34) to (31) gives us

E[‖∇f(xo)‖2]⩽ 2(f(x0)−f⋆)∑t=0T−1ηt+Lσ2M∑t=0T−1ηt∑t=0T−1ηt2 +4(1−δ)(2−δ)G2L2δ∑t=0T−1ηt∑t=0T−1∑k=0t−1ηtηt−1−k2αk +8(1−δ)(2−δ)3G2L2δ2∑t=0T−1ηt× ∑t=0T−1∑j=0t−1αt−1−j∑k=0jηtηj−k2αk,(35)

Because

ηtηt−1−k2⩽ηt−1−k3 =1(((t−k)T)1/4M+T1/3)3 ⩽1(T1/3)3=1T(36)

and

∑k=0t−1αk⩽∑k⩾0αk=11−α=2δ,(37)

we obtain

∑t=0T−1∑k=0t−1ηtηt−1−k2αk⩽1T∑t=0T−1∑k=0t−1αk ⩽1T∑t=0T−12δ=2δ.

By the same reason as in (36) and (37), we have ηtηj−k2⩽1T, ∑j=0t−1αt−1−j⩽2δ, and ∑k= 0jαk⩽2δ. Therefore

∑t=0T−1∑j=0t−1αt−1−j∑k=0jηtηj−k2αk ⩽1T∑t=0T−1∑j=0t−1αt−1−j∑k=0jαk ⩽1T∑t=0T−1∑j=0t−1αt−1−j(2δ) ⩽1T∑t=0T−14δ2=4δ2.

Moreover, we have

∑t=0T−1ηt2=∑t=0T−11(((t+1)T)1/4M+T1/3)2 ⩽∑t=0T−11(((t+1)T)1/4M)2 =∑t=0T−1M[(t+1)T]1/2 =MT∑t=1T1t ⩽2M,

where the last inequality is by the fact that ∑t=1T1t ⩽ 2T. Therefore

E[‖∇f(xo)‖2]⩽2(f(x0)−f⋆)∑t=0T−1ηt+2Lσ2∑t=0T−1ηt +8(1−δ)(2−δ)G2L2δ2∑t=0T−1ηt +32(1−δ)(2−δ)3G2L2δ4∑t=0T−1ηt.

Furthermore, becauce

∑t=0T−1ηt=∑t=0T−11((t+1)T)1/4(M)+T1/3⩾∑t=0T−11TM+T1/3 =11MT+T−2/3,

we obtain

1∑t=0T−1ηt⩽1MT+1T2/3.

Therefore

E[‖∇f(xo)‖2] ⩽ 2(1MT+1T2/3)f(x0)−f⋆+Lσ2 +4(1−δ)(2−δ)G2L2δ2(1+4δ2)

and hence Corollary 2 follows.

6. CONCLUSION

We show that the convergence proof of dist-EF-SGD of Zheng et al. [1] is problematic when the sequence of learning rate is decreasing. We explicitly provide counter-examples with certain decreasing sequences of learning rate to show the issue in the proof of Zheng et al. [1]. We fix the issue by providing a new error bound and a new convergence theorem for the dist-EF-SGD algorithm, which helps recover its mathematical foundation.

CONFLICTS OF INTEREST

The authors declare that there are no conflicts of interest.

AUTHORS' CONTRIBUTIONS

Tran Thi Phuong established the research direction. Both authors contributed to the technical contents, and to the edition of the manuscript. Both authors read, revised, and approved the final manuscript.

ACKNOWLEDGMENTS

We are grateful to Shuai Zheng for his communication and verification. We also thank the anonymous reviewers for their careful comments. The work of Le Trieu Phong was supported in part by JST CREST under Grant JPMJCR19F6.

REFERENCES

1.S. Zheng, Z. Huang, and J.T. Kwok, Communication-efficient distributed blockwise momentum SGD with error-feedback, in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS 2019) (Vancouver, Canada), 2019, pp. 11446-11456. https://arxiv.org/abs/1905.10936

2.J. Bernstein, J. Zhao, K. Azizzade-nesheli, and A. Anandkumar, signSGD with majority vote is communication efficient and fault tolerant, in 7th International Conference on Learning Representations (ICLR 2019) (New Orleans, LA, USA), 2019.

3.D. Basu, D. Data, C. Karakus, and S.N. Diggavi, Qsparse-local-SGD: distributed SGD with quantization, sparsification and local computations, in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS 2019) (Vancouver, Canada), 2019, pp. 14668-14679.

4.T. Vogels, S.P. Karimireddy, and M. Jaggi, PowerSGD: practical low-rank gradient compression for distributed optimization, Curran Associates, Inc, in Advances in Neural Information Processing Systems (Vancouver, Canada), 2019, pp. 14236-14245.

5.S.U. Stich, J.-B. Cordonnier, and M. Jaggi, Sparsified SGD with memory, in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems (Montréal, Canada), 2018, pp. 4452-4463.

6.H. Tang, C. Yu, X. Lian, T. Zhang, and J. Liu, DoubleSqueeze: parallel stochastic gradient descent with double-pass error-compensated compression, K. Chaudhuri and R. Salakhutdinov (editors), Proceedings of the 36th International Conference on Machine Learning, vol. 97 of Proceedings of Machine Learning Research (PMLR), PMLR, Long Beach, California, USA, 2019, pp. 6155-6165.

7.X. Liu, Y. Li, J. Tang, and M. Yan, A double residual compression algorithm for efficient distributed learning, in Proceedings of Machine Learning Research (PMLR) (Palermo, Italy), 2020, pp. 133-143.

8.T.T. Phuong and L.T. Phong, Distributed SGD with flexible gradient compression, IEEE Access, Vol. 8, 2020, pp. 64707-64717.

9.S.P. Karimireddy, Q. Rebjock, S.U. Stich, and M. Jaggi, Error feedback fixes signSGD and other gradient compression schemes, in Proceedings of the 36th International Conference on Machine Learning (ICML 2019) (Long Beach, California, USA), 2019, pp. 3252-3261. https://arxiv.org/abs/1901.09847

10.Y. Nesterov, Introductory Lectures on Convex Optimization, Springer Science and Business Media, Boston, MA, USA, Vol. 87, 2004.

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Journal: International Journal of Computational Intelligence Systems
Volume-Issue: 14 - 1
Pages: 1373 - 1387
Publication Date: 2021/04/20
ISSN (Online): 1875-6883
ISSN (Print): 1875-6891
DOI: 10.2991/ijcis.d.210412.001 How to use a DOI?
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

ris enw bib

TY  - JOUR
AU  - Tran Thi Phuong
AU  - Le Trieu Phong
PY  - 2021
DA  - 2021/04/20
TI  - Communication-Efficient Distributed SGD with Error-Feedback, Revisited
JO  - International Journal of Computational Intelligence Systems
SP  - 1373
EP  - 1387
VL  - 14
IS  - 1
SN  - 1875-6883
UR  - https://doi.org/10.2991/ijcis.d.210412.001
DO  - 10.2991/ijcis.d.210412.001
ID  - Phuong2021
ER  -

download .riscopy to clipboard