Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm

Caesar Wu; Hongyang Li; Lincen Bai; Mohammed Chadli; Pascal Bouvry; Said Mammar

REVIEW 3 major objections 6 minor 20 references

Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LQ-SGD layers 8-bit log quantization on PowerSGD's low-rank factors, cutting gradient traffic 75% at rank 1 while keeping accuracy within 1.6 points.

desk verdict A plausible but underspecified PowerSGD variant: the log-quantization formula contradicts the stated normalization, and without code or hyperparameters the headline numbers cannot be verified. read the letter →

arxiv 2506.17974 v1 pith:7LQATNQS submitted 2025-06-22 cs.LG

Hongyang Li , Lincen Bai , Caesar Wu , Mohammed Chadli , Said Mammar , Pascal Bouvry This is my paper · ORCID

classification cs.LG

keywords distributedlearninggradientcompressionlow-rankapproximationlogarithmicquantizationcommunicationefficiencyerrorfeedbackinversionattacktrustworthiness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LQ-SGD, a gradient compression method that combines PowerSGD's low-rank factorization with logarithmic quantization. It claims that quantizing the two low-rank factors to 8 bits before transmission reduces communication by 75% relative to PowerSGD at rank 1 (3 MB vs 14 MB per epoch on CIFAR-10) while keeping Top-1 accuracy within a few points (92.9% vs 94.5%). On ImageNet, rank 7 LQ-SGD matches plain SGD's 75% Top-1 accuracy within 300 epochs. The paper also reports that compressed gradients, including LQ-SGD's, resist gradient inversion attacks better than uncompressed gradients, offering a privacy benefit alongside efficiency.

What carries the argument

The central mechanism is the composition of PowerSGD's low-rank approximation $G \approx P Q^{\top}$ with a logarithmic quantization step applied independently to $P$ and $Q$ before All-Reduce, plus an error feedback loop that computes $E_t = G'_t - \hat{G}_t$ and reinjects it into the next iteration. The log-quantizer's curvature parameter $\alpha$ assigns higher precision to small gradient values, which the paper argues dominate typical heavy-tailed gradient distributions, while error feedback compensates for the distortion introduced by both low-rank truncation and quantization, keeping the model update $w_{t+1} = w_t - \eta \hat{G}_t$ on track.

What would settle it

Train a transformer-based language model (e.g., GPT-2 scale) from scratch with LQ-SGD at rank 1, $b = 8$ and a standard hyperparameter schedule; if the loss diverges or final perplexity degrades by more than a pre-specified margin relative to PowerSGD at the same rank, the error-feedback assumption fails. A cheaper check: record the Frobenius norm of the error feedback matrix $E_t$ over training; if it grows or oscillates instead of decaying, the reconstruction error is not being 'gradually recovered' as Section IV-B claims.

Watch

Extended reading notes

Core claim

LQ-SGD extends PowerSGD by mapping each entry of the low-rank factors $P$ and $Q$ through the log-quantizer $q(x) = \mathrm{sign}(x) \cdot \frac{\log(1 + \alpha |x|)}{\log(1 + \alpha)}$, then encoding normalized values into $2^b$ uniformly spaced bins ($b = 8$ in experiments) and transmitting only the $b$-bit indices instead of 32-bit floats. Error feedback adds the reconstruction residual to the next gradient, and $Q$ is warm-started from the previous iteration. The central discovery is that this combination reduces per-iteration communication to $r(n+m) \times b$ bits, a factor of $32/b$ smaller than PowerSGD, while retaining convergence speed: at rank 1 on CIFAR-10, LQ-SGD reaches 92.9% accuracy versus 94.5% for PowerSGD, and at rank 7 on ImageNet it matches OriginalSGD at 75% Top-1. A secondary discovery is that gradient compression itself raises resistance to gradient inversion, with LQ-SGD yielding lower SSIM between reconstructed and original images than vanilla SGD.

Load-bearing premise

The paper assumes that error feedback, which is proven to fix unbiased or bounded-error compression schemes, still guarantees convergence when PowerSGD's $P$ and $Q$ factors are quantized by a deterministic nonlinear logarithmic map; Section IV-B states this adoption without providing a proof or variance bound, and the quantization error is neither unbiased nor explicitly bounded.

Editorial extensions

If this is right

Per-epoch gradient bytes drop by a factor of $32/b$ relative to PowerSGD; with $b = 8$ that is a 75% cut, preserving the $r(n+m)$ message size structure.
On CIFAR-10, CIFAR-100, and MNIST, rank 1 LQ-SGD stays within roughly 1.6 to 2.3 accuracy points of PowerSGD and outperforms TopK-SGD at the same compression rate.
On ImageNet, rank 7 LQ-SGD reaches 75% Top-1 accuracy within 300 epochs, matching OriginalSGD's convergence curve.
Compressed gradients give lower SSIM under gradient inversion than vanilla SGD, so LQ-SGD provides privacy defense as a by-product of compression.
The added quantization and dequantization cost is $O(r(n+m))$, negligible next to PowerSGD's $O(nmr)$ matrix products, so the traffic reduction does not shift the compute bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the heavy-tailed gradient rationale is correct, LQ-SGD should transfer to transformer and large-language-model training, where gradient distributions are also heavy-tailed; the paper leaves this untested, so a natural next step is a GPT-style pretraining run at ranks 1 through 7.
The privacy benefit likely strengthens as the bit budget shrinks, since SSIM decreases with more aggressive compression; a testable extension is sweeping $b$ from 8 down to 4 or 2 while tracking both accuracy and SSIM to find the best privacy/accuracy operating point.
The missing convergence proof suggests a concrete theoretical task: bound the deterministic quantization error of the log-map under common gradient distributions and extend the error-feedback analysis of PowerSGD to the composed low-rank plus log-quantization setting.
The paper's premise that communication dominates total time implies a 75% traffic cut roughly halves the communication portion of wall-clock time, but end-to-end timing is not reported; measuring it directly would corroborate the practical speedup claim.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: LQ-SGD is a sensible incremental combination—PowerSGD's low-rank factors plus logarithmic quantization—and the headline numbers (about 4x less communication than PowerSGD at rank 1, with a small accuracy dip) are plausible. But the paper as written has a load-bearing specification error: Eq. (5) is not bounded by 1, so the claimed [0,1] normalization and the binning in Section IV-A do not follow, and Algorithm 1 is therefore not reproducible. No code, no seeds, no alpha, no ImageNet training details. That is a major-revision situation, not a reject-without-review.

What is actually new: the specific combination of log-quantizing both P and Q after orthonormalization, with error feedback, and the SSIM-based GIA evaluation across compression baselines. The individual ingredients are all in the cited literature, but the combination is not. The paper honestly lists its own limitations (image classification only, no large architectures).

Soft spots, in order of severity: - The quantization formula. The stress-test is right: for |x|>1, q(x)>1. The text says normalized values are in [0,1] and mapped to uniform bins; that only works if there is an implicit normalization by max magnitude, which is not stated. Q_t entries can easily exceed 1, so the algorithm as written would saturate. Alpha is never reported, so even the curvature is unknown. - Empirical artifacts. Single-run numbers, no standard deviations, no seeds, no code, no hyperparameter table. The ImageNet claim (Rank 7 matching OriginalSGD at 75% Top-1 at 300 epochs) is stated in one sentence and a figure, with no learning-rate schedule, batch size, or augmentation defaults. - Theory. The paper borrows error-feedback convergence from EF-SGD/PowerSGD, but deterministic log-quantization of both factors is not covered by those proofs. That is fine for an empirical paper if the experiments are solid, but here they are not yet verifiable.

The citation pattern is okay; the self-citation to the authors' earlier trustworthiness paper is relevant and not self-serving.

Who should read this: anyone working on practical gradient compression who wants to see a plausible variant of PowerSGD. It deserves a serious referee because the idea is worth checking and the errors are fixable, but in its current form I would not cite it or rely on the numbers.

Recommendation: send to peer review, but the first decision should be major revision with a hard requirement to (a) fix or clarify the quantizer (preferably add a normalization step and report alpha), (b) release code and seeds, and (c) give complete ImageNet training details.

Referee Report

3 major / 6 minor

Summary. The paper proposes LQ-SGD, a distributed gradient compression method that combines PowerSGD's low-rank factorization with logarithmic quantization and an error-feedback mechanism. The central claim is that this combination drastically reduces communication overhead while preserving convergence speed and model accuracy: on CIFAR-10 at rank 1 it transmits 3 MB versus 14 MB for PowerSGD (a 75% reduction) with 92.9% Top-1 accuracy versus 94.5%, and on ImageNet the authors state that Rank 7 matches OriginalSGD at 75% Top-1 after 300 epochs. The paper also reports SSIM-based experiments suggesting that compression-based methods, including LQ-SGD, are more resistant to gradient inversion attacks than vanilla SGD. The evaluation covers MNIST, CIFAR-10, CIFAR-100, and ImageNet with ResNet-18.

Significance. If the algorithm were fully specified and the experimental results reproducible, the contribution would be a moderate, incremental improvement over PowerSGD: adding a logarithmic quantization stage to an existing low-rank compressor, with a plausible communication/accuracy trade-off and an interesting (though preliminary) privacy-resistance observation. The paper is honest about its limitations in Section VI, explicitly noting that evaluation is restricted to image classification and has not been extended to large transformer-based models. However, the current manuscript has a load-bearing specification gap in the quantizer (Eq. (5) is not bounded as claimed) and the experimental evidence is presented as single-run point estimates without hyperparameters, seeds, or code. These issues prevent the central claims from being verified as written. The idea is salvageable with a corrected quantizer and a more rigorous empirical protocol, but the present version is not yet reproducible.

major comments (3)

[Section IV-A, Eqs. (5)–(6), and Algorithm 1] The quantizer defined in Eq. (5) is not normalized: for any input with |x|>1, z=q(x) has |z|>1, which contradicts the assertion in Section IV-A that the normalized quantized values satisfy |q(x)|∈[0,1]. Consequently, Eq. (6) inverts the continuous map only when |q(x)|≤1, and the described uniform binning of [0,1] would map every entry with |x|>1 to the same saturated bin. In Algorithm 1, Q_t is computed as G'^T_t P_t after P_t is orthonormalized, and its entries can readily exceed 1 in magnitude; as written, the method would collapse large entries and Ghat_t=P_t Q_t^T would not approximate G'_t. Because the logarithmic scale α is never reported and no code is provided, the reported communication savings and accuracies depend on an unstated scaling choice. Please either normalize the input (e.g., by the maximum absolute value), or specify a properly bounded log quantizer, and report the exact quantization procedure and α for each experiment.
[Section IV-B] The paper states that the error-feedback mechanism adopted from EF-SGD [9] and PowerSGD [20] 'guarantees stable convergence' and that the convergence properties are preserved. These guarantees do not automatically transfer to LQ-SGD: EF-SGD and PowerSGD analyze compressors with specific unbiasedness or bounded-error/variance properties, whereas the log-quantization used here is deterministic, nonlinear, and, as defined, unbounded. No contraction or variance bound is proved for the combined low-rank plus quantized operator, and no new analysis is supplied. The convergence behavior of Algorithm 1 is therefore an unverified empirical assumption rather than a guaranteed property. Please provide a formal bounded-error or variance condition satisfied by the proposed compressor, or explicitly weaken the convergence claims to empirical observations.
[Section V (Tables I–III and Figure 4)] The experimental section reports single-run numbers without seeds, standard deviations, or full hyperparameters (learning rate, batch size, number of epochs, α, b_p, b_q, and the TopK sparsity ratio). The accuracy differences in Tables I–III (e.g., 0.9290 vs 0.9451 on CIFAR-10) are presented as point estimates, so it is impossible to assess statistical significance. Moreover, the ImageNet claim that Rank 7 matches OriginalSGD at 75% Top-1 after 300 epochs is supported only by Figure 4, with no training details or numerical table. Since the central claim is that LQ-SGD preserves accuracy while cutting communication, multi-seed runs and complete experimental settings are required before the claim can be evaluated.

minor comments (6)

[Section V-B] The sentence 'achieving an effective compression ratio similar to PowerSGD at rank 1 (approximately 4.7× compression in our experiments)' is ambiguous: PowerSGD compresses by a factor of roughly 237 relative to Original SGD, while 4.7× is the additional reduction of LQ-SGD over PowerSGD.
[Figures 1–3] The legend label 'T opK' contains a spurious space, and the y-axis 'Log cross entropy' needs clarification as to whether the loss is log-transformed or plotted on a log scale.
[Section III-A] The notation in Eq. (2) uses g_t for the global gradient, while Algorithm 1 later updates with Ghat_t; please clarify that Ghat_t is the approximate global gradient used in place of g_t.
[References] Reference [15] is incomplete: it lacks publication venue and year.
[Tables I–III] The column header 'Size (x1)' is unclear; it should state the units (MB) and explain the multiplicative factors in parentheses.
[Sections I and II] There are minor grammatical errors, e.g., 'transmit only the some significant gradients' in Section II-B and 'that possesses stronger robustness' in Section I.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are externally benchmarked and the communication savings follow arithmetically from the 8-bit log-quantization design; the one self-citation is incidental.

full rationale

The paper's central claims are empirical and self-contained against external baselines (Original SGD, PowerSGD, TopK-SGD on CIFAR-10/100, MNIST, and ImageNet). LQ-SGD is specified as PowerSGD plus 8-bit log quantization, and the reported communication reduction is an arithmetic consequence of transmitting 8 bits instead of 32 bits per factor entry, not a fitted parameter renamed as a prediction. No convergence theorem is derived in the paper; the error-feedback mechanism is imported from EF-SGD and PowerSGD as external references, and the lack of a proof that their guarantees transfer to this quantization scheme is a correctness gap, not circularity. The only self-citation, reference [10], appears in Section V-C where the paper says the SSIM observation is 'in line with our earlier findings'; this is not load-bearing because the SSIM evidence is generated by the present experiments. The internal inconsistency between Eq. (5) and the claimed |q(x)|∈[0,1] bound, as well as the unreported value of α, is a specification and reproducibility defect, not a definitional circularity. No step in the claimed derivation reduces to its own inputs, so the circularity burden is not met.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No invented entities. The central claim depends on three chosen compression hyperparameters and on the assumed transfer of error-feedback convergence and low-rank gradient structure from prior work.

free parameters (3)

rank r = 1, 2, 4, 30, 50, 7 (ImageNet)
Compression rank for the low-rank factorization; chosen per experiment, not derived, and directly impacts communication cost and accuracy.
quantization bits b = 8
Number of bits for log-quantized P and Q; set to 8 in the experiments, not swept, and determines the 32/b compression factor.
logarithmic scale alpha = not reported
Controls the curvature of the log quantization in Eq. (5); the chosen value is not reported in the experiments, so the sensitivity is unknown.

assumptions (3)

domain assumption Error feedback convergence theory (EF-SGD) applies to LQ-SGD's combined low-rank and log-quantization compression
Section IV-B states that the error feedback mechanism is adopted to guarantee convergence, but no theorem or proof is given for this combination; the deterministic nonlinear log-quantization may violate the bounded-perturbation assumptions of EF-SGD.
domain assumption PowerSGD low-rank approximation quality transfers to ResNet-18 and ImageNet training
The paper relies on PowerSGD's practical success on benchmarks, as cited from reference [20], and assumes the same low-rank structure holds in their setup without verification.
domain assumption Gradient inversion attack with cosine similarity loss and total variation regularization is a representative threat model
Section III-C defines GIA via Eq. (4) and Section V-C measures SSIM for this single attack configuration; real attackers could use stronger methods, so the privacy claim is conditional on this threat model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm." pith.science (2026). https://pith.science/paper/7LQATNQS

@misc{pith2026250617974,
  author       = {Pith},
  title        = {Pith review of: Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LQATNQS}},
  note         = {Machine review of arXiv:2506.17974}
}

read the original abstract

We propose LQ-SGD (Low-Rank Quantized Stochastic Gradient Descent), an efficient communication gradient compression algorithm designed for distributed training. LQ-SGD further develops on the basis of PowerSGD by incorporating the low-rank approximation and log-quantization techniques, which drastically reduce the communication overhead, while still ensuring the convergence speed of training and model accuracy. In addition, LQ-SGD and other compression-based methods show stronger resistance to gradient inversion than traditional SGD, providing a more robust and efficient optimization path for distributed learning systems.

Figures

Figures reproduced from arXiv: 2506.17974 by the authors.

**Figure 2.** Results for CIFAR-100 dataset with different compression ranks. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

**Figure 3.** Results for MNIST dataset with different compression ranks. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Demonstrates stable convergence up to 300 epochs. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: SSIM scores under different compression ranks. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

[9]

Error feedback fixes signsgd and other gradient compression schemes

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. InInternational Conference on Machine Learning (ICML), pages 3252–3261, 2019

work page 2019
[20]

Powersgd: Practical low-rank gradient compression for distributed optimization

Thijs V ogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. InAdvances in Neural Information Processing Systems (NeurIPS), pages 14236–14246, 2019

work page 2019
[1]

Sparse communication for distributed gradient descent

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 440–445, 2017

work page 2017
[2]

Qsgd: Communication-efficient sgd via gradient quantization and encoding

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan V ojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. InAdvances in Neural Information Processing Systems (NeurIPS), pages 1709–1720, 2017

work page 2017
[3]

Demystifying the communication characteristics for distributed transformer models

Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abduljabbai, Aamir Shafi, Hari Subramoni, and Dhabaleswar K Panda. Demystifying the communication characteristics for distributed transformer models. In2024 IEEE Symposium on High-Performance Interconnects (HOTI), pages 57–65. IEEE, 2024

work page 2024
[4]

Communication-efficient distributed learning: A comprehensive survey.IEEE Transactions on Parallel and Distributed Systems, 2023

Yang Chen, Min Chen, Yanzhi Zhang, Liang Yang, and Victor CM Leung. Communication-efficient distributed learning: A comprehensive survey.IEEE Transactions on Parallel and Distributed Systems, 2023

work page 2023
[5]

Inverting gradients-how easy is it to break privacy in federated learning?Advances in neural information processing systems, 33:16937–16947, 2020

Jonas Geiping, Hartmut Bauermeister, Hannah Dr ¨oge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning?Advances in neural information processing systems, 33:16937–16947, 2020

work page 2020
[6]

Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

Priya Goyal, Piotr Dollar, Ross Girshick, et al. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017. 13 0 50 100 150 200 250 300 Epoch 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Accuracy Accuracy T opK PowerSGD LQSGD OriginalSGD 4 3 2 1 0 1 Log cross entropy Cross entropy T opK PowerSGD LQSGD OriginalSGD (a) MNIST, Rank 1...

arXiv 2017

Show all 20 references

[7]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778, 2016

2016
[8]

Models of heavy-tailed mechanistic universality.arXiv preprint arXiv:2506.03470, 2025

Liam Hodgkinson, Zhichao Wang, and Michael W Mahoney. Models of heavy-tailed mechanistic universality.arXiv preprint arXiv:2506.03470, 2025

2025 arXiv
[10]

Trustworthiness of stochastic gradient descent in distributed learning.arXiv preprint arXiv:2410.21491, 2024

Hongyang Li, Caesar Wu, Mohammed Chadli, Said Mammar, and Pascal Bouvry. Trustworthiness of stochastic gradient descent in distributed learning.arXiv preprint arXiv:2410.21491, 2024

2024 arXiv
[11]

Lightweight trustworthy distributed clustering.arXiv preprint arXiv:2504.10109, 2025

Hongyang Li, Caesar Wu, Mohammed Chadli, Said Mammar, and Pascal Bouvry. Lightweight trustworthy distributed clustering.arXiv preprint arXiv:2504.10109, 2025

2025 arXiv
[12]

Qiongxiu Li, Jaron Skovsted Gundersen, Milan Lopuha ¨a-Zwakenberg, and Richard Heusdens. Adaptive differentially quantized subspace perturbation (adqsp): A unified framework for privacy-preserving distributed average consensus.IEEE Transactions on Information F orensics and Se...

2023
[13]

Communication efficient privacy-preserving distributed optimization using adaptive differential quantization.Signal Processing, 194:108456, 2022

Qiongxiu Li, Richard Heusdens, and Mads Græsbøll Christensen. Communication efficient privacy-preserving distributed optimization using adaptive differential quantization.Signal Processing, 194:108456, 2022

2022
[14]

Deep gradient compression: Reducing the communication bandwidth for distributed training

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. InInternational Conference on Learning Representations (ICLR), 2018

2018
[15]

Provably communication-efficient data-parallel sgd via nonuniform quantization

Ali Ramezani-Kebrya, Fartash Faghri, Ilya Markov, Vitalii Aksenov, Dan Alistarh, and Daniel M Roy. Provably communication-efficient data-parallel sgd via nonuniform quantization
[16]

1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns

Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. InINTERSPEECH, pages 1058–1062, 2014. 15

2014
[17]

Measuring the effects of data parallelism on neural network training.arXiv preprint arXiv:1811.03600, 2018

Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training.arXiv preprint arXiv:1811.03600, 2018

2018 arXiv
[18]

A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks

Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang, Xiang Huang, and Xiaowen Chu. A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks. In2019 IEEE 39th International Conference on Distributed Computing Systems (ICD...

2019
[19]

Sparsified sgd with memory

Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. InAdvances in Neural Information Processing Systems (NeurIPS), pages 4447–4458, 2018

2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [9]

Error feedback fixes signsgd and other gradient compression schemes

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. InInternational Conference on Machine Learning (ICML), pages 3252–3261, 2019

work page 2019

[2] [20]

Powersgd: Practical low-rank gradient compression for distributed optimization

Thijs V ogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. InAdvances in Neural Information Processing Systems (NeurIPS), pages 14236–14246, 2019

work page 2019

[3] [1]

Sparse communication for distributed gradient descent

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 440–445, 2017

work page 2017

[4] [2]

Qsgd: Communication-efficient sgd via gradient quantization and encoding

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan V ojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. InAdvances in Neural Information Processing Systems (NeurIPS), pages 1709–1720, 2017

work page 2017

[5] [3]

Demystifying the communication characteristics for distributed transformer models

Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abduljabbai, Aamir Shafi, Hari Subramoni, and Dhabaleswar K Panda. Demystifying the communication characteristics for distributed transformer models. In2024 IEEE Symposium on High-Performance Interconnects (HOTI), pages 57–65. IEEE, 2024

work page 2024

[6] [4]

Communication-efficient distributed learning: A comprehensive survey.IEEE Transactions on Parallel and Distributed Systems, 2023

Yang Chen, Min Chen, Yanzhi Zhang, Liang Yang, and Victor CM Leung. Communication-efficient distributed learning: A comprehensive survey.IEEE Transactions on Parallel and Distributed Systems, 2023

work page 2023

[7] [5]

Inverting gradients-how easy is it to break privacy in federated learning?Advances in neural information processing systems, 33:16937–16947, 2020

Jonas Geiping, Hartmut Bauermeister, Hannah Dr ¨oge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning?Advances in neural information processing systems, 33:16937–16947, 2020

work page 2020

[8] [6]

Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

Priya Goyal, Piotr Dollar, Ross Girshick, et al. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017. 13 0 50 100 150 200 250 300 Epoch 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Accuracy Accuracy T opK PowerSGD LQSGD OriginalSGD 4 3 2 1 0 1 Log cross entropy Cross entropy T opK PowerSGD LQSGD OriginalSGD (a) MNIST, Rank 1...

arXiv 2017