Pith. sign in

REVIEW 4 major objections 6 minor 59 references

Efficient Distributed Training through Gradient Compression with Sparsification and Quantization Techniques

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mild gradient sparsification at 50x compression can lower test perplexity slightly below the uncompressed baseline on small LSTM language models.

desk verdict Transparent, small-scale benchmark of known compression methods on LSTM/PTB; the 0.06 perplexity gain is real but rests entirely on seed 42, so treat the regularization claim as a hypothesis, not a result. read the letter →

arxiv 2502.07634 v1 pith:CEX6QJI6 submitted 2024-12-07 cs.LG cs.MM

Shruti Singh , Shantanu Kumar This is my paper · ORCID
classification cs.LGcs.MM
keywords gradientcompressiondistributedtrainingsparsificationquantizationtop-kDGCQSGDLSTMlanguagemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether gradient compression can reduce the communication bottleneck in distributed training without hurting the model, and it answers with a qualified yes for language models. On a word-level LSTM trained on Penn Treebank, the authors compare random-k, top-k, DGC, and QSGD across 1, 2, and 4 workers while tuning hyperparameters separately for each method. Their central finding is that conservative sparsification, top-k and DGC at 50x compression, can slightly improve test perplexity, up to 0.06 lower than the uncompressed baseline, an effect they attribute to a regularizing action that is strongest for smaller models. The paper also reports that random-k is far worse than baseline, that compression ratios above 5000x degrade performance, that sparsified training converges in fewer epochs, and that communication time drops sharply for all methods. If these results hold, practitioners could treat modest sparsification as a free improvement that also cuts communication, rather than as an accuracy tax.

What carries the argument

The load-bearing mechanism is residual error feedback, called residual memory here, for sparsification: each compressed update leaves behind the difference between the true gradient and the transmitted one, and that error is added back into the next gradient so no information is permanently discarded. Top-k applies this to the largest-magnitude components, and DGC layers on momentum correction, momentum factor masking, and local gradient clipping to keep residual updates from going stale. The comparison is carried by a custom optimizer that simulates data-parallel distributed training on one GPU through sequential gradient accumulation, plus a per-method random hyperparameter search that tunes learning rate, dropout, and momentum separately for each technique and model size.

What would settle it

Re-run the medium-LSTM comparison of top-k and DGC at 50x compression against the uncompressed baseline across at least ten random seeds using the same tuned hyperparameters; if the perplexity improvement does not keep the same sign or falls within the seed noise, the claimed regularizing effect is not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that mild gradient sparsification can outperform no compression at all. With each method given its own randomly tuned hyperparameters, top-k (transmitting only the largest-magnitude gradient entries) and DGC (deep gradient compression, which adds momentum-corrected residual feedback to top-k-style sparsification) achieve lower test perplexity at 50x compression than the zero-compression baseline across one, two, and four workers, with the largest improvement around 0.06. This contradicts the usual expectation that compression is a lossy trade-off, and the paper interprets it as a regularizing effect that fades as model size grows. By contrast, random-k degrades perplexity by roughly 76% with residual error correction and 271% without it, QSGD (quantized SGD with stochastic rounding) remains near baseline, and at 5000x compression DGC's performance suffers while top-k degrades less. The paper's conclusion is that technique-specific hyperparameter tuning, not just the compression ratio, determines whether compressed training is competitive.

Load-bearing premise

The central comparison assumes that a single training run with seed 42 represents each method, even though the authors report seed-to-seed performance variations larger than the 0.06 perplexity gain they highlight.

Editorial extensions

If this is right

  • At a 50x compression ratio, top-k and DGC match or slightly beat the uncompressed baseline in perplexity when each method receives its own hyperparameter tuning, so mild sparsification need not be an accuracy loss.
  • Sparsification cuts communication time to a negligible fraction of runtime at high ratios, but top-k adds about 26% computation overhead from sorting, making DGC and QSGD more scalable in wall-clock terms.
  • Compression beyond about 5000x degrades performance, especially for DGC, implying a practical operating range below that ratio.
  • Sparsified training converges in fewer epochs than uncompressed training, so the time saved is larger than per-epoch comparisons suggest, provided early stopping is used.
  • The regularizing benefit of 50x sparsification is strongest for smaller models and shrinks as model size grows, so the headline effect is most relevant for small LSTM-scale models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reported gain of 0.06 in perplexity is smaller than the seed-to-seed variation the paper admits, a multi-seed replication is needed before treating the regularizing effect as reliable; if it survives, it suggests sparsification acts like a mild implicit regularizer.
  • The accelerated convergence under sparsification implies that epoch budgets and early-stopping patience tuned on uncompressed training may be wasteful for compressed training; a practitioner could shorten training schedules.
  • The communication-time estimates are based on idealized assumptions, such as bidirectional parameter-server traffic at 10 Gbps with no protocol overhead, so on real allreduce clusters with variable bandwidth the efficiency ranking between top-k, DGC, and QSGD could change.
  • DGC's failure at 5000x compression despite momentum masking hints that stale-update correction has an effective limit; an adaptive schedule that relaxes compression as training progresses might extend DGC's usable range.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical comparison of gradient compression techniques—random-k, top-k, DGC, and QSGD—for distributed training of LSTM language models on the Penn Treebank dataset. Training is simulated on a single GPU by sequential gradient accumulation for 1, 2, and 4 workers. The main claims are that conservative sparsification (top-k and DGC at 50x compression) slightly improves test perplexity over an uncompressed baseline (by up to 0.06), that very high compression ratios degrade performance, that sparsification accelerates convergence, and that per-method hyperparameter tuning is important. The paper also reports communication-time reductions and computation-time overheads, with top-k being computationally expensive due to sorting.

Significance. If the results were statistically robust, this would be a useful practical comparison because it focuses on RNN/LSTM language modeling rather than the vision benchmarks common in the compression literature, and it tunes hyperparameters separately for each compression method, which is more realistic than using baseline defaults. The paper is transparent about its limitations, reports findings that contradict its own hypotheses (DGC superiority was not confirmed), and uses an external uncompressed baseline. However, the quantitative evidence is fragile: all experiments use a single seed, the distributed environment is a simulation, and the convergence metric is affected by a disclosed early-stopping bug. These issues are acknowledged in the text but they nevertheless undermine the headline claims.

major comments (4)
  1. [Abstract; Section V.A; Section VI.B] The headline result—that top-k and DGC at 50x compression reduce test perplexity by up to 0.06 relative to baseline—is supported only by single-seed experiments. Section V.A states that 'due to significant performance variations observed even with different random seeds, this procedure is limited to experiments run on a single seed (42).' Since the baseline perplexity is on the order of 100, a 0.06 absolute improvement is approximately 0.05% and is much smaller than the admitted seed-to-seed variation. Without repeated runs, confidence intervals, or an explicit demonstration that the effect exceeds seed noise, the regularizing effect claimed in Section VI.B is not statistically established. The authors should either provide multi-seed results or substantially weaken the abstract and conclusion claims.
  2. [Section V.C; Section VI.C] The distributed training results are obtained from a simulation that sequentially accumulates compressed gradients on a single GPU, rather than from real multi-worker training. Section V.C acknowledges that this changes computation timing and omits allreduce communication, and that communication time is estimated from an assumed Ethernet speed. Consequently, the claims in Section VI.C about communication-time reductions and the scalability comparison between top-k and DGC are not validated against a real distributed system. The optimization dynamics of sequentially accumulating compressed gradients also differ from parallel workers averaging their compressed updates. The authors should reframe these results as simulation-based estimates or validate them in a real distributed setup.
  3. [Section VI.D] The convergence analysis is compromised by the acknowledged early-stopping bug. Section VI.D reports that the early-stopping counter incremented on any non-improvement rather than on a streak of non-improvements, so the epoch counts underlying Figure 8 and the accelerated-convergence claim in Figure 7 are unreliable. This directly affects the abstract's statement that sparsification techniques 'show accelerated convergence, requiring fewer epochs than the baseline.' The authors should fix the implementation and rerun the affected experiments, or remove these convergence claims.
  4. [Section V.A; Section VI.B] Even within the single-seed setting, the hyperparameter random search has 95% probability of selecting parameters within 5% of the optimal values (Section V.A). A 5% hyperparameter tuning tolerance is large relative to the reported 0.06 perplexity improvement, so different random-search draws could easily reverse the ranking between compressed and uncompressed training. The paper should demonstrate that the reported ordering is stable across the random-search distribution, for example by repeating the best-configuration runs with multiple seeds or reporting the distribution of validation perplexities over random-search trials.
minor comments (6)
  1. [Figure 1; Figure 7] The captions of Figure 1 and Figure 7 refer to 'training ambiguity' rather than 'training perplexity'; this typo should be corrected.
  2. [Section VI.B] The sentence 'a study on DGC with 462x compression reported a relative perplexity reduction of only 0.06' is unclear: it does not specify which study is meant, whether 0.06 is absolute or relative, or how it supports the argument about per-method tuning. Please clarify.
  3. [Figure 4] The caption of Figure 4 says that DGC and top-k are 'not distinguishable' in the plot; the curves should either be distinguished with different markers or the caption should explain why they coincide.
  4. [Section V.A] The claim that 60 random-search iterations give a 95% chance of finding parameters within 5% of optimal is used without a citation or derivation; adding a reference or a brief justification would be helpful.
  5. [References] Several references are incomplete, for example [49] lists 'Uniw. śląski, pp. 343–354' without a title or venue; the reference list should be completed before publication.
  6. [Section II.B.1] The statement that the maximum compression ratio for quantization is approximately 32x would benefit from a citation or derivation, since later the paper reports QSGD ratios above 10x without discussion of this limit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; all comparisons are empirical against an external uncompressed baseline, with no self-citation or constructional equivalence.

full rationale

This paper is an empirical benchmark study. The central comparison (top-k and DGC at 50x compression versus an uncompressed baseline) is evaluated by training LSTMs on PTB and measuring test perplexity in a custom simulated distributed environment. The compressed methods and baseline each receive their own hyperparameter search, and the test perplexity is an external metric, not an output of the compression formulas. The compression ratios are definitions (e.g., 100/2 for value-index sparsification) but are not used to derive the performance claims. No load-bearing result is imported from the authors' own prior work; the paper cites external sources (DGC, QSGD, etc.) for the techniques. The only self-referential-looking element is the convergence discussion, where faster convergence is attributed to the lower learning rates that the per-method tuning selected; this is an explicit post-hoc interpretation, not a prediction derived from the compression mechanism. Concerns about single-seed results (Section V.A) and the gradient-accumulation simulation (Section V.C) are statistical and external-validity risks, not circularity: they do not make the reported numbers equal to their inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no invented entities and no new theoretical constructs. Its central claims rest on five assumptions: the fidelity of single-GPU gradient-accumulation simulation to real distributed training, the representativeness of seed 42, the quality guarantee of the 60-iteration random search, the convergence of error-feedback sparsification inherited from the cited literature, and the fairness of stripping Elias and zero-run encodings from the comparisons. The tuned hyperparameters (learning rate, dropout, DGC momentum), the hand-chosen batch size, and the assumed 10 Gbps link speed are the effective free parameters of the study.

free parameters (5)
  • learning rate = varies per experiment; searched over U~[1,50]
    Tuned per method, model size, and worker count via random search; the paper shows optimal learning rates differ strongly between sparsification and quantization regimes, and this drives the convergence claims.
  • dropout rate = varies per experiment; searched over U~[0,0.8]
    Tuned per method; the paper reports a steady decline in optimal dropout as sparsification ratio increases, which is used to explain sparsification behavior.
  • DGC momentum = varies; searched over U~[0.1,0.9]
    Hyperparameter specific to the DGC implementation, treated as tunable in this study and fitted to validation performance.
  • batch size = 256
    Hand-chosen to maximize GPU memory usage as monitored through nvidia-smi; fixed across all experiments, so it affects gradient statistics and all comparisons.
  • assumed Ethernet speed = 10 Gbps
    Used to convert compressed gradient sizes into communication-time estimates; the paper calls the choice somewhat arbitrary and warns that the resulting computation-to-communication ratios are approximate.
assumptions (5)
  • domain assumption Sequential gradient accumulation on one GPU faithfully simulates synchronous data-parallel training with N workers, including the effect of compressing each worker's gradient before aggregation.
    Stated in Section V.C: workers are simulated in series on a single machine; the paper notes serial processing raises computation time and the absence of real allreduce reduces the accuracy of communication metrics. All worker-count comparisons depend on this.
  • domain assumption Single-seed (seed 42) runs represent each configuration sufficiently for ranking methods.
    Section V.A acknowledges significant performance variation across random seeds and restricts experiments to one seed; the paper's quantitative rankings, including the 0.06 perplexity improvement, inherit this assumption.
  • standard math A random search of 60 iterations has a 95 percent chance of finding hyperparameters within 5 percent of optimal for each method.
    Invoked in Section V.A to justify hyperparameter tuning quality; the paper does not verify this for its search distributions and did not confirm tuned configurations with repeated runs.
  • domain assumption Sparsified SGD with error feedback converges for the used LSTM/PTB setting.
    Taken from the cited literature (Stich et al. [38] for error feedback, Lin et al. [18] for DGC momentum correction); the paper relies on these convergence properties inside the simulated environment.
  • domain assumption Elias encoding for QSGD and zero-run encoding for DGC can be dropped without changing relative method rankings.
    Section IV states QSGD's Elias coding and DGC's efficient encoding are omitted so all techniques are compared on a comparable basis; this changes the effective compression ratios and communicated byte counts for those methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Distributed Training through Gradient Compression with Sparsification and Quantization Techniques." pith.science (2026). https://pith.science/paper/CEX6QJI6

@misc{pith2026250207634,
  author       = {Pith},
  title        = {Pith review of: Efficient Distributed Training through Gradient Compression with Sparsification and Quantization Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CEX6QJI6}},
  note         = {Machine review of arXiv:2502.07634}
}
read the original abstract

This study investigates the impact of gradient compression on distributed training performance, focusing on sparsification and quantization techniques, including top-k, DGC, and QSGD. In baseline experiments, random-k compression results in severe performance degradation, highlighting its inefficacy. In contrast, using top-k and DGC at 50 times compression yields performance improvements, reducing perplexity by up to 0.06 compared to baseline. Experiments across 1, 2, and 4 workers demonstrate that conservative sparsification can have a regularizing effect, especially for smaller models, while compression ratios above 5000 times impair performance, particularly for DGC. Communication times are reduced across all compression methods, with top-k and DGC decreasing communication to negligible levels at high compression ratios. However, increased computation times offset this efficiency for top-k due to sorting demands, making it less scalable than DGC or QSGD. In convergence tests, sparsification techniques show accelerated convergence, requiring fewer epochs than the baseline, which has implications for computational savings. Although precision trade-offs emerge, floating point errors are mitigated by compression. This study's findings underscore the need to tune hyperparameters specifically for each compression technique to achieve optimal model performance, especially in distributed training systems.

Figures

Figures reproduced from arXiv: 2502.07634 by the authors.

Figure 1
Figure 1. Training ambiguity was monitored during the medium [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Test the medium-sized model's perplexity vs. compression ratio for 1, 2, and 4 workers. For the sake of clarity, the results solely display top-k for sparsification; a comparison between Top-k and DGC may be found in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Test the ratio of compression to perplexity in an environment with just [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: highlights the results of the sparsification regime. For lower levels of compression, DGC demonstrates better performance. However, at extremely high compression ratios (5000×), performance diverges significantly in favor of top-k. This is beyond the range typically re…
Figure 6
Figure 6. Figure 6: Computation time for each compression method and ratio examined [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 5
Figure 5. Figure 5: Computation and communication times for every compression method [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 8
Figure 8. Figure 8: Grouped bar graph showing, for both the 1 and 2 worker settings, the [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: A box plot that shows the range of learning rate values used across 1, [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: A box plot that shows the range of dropout levels used for one, two, [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 34 canonical work pages

  1. [1]

    Document-Level Machine Translation with Large Language Models,

    L. Wang et al., “Document-Level Machine Translation with Large Language Models,” Apr. 2023, Accessed: Aug. 23,

    work page 2023

  2. [2]

    Optimizing Statistical Machine Translation for Text Simplification,

    W. Xu, C. Napoles, E. Pavlick, Q. Chen, and C. Callison-Burch, “Optimizing Statistical Machine Translation for Text Simplification,” Trans. Assoc. Comput. Linguist., vol. 4, pp. 401–415, Dec. 2016, doi: 10.1162/tacl_a_00107

    doi:10.1162/tacl_a_00107 2016

  3. [3]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. H. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, International Conference on Learning Representations, ICLR, Sep

    work page 2015

  4. [4]

    Guided source separation meets a strong ASR backend: Hitachi/Paderborn university joint investigation for dinner party ASR,

    N. Kanda et al., “Guided source separation meets a strong ASR backend: Hitachi/Paderborn university joint investigation for dinner party ASR,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, International Speech Communication Association, May 2019, pp. 1248–1252. doi: 10.21437/Interspeech.2019-1167

    work page doi:10.21437/interspeech.2019-1167 2019

  5. [5]

    An Enhanced Human Speech Emotion Recognition Using Hybrid of PRNN and KNN,

    J. Umamaheswari and A. Akila, “An Enhanced Human Speech Emotion Recognition Using Hybrid of PRNN and KNN,” in Proceedings of the International Conference on Machine Learning, Big Data, Cloud and Parallel Computing: Trends, Prespectives and Prospects, COMITCon 2019, Institute of Electrical and Electronics Engineers Inc., Feb. 2019, pp. 177–183. doi: 10.110...

    work page arXiv 2019

  6. [6]

    Available: https://arxiv.org/abs/1409.0473v7

    [Online]. Available: https://arxiv.org/abs/1409.0473v7

    arXiv

  7. [7]

    Low-cost ultrasonic based object detection and collision avoidance method for autonomous robots,

    J. N. Yasin, S. A. S. Mohamed, M. H. Haghbayan, J. Heikkonen, H. Tenhunen, and J. Plosila, “Low-cost ultrasonic based object detection and collision avoidance method for autonomous robots,” Int. J. Inf. Technol., vol. 13, no. 1, pp. 97–107, Feb. 2021, doi: 10.1007/s41870-020-00513-w

    work page doi:10.1007/s41870-020-00513-w 2021

  8. [8]

    A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,

    Y. Bang et al., “A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,” Feb. 2023, Accessed: Aug. 23,

    work page 2023

Show all 59 references
  1. [9]

    Few-shot object detection via feature reweighting,

    B. Kang, Z. Liu, X. Wang, F. Yu, J. Feng, and T. Darrell, “Few-shot object detection via feature reweighting,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 8419–8428. doi: 10.1109/ICCV.2019.00851

    2019

  2. [10]

    Deep learning,

    Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” May 27, 2015, Nature Publishing Group. doi: 10.1038/nature14539

    2015 doi

  3. [11]

    Neural GPUs learn algorithms,

    Ł. Kaiser and I. Sutskever, “Neural GPUs learn algorithms,” in 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, International Conference on Learning Representations, ICLR, Nov

    2016

  4. [12]

    Available: https://arxiv.org/abs/2302.04023v2

    [Online]. Available: https://arxiv.org/abs/2302.04023v2

    arXiv

  5. [13]

    Incorporating Visual Information in Audio Based Self-Supervised Speaker Recognition,

    D. Cai, W. Wang, and M. Li, “Incorporating Visual Information in Audio Based Self-Supervised Speaker Recognition,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 30, pp. 1422–1435, 2022, doi: 10.1109/TASLP.2022.3162078

    2022

  6. [14]

    Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization under Privacy Constraints,

    F. Sattler, K. R. Muller, and W. Samek, “Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization under Privacy Constraints,” IEEE Trans. Neural Networks Learn. Syst., vol. 32, no. 8, pp. 3710–3722, Aug. 2021, doi: 13 10.1109/TNNLS.2020.3015958

    2021

  7. [15]

    IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures,

    L. Espeholt et al., “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures,” in 35th International Conference on Machine Learning, ICML 2018, PMLR, Jul. 2018, pp. 2263–2284. Accessed: Dec. 03,

    2018

  8. [16]

    Available: https://arxiv.org/abs/1511.08228v3

    [Online]. Available: https://arxiv.org/abs/1511.08228v3

    arXiv

  9. [17]

    Pre-trained models for natural language processing: A survey,

    X. P. Qiu, T. X. Sun, Y. G. Xu, Y. F. Shao, N. Dai, and X. J. Huang, “Pre-trained models for natural language processing: A survey,” Oct. 01, 2020, Springer Verlag. doi: 10.1007/s11431-020-1647-3

    2020 doi

  10. [18]

    Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning,

    F. P. Such, V. Madhavan, E. Conti, J. Lehman, K. O. Stanley, and J. Clune, “Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning,” Dec. 2017, doi: 10.48550/arxiv.1712.06567

    2017 doi arXiv

  11. [19]

    Quantization networks,

    J. Yang et al., “Quantization networks,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 7300–7308. doi: 10.1109/CVPR.2019.00748

    2019

  12. [20]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,

    F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2014, pp. 105...

    2014 doi

  13. [21]

    This reduction is accomplished through the use of randomized rounding, which stochastically assigns gradient values to a set of discrete quantisation levels

    represents a family of gradient compression techniques that aim to reduce the bit representation of gradient values. This reduction is accomplished through the use of randomized rounding, which stochastically assigns gradient values to a set of discrete quantisation levels. By...

  14. [22]

    An Incentive Mechanism Design for Efficient Edge Learning by Deep Reinforcement Learning Approach,

    Y. Zhan and J. Zhang, “An Incentive Mechanism Design for Efficient Edge Learning by Deep Reinforcement Learning Approach,” in Proceedings - IEEE INFOCOM, Institute of Electrical and Electronics Engineers Inc., Jul. 2020, pp. 2489–2498. doi: 10.1109/INFOCOM41043.2020.9155268

    2020

  15. [23]

    Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training,

    Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training,” 6th Int. Conf. Learn. Represent. ICLR 2018 - Conf. Track Proc., Dec. 2017, Accessed: Oct. 13,

    2018

  16. [24]

    Computer vision for SHM of civil infrastructure: From dynamic response measurement to damage detection – A review,

    D. Feng and M. Q. Feng, “Computer vision for SHM of civil infrastructure: From dynamic response measurement to damage detection – A review,” Eng. Struct., vol. 156, pp. 105–117, Feb. 2018, doi: 10.1016/J.ENGSTRUCT.2017.11.018

    2018 doi

  17. [25]

    Rethinking the Inception Architecture for Computer Vision,

    C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826. doi: 10.1109/CVPR.2016.308

    2016 doi

  18. [26]

    QSGD: Communication-efficient SGD via gradient quantization and encoding,

    D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-efficient SGD via gradient quantization and encoding,” in Advances in Neural Information Processing Systems, 2017, pp. 1710–1721

    2017

  19. [27]

    Multiagent systems: a survey from a machine learning perspective,

    P. Stone and M. Veloso, “Multiagent systems: a survey from a machine learning perspective,” Auton. Robots, vol. 8, no. 3, pp. 345–383, Jun. 2000, doi: 10.1023/A:1008942012299

    doi

  20. [28]

    A survey on deep learning and its applications,

    S. Dong, P. Wang, and K. Abbas, “A survey on deep learning and its applications,” May 01, 2021, Elsevier. doi: 10.1016/j.cosrev.2021.100379

    2021

  21. [29]

    PyTorch Distributed: Experiences on Accelerating Data Parallel Training,

    S. Li et al., “PyTorch Distributed: Experiences on Accelerating Data Parallel Training,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3005–3018, Jun. 2020, doi: 10.14778/3415478.3415530

    2020

  22. [30]

    Beyond Data and Model Parallelism for Deep Neural Networks,

    Z. Jia, M. Zaharia, and A. Aiken, “Beyond Data and Model Parallelism for Deep Neural Networks,” Proc. Mach. Learn. Syst., vol. 1, pp. 1–13, Apr. 2018, Accessed: Oct. 13,

    2018

  23. [31]

    A comaparative study of GPU programming models and architectures using neural networks,

    V. K. Pallipuram, M. Bhuiyan, and M. C. Smith, “A comaparative study of GPU programming models and architectures using neural networks,” J. Supercomput., vol. 61, no. 3, pp. 673–718, Sep. 2012, doi: 10.1007/s11227-011-0631-3

    2012 doi

  24. [32]

    Models and Languages for Parallel Computation,

    D. B. Skillicorn and D. Talia, “Models and Languages for Parallel Computation,” ACM Comput. Surv., vol. 30, no. 2, pp. 123–169, Jun. 1998, doi: 10.1145/280277.280278

    1998

  25. [33]

    TicTac: Accelerating Distributed Deep Learning with Communication Scheduling,

    S. H. Hashemi, S. A. Jyothi, and R. H. Campbell, “TicTac: Accelerating Distributed Deep Learning with Communication Scheduling,” Proc. Mach. Learn. Syst., vol. 1, pp. 418–430, Apr. 2018, Accessed: Oct. 13,

    2018

  26. [34]

    Gradient sparsification for communication-efficient distributed optimization,

    J. Wangni, J. Liu, J. Wang, and T. Zhang, “Gradient sparsification for communication-efficient distributed optimization,” in Advances in Neural Information Processing Systems, 2018, pp. 1299–1309

    2018

  27. [35]

    Available: http://arxiv.org/abs/1807.05358

    [Online]. Available: http://arxiv.org/abs/1807.05358

    arXiv

  28. [36]

    Centralized, Distributed, and Everything in between: Reviewing Access Control Solutions for the IoT,

    S. Dramé-Maigné, M. Laurent, L. Castillo, and H. Ganem, “Centralized, Distributed, and Everything in between: Reviewing Access Control Solutions for the IoT,” ACM Comput. Surv., vol. 54, no. 7, Sep. 2022, doi: 10.1145/3465170

    2022 doi

  29. [37]

    Synchronization in distributed systems,

    A. Sampath and C. Tripti, “Synchronization in distributed systems,” in Advances in Intelligent Systems and Computing, Springer, Berlin, Heidelberg, 2012, pp. 417–424. doi: 10.1007/978-3-642-31513-8_43

    2012 doi

  30. [38]

    Sparsified SGD with memory,

    S. U. Stich, J. B. Cordonnier, and M. Jaggi, “Sparsified SGD with memory,” in Advances in Neural Information Processing Systems, 2018, pp. 4447–4458

    2018

  31. [39]

    AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training,

    C. Y. Chen, J. Choi, D. Brand, A. Agrawal, W. Zhang, and K. Gopalakrishnan, “AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training,” Proc. AAAI Conf. Artif. Intell., vol. 32, no. 1, pp. 2827–2835, Apr. 2018, doi: 10.1609/AAAI.V32I1.11728

    2018 doi

  32. [40]

    Adaptive quantization for deep neural network,

    Y. Zhou, S. M. Moosavi-Dezfooli, N. M. Cheung, and P. Frossard, “Adaptive quantization for deep neural network,” in 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, AAAI press, Apr. 2018, pp. 4596–4604. doi: 10.1609/aaai.v32i1.11623

    2018 doi

  33. [41]

    Characterisation of a split gradient coil design induced systemic imaging artefact on 0.35 T MR-linac systems,

    L. Marage et al., “Characterisation of a split gradient coil design induced systemic imaging artefact on 0.35 T MR-linac systems,” Phys. Med. Biol., vol. 68, no. 1, p. 01NT03, Dec. 2023, doi: 10.1088/1361-6560/aca876

    2023 doi

  34. [42]

    Sparse online learning via truncated gradient,

    J. Langford, L. Li, and T. Zhang, “Sparse online learning via truncated gradient,” in Advances in Neural Information Processing Systems 21 - Proceedings of the 2008 Conference, 2009, pp. 905–912

    2008

  35. [43]

    Long short-term memory,

    S. Hochreiter, J. S.-N. Computation, and U. 1997, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997, Accessed: Nov. 19,

    1997

  36. [44]

    TernGrad: Ternary gradients to reduce communication in distributed deep learning,

    W. Wen et al., “TernGrad: Ternary gradients to reduce communication in distributed deep learning,” in Advances in Neural Information Processing Systems, 2017, pp. 1510–1520. Accessed: Oct. 14,

    2017

  37. [45]

    Communication-Efficient Data Parallel Distributed Deep Learning: A Comprehensive Survey,

    Z. Tang, S. Shi, W. Wang, B. Li, and X. Chu, “Communication-Efficient Data Parallel Distributed Deep Learning: A Comprehensive Survey,” vol. 1, 2023, doi: 10.1145/nnnnnnn.nnnnnnn

    2023 doi

  38. [46]

    From Text to Transformation: A Comprehensive Review of Large Language Models’ Versatility,

    P. Kaur, G. S. Kashyap, A. Kumar, M. T. Nafis, S. Kumar, and V. Shokeen, “From Text to Transformation: A Comprehensive Review of Large Language Models’ Versatility,” Feb. 2024, Accessed: Mar. 21,

    2024

  39. [47]

    Available: https://arxiv.org/abs/2402.16142v1

    [Online]. Available: https://arxiv.org/abs/2402.16142v1

    arXiv

  40. [48]

    The resurgence of structure in deep neural networks,

    P. Veličković, “The resurgence of structure in deep neural networks,” Icml, no. January, 2019, Accessed: May 09,

    2019

  41. [49]

    Compressed Communication for Distributed Deep Learning: Survey and Quantitative Evaluation,

    H. Xu et al., “Compressed Communication for Distributed Deep Learning: Survey and Quantitative Evaluation,” Uniw. śląski, pp. 343–354, 2020, Accessed: Oct. 13,

    2020

  42. [50]

    Building a large annotated corpus of English: the Penn Treebank (1993),

    M. P. Marcus, “Building a large annotated corpus of English: the Penn Treebank (1993),” in Corpus Linguistics: Readings in a Widening Discipline, 2005, pp. 242–257. 14

    1993

  43. [51]

    Accumulated gradient normalization,

    J. R. Hermans, G. Spanakis, and R. Möckel, “Accumulated gradient normalization,” in Journal of Machine Learning Research, PMLR, Nov. 2017, pp. 439–454. Accessed: Oct. 13,

    2017

  44. [52]

    DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients,

    S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients,” Jun. 2016, Accessed: Oct. 14,

    2016

  45. [53]

    Available: https://arxiv.org/abs/1606.06160v3

    [Online]. Available: https://arxiv.org/abs/1606.06160v3

    arXiv

  46. [54]

    Scalable distributed DNN training using commodity GPU cloud computing,

    N. Strom, “Scalable distributed DNN training using commodity GPU cloud computing,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2015, pp. 1488–1492. doi: 10.21437/interspeech.2015-354

    2015 doi

  47. [55]

    Communication Quantization for Data-Parallel Training of Deep Neural Networks,

    N. Dryden, T. Moon, S. A. Jacobs, and B. Van Essen, “Communication Quantization for Data-Parallel Training of Deep Neural Networks,” Institute of Electrical and Electronics Engineers (IEEE), Jan. 2017, pp. 1–8. doi: 10.1109/mlhpc.2016.004

    2017 doi

  48. [56]

    Sparse communication for distributed gradient descent,

    A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” in EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings, Association for Computational Linguistics (ACL), Apr. 2017, pp. 440–445. doi: 10.18653/v1/d17-1045

    2017 doi

  49. [2022]

    Available: https://ieeexplore.ieee.org/abstract/document/6795963/

    [Online]. Available: https://ieeexplore.ieee.org/abstract/document/6795963/

  50. [2023]

    Available: https://arxiv.org/abs/2304.02210v1

    [Online]. Available: https://arxiv.org/abs/2304.02210v1

    arXiv

  51. [2024]

    Available: https://arxiv.org/abs/1712.01887v3

    [Online]. Available: https://arxiv.org/abs/1712.01887v3

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.