Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging

T0 review · 3 major / 2 minor · reviewed 2026-05-21 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LOSCAR-SGD combines sparse local updates with computation-communication overlap and a delay-corrected merge to converge on smooth non-convex objectives.

desk verdict LOSCAR-SGD combines sparse local SGD, overlap, and a delay-corrected merge with non-convex rates that track heterogeneity, but the sparse delay correction under varying worker delays is the part that needs verification. read the letter →

arxiv 2605.20866 v1 pith:GLH3PFKX submitted 2026-05-20 cs.LG cs.DCmath.OCstat.ML

Yassine Maziane , Ammar Mahran , Artavazd Maranjyan , Peter Richtárik This is my paper · ORCID
classification cs.LGcs.DCmath.OCstat.ML
keywords localSGDsparsemodelaveragingcommunication-computationoverlapdelaycorrectionnon-convexoptimizationdistributedlearningheterogeneousworkersconvergenceanalysis
checked against Cost.FunctionalEquation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LOSCAR-SGD as a local SGD variant for distributed settings where workers have heterogeneous compute speeds. It communicates only sparse model coordinates while allowing local optimization to continue during communication, using a delay-corrected merge to integrate the delayed information. Convergence guarantees are derived for smooth non-convex objectives, with rates that explicitly depend on the sparsity level, the amount of overlap, and the degree of worker heterogeneity. This supplies the first theoretical analysis for the practical combination of local training, sparsity, and overlap.

What carries the argument

The delay-corrected merge rule, which folds delayed sparse updates from heterogeneous workers back into the local models without erasing progress accumulated during the overlap interval.

What would settle it

Replace the delay-corrected merge with naive overwriting in a heterogeneous testbed with measurable overlap periods and observe whether convergence slows or fails relative to the predicted rate.

Watch

Extended reading notes

Core claim

LOSCAR-SGD is a Local SGD method that communicates only a sparse subset of model coordinates and continues optimizing while communication is in flight. A key ingredient is a delay-corrected merge rule that incorporates delayed synchronized information without discarding the progress made during the overlap phase. We give convergence guarantees for smooth non-convex objectives and show how sparsity, overlap, and worker heterogeneity affect the rate. This is the first theory for this combination of ingredients.

Load-bearing premise

The delay-corrected merge rule correctly incorporates delayed synchronized information without discarding the progress made during the overlap phase.

Editorial extensions

If this is right

  • Sparsity level directly modulates the communication volume and appears in the convergence bound.
  • Communication-computation overlap shortens wall-clock training time without harming the asymptotic rate.
  • Worker heterogeneity increases the effective delay term and slows the rate in a quantifiable way.
  • The delay-corrected merge outperforms naive overwriting on both theory and reported experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same overlap-plus-correction idea could be applied to other first-order methods such as Adam or momentum variants.
  • Pairing the sparse merge with coordinate-wise quantization might yield multiplicative communication savings.
  • In federated settings the optimal overlap length could be tuned from measured round-trip times and compute variance.
  • The analysis suggests that very high sparsity may require compensatory increases in local steps to keep the rate acceptable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper proposes LOSCAR-SGD, a Local SGD algorithm for heterogeneous distributed settings that combines sparse coordinate communication, communication-computation overlap, and a delay-corrected merge rule to incorporate delayed updates without discarding local progress during overlap. It claims convergence guarantees for smooth non-convex objectives, with explicit dependence of the rate on sparsity level, overlap duration, and worker heterogeneity, and reports experiments showing reduced wall-clock time and better performance than naive overwriting.

Significance. If the convergence analysis is correct, the work would be significant as the first explicit theory for the joint combination of local steps, sparsity, overlap, and heterogeneity; the parameter dependence could directly inform practical tuning in large-scale training. The experiments provide supporting evidence for the overlap benefit, though verification is limited by the absence of full proof details and statistical error bars.

major comments (3)
  1. §3 (delay-corrected merge rule): the claim that the rule produces an unbiased estimator of the averaged model while preserving overlap-phase progress is load-bearing for all rate statements, yet the description does not specify whether the correction is applied coordinate-wise only to the sparse mask that was actually sent at the delayed time or uniformly; under heterogeneous delays and per-worker sparsity this risks introducing a bias term proportional to sparsity level times delay variance, which would invalidate the claimed rate.
  2. Theorem 1 (convergence bound): the rate is stated to depend explicitly on sparsity, overlap, and heterogeneity, but the proof sketch relies on the merge rule remaining unbiased without additional assumptions on consistent sparse masks across send/receive times; if the analysis applies a uniform correction, the variance term from coordinate-wise heterogeneity could grow and contradict the stated bound.
  3. §5 (experiments): the reported improvements in training time lack error bars or multiple independent runs, so it is impossible to assess whether the observed gains over naive overwriting are statistically reliable or sensitive to random seeds.
minor comments (2)
  1. Notation for the sparse mask and delay variables is introduced without a consolidated table; a single reference table would improve readability of the rate expressions.
  2. The abstract states this is the first theory for the combination, but the introduction omits explicit comparison to prior overlap analyses in Local SGD (e.g., those handling fixed delays without sparsity).

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review of our manuscript. We address each major comment in turn below and have revised the paper to improve clarity and completeness where needed.

read point-by-point responses

  1. Referee: §3 (delay-corrected merge rule): the claim that the rule produces an unbiased estimator of the averaged model while preserving overlap-phase progress is load-bearing for all rate statements, yet the description does not specify whether the correction is applied coordinate-wise only to the sparse mask that was actually sent at the delayed time or uniformly; under heterogeneous delays and per-worker sparsity this risks introducing a bias term proportional to sparsity level times delay variance, which would invalidate the claimed rate.

    Authors: The delay-corrected merge rule is defined to apply the correction coordinate-wise and exclusively to the coordinates present in the sparse mask that was transmitted at the delayed communication round. This is stated in Section 3 immediately after the algorithm pseudocode and is used in the subsequent analysis. Because only the communicated coordinates receive the delay adjustment, the estimator for the averaged model remains unbiased; local progress on non-communicated coordinates is retained without introducing an extra bias term. The dependence on sparsity level and delay already appears in the convergence bound of Theorem 1. We have added a short clarifying paragraph and a supporting lemma in the revised §3 to make the coordinate-wise application explicit. revision: yes

  2. Referee: Theorem 1 (convergence bound): the rate is stated to depend explicitly on sparsity, overlap, and heterogeneity, but the proof sketch relies on the merge rule remaining unbiased without additional assumptions on consistent sparse masks across send/receive times; if the analysis applies a uniform correction, the variance term from coordinate-wise heterogeneity could grow and contradict the stated bound.

    Authors: The full proof in the appendix explicitly assumes that the sparse masks are those chosen at the sending time and that the correction is applied only to those coordinates; a uniform correction is never used. Under this construction the unbiasedness holds and the variance contribution from coordinate-wise heterogeneity is controlled by the sparsity factor already present in the rate. We have expanded the proof sketch in the main text of the revised manuscript with a one-paragraph outline of the unbiasedness argument and a pointer to the relevant appendix lemma. revision: yes

  3. Referee: §5 (experiments): the reported improvements in training time lack error bars or multiple independent runs, so it is impossible to assess whether the observed gains over naive overwriting are statistically reliable or sensitive to random seeds.

    Authors: We agree that the experimental presentation would be strengthened by statistical reporting. In the revised manuscript we have repeated the wall-clock time experiments over five independent random seeds and added error bars (mean ± one standard deviation) to the relevant plots in §5. The observed gains of LOSCAR-SGD over naive overwriting remain consistent across seeds. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Convergence analysis derives from standard smoothness assumptions and proposed merge rule without tautological reduction

full rationale

The paper proposes LOSCAR-SGD with a delay-corrected sparse merge and derives convergence rates for smooth non-convex objectives directly from the algorithm's update rules, sparsity masks, overlap phases, and heterogeneity parameters. The rate expressions follow from standard bounded-variance and smoothness assumptions applied to the new merge operator; no step equates a claimed prediction or theorem to a fitted input or prior self-citation by construction. The 'first theory' claim for the combination of ingredients further indicates the derivation chain is self-contained rather than relying on load-bearing self-citations or ansatzes imported from the authors' prior work.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The analysis rests on standard smoothness and bounded-variance assumptions common to non-convex SGD theory plus the novel delay-correction mechanism; no new particles or dimensions are introduced.

free parameters (2)
  • sparsity level
    Fraction of coordinates communicated; chosen to trade communication cost against convergence speed.
  • overlap duration
    Time window during which local computation continues while communication is in flight; affects the delay term in the analysis.
assumptions (2)
  • domain assumption Objective function is L-smooth
    Standard assumption invoked for non-convex convergence analysis of SGD variants.
  • domain assumption Workers may perform different numbers of local steps
    Heterogeneous compute setting explicitly stated as the operating regime.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging." pith.science (2026). https://pith.science/paper/GLH3PFKX

@misc{pith2026260520866,
  author       = {Pith},
  title        = {Pith review of: LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLH3PFKX}},
  note         = {Machine review of arXiv:2605.20866}
}
read the original abstract

Communication is a major bottleneck in distributed learning, especially in large-scale settings and in federated learning environments with slow links. Three standard ways to reduce this cost are communication compression, local training, and communication-computation overlap. Methods that combine these ingredients are used in practice and have been found to be effective for large-scale training, but there is little theory for methods that combine all three. We study a heterogeneous-compute setting in which different workers may take different numbers of local steps, and we propose LOSCAR-SGD, a Local SGD method that communicates only a sparse subset of model coordinates and continues optimizing while communication is in flight. A key ingredient is a delay-corrected merge rule that incorporates delayed synchronized information without discarding the progress made during the overlap phase. We give convergence guarantees for smooth non-convex objectives and show how sparsity, overlap, and worker heterogeneity affect the rate. To the best of our knowledge, this is the first theory for this combination of ingredients. Experiments further show that communication-computation overlap reduces training time and that the delay-corrected merge outperforms naive overwriting.

Figures

Figures reproduced from arXiv: 2605.20866 by the authors.

Figure 1
Figure 1. Main experimental summary on a9a. (a) Overlap improves over blocking sparse averaging, and delay correction is best. (b) Smaller sparsity levels reduce the logical communication cost by orders of magnitude for the delay-corrected method. (c) Under a long communication delay, delay correction substantially outperforms overwrite. regime covered by the theory. It also includes CIFAR-10 and Tiny ImageNet neural-network … view at source ↗
Figure 2
Figure 2. Accuracy comparison for the controlled sparse-overlap experiment on [PITH_FULL_IMAGE:figures/full_fig_p031_2.png] view at source ↗
Figure 3
Figure 3. Training gradient norm for the controlled sparse-overlap experiment on [PITH_FULL_IMAGE:figures/full_fig_p032_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Loss curves under equivalent resource parametrizations for the controlled sparse-overlap [PITH_FULL_IMAGE:figures/full_fig_p032_4.png]
Figure 5
Figure 5. Figure 5: Long-compute, short-delay regime (M = 8, ζ = 6, worker times (1, 2, 3, 6)). Delay correction gives a small but persistent improvement over overwrite on both training and validation loss. (a) Training loss. (b) Validation loss [PITH_FULL_IMAGE:figures/full_fig_p033_5.png]
Figure 6
Figure 6. Figure 6: Short-compute, long-delay regime (M = 2, ζ = 24, worker times (1, 2, 3, 6)). This is the regime with the largest gap: overwrite discards a substantial amount of overlap-phase progress, while delay correction preserves it. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]
Figure 7
Figure 7. Figure 7: Very heterogeneous worker-speed regime ( [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: Sparsity-level ablation on a9a. We vary p ∈ {0.001, 0.01, 0.1, 1.0} with M = 4, ζ = 6, worker times (1, 2, 3, 6), and otherwise identical training settings. Smaller p dramatically reduces the cumulative number of communicated bits while preserving a similar loss trajec…
Figure 9
Figure 9. Figure 9: Ablation on the local computation budget. The blocking local-sparse baseline is sensitive [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]
Figure 10
Figure 10. Figure 10: Training loss versus cumulative logical time for the communication-delay ablation. The [PITH_FULL_IMAGE:figures/full_fig_p037_10.png]
Figure 11
Figure 11. Figure 11: Negative stress test with strongly heterogeneous data and very slow communication. The [PITH_FULL_IMAGE:figures/full_fig_p038_11.png]
Figure 12
Figure 12. Figure 12: CIFAR-10 convolutional-network experiment in the normal communication regime, with [PITH_FULL_IMAGE:figures/full_fig_p039_12.png]
Figure 13
Figure 13. Figure 13: CIFAR-10 convolutional-network experiment in the communication-stress regime, with [PITH_FULL_IMAGE:figures/full_fig_p040_13.png]
Figure 14
Figure 14. Figure 14: Tiny ImageNet experiment in the normal communication regime, with [PITH_FULL_IMAGE:figures/full_fig_p041_14.png]
Figure 15
Figure 15. Figure 15: Tiny ImageNet experiment in the communication-stress regime, with [PITH_FULL_IMAGE:figures/full_fig_p042_15.png]

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Agora is a protocol that trained an 8.6B-parameter LLM on 500B tokens using churning, heterogeneous, internet-connected consumer GPUs at ~63% of centralized H100 token-per-FLOP efficiency.

Reference graph

Works this paper leans on

299 extracted references · 299 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reduction

    Tovmasyan, Zhirayr and Maranjyan, Artavazd and Richt. arXiv preprint arXiv:2605.08871 , year=

    work page Pith review arXiv

  2. [2]

    Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity

    Ammar Mahran and Artavazd Maranjyan and Peter Richt. Rescaled Asynchronous. arXiv preprint arXiv:2605.13434 , year=

    work page Pith review arXiv

  3. [3]

    Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method

    Abdurakhmon Sadiev and Artavazd Maranjyan and Ivan Ilin and Peter Richt. Ringmaster. arXiv preprint arXiv:2605.18174 , year=

    work page Pith review arXiv

  4. [4]

    2025 , publisher =

    Andrej Karpathy , title =. 2025 , publisher =

    work page 2025

  5. [5]

    First Provably Optimal Asynchronous

    Artavazd Maranjyan , year =. First Provably Optimal Asynchronous

    work page

  6. [6]

    Ringleader

    Artavazd Maranjyan and Peter Richt. Ringleader. The Fourteenth International Conference on Learning Representations , year=

    work page

  7. [7]

    Ringmaster

    Artavazd Maranjyan and Alexander Tyurin and Peter Richt. Ringmaster. 2025 , booktitle=

    work page 2025

  8. [8]

    2025 , booktitle=

    Maranjyan, Artavazd and Saad, El Mehdi and Richt. 2025 , booktitle=

    work page 2025

Show all 299 references
  1. [9]

    MindFlayer

    Artavazd Maranjyan and Omar Shaikh Omar and Peter Richt. MindFlayer. The 41st Conference on Uncertainty in Artificial Intelligence , year=

  2. [10]

    Transactions on Machine Learning Research , issn=

    Artavazd Maranjyan and Mher Safaryan and Peter Richt. Transactions on Machine Learning Research , issn=. 2025 , url=

    2025

  3. [11]

    arXiv preprint arXiv:2412.17054 , year=

    Differentially Private Random Block Coordinate Descent , author=. arXiv preprint arXiv:2412.17054 , year=

  4. [12]

    The Thirteenth International Conference on Learning Representations , year=

    Laurent Condat and Artavazd Maranjyan and Peter Richt. The Thirteenth International Conference on Learning Representations , year=

  5. [13]

    arXiv preprint arXiv:2601.12400 , year=

    Condat, Laurent and Maranjyan, Artavazd and Richt. arXiv preprint arXiv:2601.12400 , year=

  6. [14]

    Journal of Contemporary Mathematical Analysis (Armenian Academy of Sciences) , volume=

    Grigoryan, Martin and Kamont, Anna and Maranjyan, Artavazd , title=. Journal of Contemporary Mathematical Analysis (Armenian Academy of Sciences) , volume=. 2023 , publisher=

    2023

  7. [15]

    On the divergence of

    Grigoryan, Martin and Maranjyan, Artavazd , journal=. On the divergence of

  8. [16]

    On the unconditional convergence of

    Grigoryan, Tigran M and Maranjyan, Artavazd , journal=. On the unconditional convergence of

  9. [17]

    On the Ineffectiveness of Variance Reduced Optimization for Deep Learning , volume =

    Defazio, Aaron and Bottou, Leon , booktitle =. On the Ineffectiveness of Variance Reduced Optimization for Deep Learning , volume =

  10. [18]

    We did the math on

    O'Donnell, James and Crownhart, Casey , journal =. We did the math on. 2025 , month =

    2025

  11. [19]

    Joule , volume=

    The growing energy footprint of artificial intelligence , author=. Joule , volume=. 2023 , publisher=

    2023

  12. [20]

    Measuring the environmental impact of delivering

    Elsworth, Cooper and Huang, Keguo and Patterson, David and Schneider, Ian and Sedivy, Robert and Goodman, Savannah and Townsend, Ben and Ranganathan, Parthasarathy and Dean, Jeff and Vahdat, Amin and others , journal=. Measuring the environmental impact of delivering

  13. [21]

    The rising costs of training frontier

    Cottier, Ben and Rahman, Robi and Fattorini, Loredana and Maslej, Nestor and Besiroglu, Tamay and Owen, David , journal=. The rising costs of training frontier

  14. [22]

    Fradin, Adrien and Richt. Local. arXiv preprint arXiv:2509.23207 , year=

  15. [23]

    2024 , url =

    Keller Jordan and Yuchen Jin and Vlado Boza and Jiacheng You and Franz Cesista and Laker Newhouse and Jeremy Bernstein , title =. 2024 , url =

    2024

  16. [24]

    2025 , booktitle=

    Nesterov Method for Asynchronous Pipeline Parallel Optimization , author=. 2025 , booktitle=

    2025

  17. [25]

    arXiv preprint arXiv:1910.05124 , year=

    Yang, Bowen and Zhang, Jian and Li, Jonathan and R. arXiv preprint arXiv:1910.05124 , year=

    1910

  18. [26]

    arXiv preprint arXiv:2509.19029 , year=

    Clapping: Removing Per-sample Storage for Pipeline Parallel Distributed Optimization with Communication Compression , author=. arXiv preprint arXiv:2509.19029 , year=

  19. [27]

    International Conference on Machine Learning , pages=

    Shampoo: Preconditioned stochastic tensor optimization , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  20. [28]

    Proceedings of the 30th International Conference on Machine Learning , pages =

    Online Learning under Delayed Feedback , author =. Proceedings of the 30th International Conference on Machine Learning , pages =. 2013 , editor =

    2013

  21. [29]

    Bistritz, Ilai and Zhou, Zhengyuan and Chen, Xi and Bambos, Nicholas and Blanchet, Jose , booktitle =. Online

  22. [30]

    International Conference on Machine Learning , pages=

    Adapting to delays and data in adversarial multi-armed bandits , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  23. [31]

    Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages =

    Bandit Online Learning with Unknown Delays , author =. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages =. 2019 , editor =

    2019

  24. [32]

    The Nonstochastic Multiarmed Bandit Problem , journal =

    Auer, Peter and Cesa-Bianchi, Nicol\`. The Nonstochastic Multiarmed Bandit Problem , journal =. 2002 , doi =. https://doi.org/10.1137/S0097539701398375 , abstract =

    2002 doi

  25. [33]

    arXiv preprint arXiv:1903.03934 , year=

    Asynchronous federated optimization , author=. arXiv preprint arXiv:1903.03934 , year=

    1903

  26. [34]

    Journal of Machine Learning Research , volume=

    A general theory for federated optimization with asynchronous and heterogeneous clients updates , author=. Journal of Machine Learning Research , volume=

  27. [35]

    Advances in Neural Information Processing Systems , volume=

    Asynchronous parallel stochastic gradient for nonconvex optimization , author=. Advances in Neural Information Processing Systems , volume=

  28. [36]

    arXiv preprint arXiv:2408.04929 , year=

    Tight time complexities in parallel stochastic optimization with arbitrary computation dynamics , author=. arXiv preprint arXiv:2408.04929 , year=

  29. [37]

    Wang, Qiyuan and Yang, Qianqian and He, Shibo and Shi, Zhiguo and Chen, Jiming , journal=

  30. [38]

    IEEE Transactions on Wireless Communications , volume=

    Asynchronous federated learning over wireless communication networks , author=. IEEE Transactions on Wireless Communications , volume=. 2022 , publisher=

    2022

  31. [39]

    IEEE Transactions on Automatic Control , volume=

    Distributed asynchronous deterministic and stochastic gradient optimization algorithms , author=. IEEE Transactions on Automatic Control , volume=. 1986 , publisher=

    1986

  32. [40]

    Journal of Machine Learning Research , volume=

    Asynchronous iterations in optimization: New sequence results and sharper algorithmic guarantees , author=. Journal of Machine Learning Research , volume=

  33. [41]

    Megatron-

    Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanzaro, Bryan , journal=. Megatron-

  34. [42]

    Efficient large-scale language model training on

    Narayanan, Deepak and Shoeybi, Mohammad and Casper, Jared and LeGresley, Patrick and Patwary, Mostofa and Korthikanti, Vijay and Vainbrand, Dmitri and Kashinkunti, Prethvi and Bernauer, Julie and Catanzaro, Bryan and others , booktitle=. Efficient large-scale language model tr...

  35. [43]

    Proceedings of the 44th Annual International Symposium on Computer Architecture , pages=

    In-Datacenter Performance Analysis of a Tensor Processing Unit , author=. Proceedings of the 44th Annual International Symposium on Computer Architecture , pages=. 2017 , month=

    2017

  36. [44]

    Energy and

    International Energy Agency , year=. Energy and

  37. [45]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Energy and policy considerations for modern deep learning research , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  38. [46]

    Advances in Neural Information Processing Systems , editor =

    Cyclades: Conflict-free Asynchronous Machine Learning , author =. Advances in Neural Information Processing Systems , editor =

  39. [47]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Delay-Adaptive Step-sizes for Asynchronous Learning , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  40. [48]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    Asynchronous Stochastic Gradient Descent with Delay Compensation , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor =

    2017

  41. [49]

    2020 , organization=

    Rajbhandari, Samyam and Rasley, Jeff and Ruwase, Olatunji and He, Yuxiong , booktitle=. 2020 , organization=

    2020

  42. [50]

    Edward Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Weizhu Chen , booktitle=

    J. Edward Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Weizhu Chen , booktitle=

  43. [51]

    Transactions on Machine Learning Research , issn=

    Efficient Large Language Models: A Survey , author=. Transactions on Machine Learning Research , issn=. 2024 , url=

    2024

  44. [52]

    Accurate, Large Minibatch

    Goyal, Priya and Doll. Accurate, Large Minibatch. arXiv preprint arXiv:1706.02677 , year=

    arXiv

  45. [53]

    Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation , pages =

    Ananthanarayanan, Ganesh and Ghodsi, Ali and Shenker, Scott and Stoica, Ion , title =. Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation , pages =. 2013 , publisher =

    2013

  46. [54]

    Annals of Mathematical Statistics , volume=

    A Stochastic Approximation Method , author=. Annals of Mathematical Statistics , volume=

  47. [55]

    Optimization Methods for Large-Scale Machine Learning , journal =

    Bottou, L\'. Optimization Methods for Large-Scale Machine Learning , journal =. 2018 , doi =

    2018

  48. [56]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal=. The

  49. [57]

    Deep neural networks for

    Covington, Paul and Adams, Jay and Sargin, Emre , booktitle=. Deep neural networks for

  50. [58]

    arXiv preprint arXiv:1604.07316 , year=

    End to end learning for self-driving cars , author=. arXiv preprint arXiv:1604.07316 , year=

    arXiv

  51. [59]

    Large Scale Distributed Deep Networks , url =

    Dean, Jeffrey and Corrado, Greg and Monga, Rajat and Chen, Kai and Devin, Matthieu and Mao, Mark and Ranzato, Marc aurelio and Senior, Andrew and Tucker, Paul and Yang, Ke and Le, Quoc and Ng, Andrew , booktitle =. Large Scale Distributed Deep Networks , url =

  52. [60]

    arXiv preprint arXiv:2006.15704 , year=

    Pytorch distributed: Experiences on accelerating data parallel training , author=. arXiv preprint arXiv:2006.15704 , year=

    2006 arXiv

  53. [61]

    Advances in Neural Information Processing Systems , volume=

    Communication efficient distributed machine learning with the parameter server , author=. Advances in Neural Information Processing Systems , volume=

  54. [62]

    arXiv preprint arXiv:1806.00582 , year=

    Federated learning with non-iid data , author=. arXiv preprint arXiv:1806.00582 , year=

    arXiv

  55. [63]

    IEEE Transactions on Neural Networks and Learning Systems , volume=

    Towards personalized federated learning , author=. IEEE Transactions on Neural Networks and Learning Systems , volume=. 2022 , publisher=

    2022

  56. [64]

    SIAM Journal on Optimization , volume=

    A convergent incremental gradient method with a constant step size , author=. SIAM Journal on Optimization , volume=. 2007 , publisher=

    2007

  57. [65]

    Defazio, Aaron and Bach, Francis and Lacoste-Julien, Simon , journal=

  58. [66]

    SIAM Journal on Optimization , volume=

    On the convergence rate of incremental aggregated gradient algorithms , author=. SIAM Journal on Optimization , volume=. 2017 , publisher=

    2017

  59. [67]

    Advances in Neural Information Processing Systems , volume=

    A stochastic gradient method with an exponential convergence rate for finite training sets , author=. Advances in Neural Information Processing Systems , volume=

  60. [68]

    Mathematical Programming , volume=

    Minimizing finite sums with the stochastic average gradient , author=. Mathematical Programming , volume=. 2017 , publisher=

    2017

  61. [69]

    International Conference on Machine Learning , pages=

    No one idles: Efficient heterogeneous federated learning with parallel edge and server computation , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  62. [70]

    IEEE Transactions on Mobile Computing , year=

    Achieving linear speedup in asynchronous federated learning with heterogeneous clients , author=. IEEE Transactions on Mobile Computing , year=

  63. [71]

    International Conference on Artificial Intelligence and Statistics , pages=

    Asynchronous distributed optimization with stochastic delays , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

  64. [72]

    Incremental Aggregated Asynchronous

    Xiaolu Wang and Yuchang Sun and Hoi To Wai and Jun Zhang , year=. Incremental Aggregated Asynchronous

  65. [73]

    SIAM Journal on Optimization , volume=

    Global convergence rate of proximal incremental aggregated gradient methods , author=. SIAM Journal on Optimization , volume=. 2018 , publisher=

    2018

  66. [74]

    SIAM Journal on Optimization , volume=

    Perturbed iterate analysis for asynchronous stochastic optimization , author=. SIAM Journal on Optimization , volume=. 2017 , publisher=

    2017

  67. [75]

    Advances in Neural Information Processing Systems , volume=

    Distributed delayed stochastic optimization , author=. Advances in Neural Information Processing Systems , volume=

  68. [76]

    arXiv preprint arXiv:2502.08206 , year=

    Optimizing Asynchronous Federated Learning: A Delicate Trade-Off Between Model-Parameter Staleness and Update Frequency , author=. arXiv preprint arXiv:2502.08206 , year=

  69. [77]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  70. [78]

    Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication , year =

    Anastasia Koloskova and Sebastian U Stich and Martin Jaggi , booktitle =. Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication , year =

  71. [79]

    Language Models are Few-Shot Learners , url =

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom a...

  72. [80]

    Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal=

  73. [81]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

  74. [82]

    Cong Fang and Chris Junchi Li and Zhouchen Lin and Tong Zhang , booktitle =

  75. [83]

    The 34th International Conference on Machine Learning , year =

    Nguyen, Lam and Liu, Jie and Scheinberg, Katya and Tak. The 34th International Conference on Machine Learning , year =

  76. [84]

    arXiv preprint arXiv:2106.05203 , title =

    Peter Richt\'. arXiv preprint arXiv:2106.05203 , title =

  77. [86]

    International Conference on Machine Learning , title =

    Zhize Li and Dmitry Kovalev and Xun Qian and Peter Richt\'. International Conference on Machine Learning , title =

  78. [87]

    Distributed learning with compressed gradients , year =

    Sarit Khirirat and Hamid Reza Feyzmahdavian and Mikael Johansson , journal =. Distributed learning with compressed gradients , year =

  79. [88]

    The convergence of sparsified gradient methods , booktitle =

    Alistarh, Dan and Hoefler, Torsten and Johansson, Mikael and Khirirat, Sarit and Konstantinov, Nikola and Renggli, C\'. The convergence of sparsified gradient methods , booktitle =

  80. [89]

    Fifteenth Annual Conference of the International Speech Communication Association , year =

    Seide, Frank and Fu, Hao and Droppo, Jasha and Li, Gang and Yu, Dong , title =. Fifteenth Annual Conference of the International Speech Communication Association , year =

  81. [90]

    Thijs Vogels and Sai Praneeth Karimireddy and Martin Jaggi , booktitle =. Power

  82. [91]

    38th International Conference on Machine Learning , title =

    Eduard Gorbunov and Konstantin Burlachenko and Zhize Li and Peter Richt\'. 38th International Conference on Machine Learning , title =

  83. [92]

    34th Conference on Neural Information Processing Systems (NeurIPS 2020) , title =

    Gorbunov, Eduard and Kovalev, Dmitry and Makarenko, Dmitry and Richt\'. 34th Conference on Neural Information Processing Systems (NeurIPS 2020) , title =

    2020

  84. [93]

    Advances in Neural Information Processing Systems , volume=

    Accelerating stochastic gradient descent using predictive variance reduction , author=. Advances in Neural Information Processing Systems , volume=

  85. [94]

    Improved

    Allen-Zhu, Zeyuan and Yuan, Yang , booktitle =. Improved. 2016 , editor =

    2016

  86. [95]

    OPT2020: 12th Annual Workshop on Optimization for Machine Learning (NeurIPS 2020 Workshop) , title =

    Xun Qian and Hanze Dong and Peter Richt\'. OPT2020: 12th Annual Workshop on Optimization for Machine Learning (NeurIPS 2020 Workshop) , title =

    2020

  87. [96]

    DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated Compression , year =

    Tang, Hanlin and Yu, Chen and Lian, Xiangru and Zhang, Tong and Liu, Ji , booktitle =. DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated Compression , year =

  88. [97]

    Proceedings of the 35th International Conference on Machine Learning , year =

    Wu, Jiaxiang and Huang, Weidong and Huang, Junzhou and Zhang, Tong , title =. Proceedings of the 35th International Conference on Machine Learning , year =

  89. [98]

    Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees , year =

    Constantin Philippenko and Aymeric Dieuleveut , journal =. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees , year =

  90. [99]

    arXiv preprint arXiv:1901.09269 , title =

    Mishchenko, Konstantin and Gorbunov, Eduard and Tak. arXiv preprint arXiv:1901.09269 , title =

    1901

  91. [100]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  92. [101]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  93. [102]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  94. [103]

    Better Theory for

    Khaled, Ahmed and Richt. Better Theory for. arXiv preprint arXiv:2002.03329 , year=

    2002

  95. [104]

    Analysis of

    Ajalloeian, Ahmad and Stich, Sebastian U , journal=. Analysis of

  96. [105]

    Journal of Machine Learning Research , volume=

    On biased compression for distributed learning , author=. Journal of Machine Learning Research , volume=

  97. [106]

    Information and Inference: A Journal of the IMA , title =

    Mher Safaryan and Egor Shulgin and Peter Richt\'. Information and Inference: A Journal of the IMA , title =

  98. [107]

    International Conference on Machine Learning , pages=

    Li, Zhize and Bao, Hongyan and Zhang, Xiangliang and Richt. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  99. [108]

    The Error-Feedback Framework: Better Rates for

    Stich, Sebastian and Sai Praneeth Karimireddy , journal=. The Error-Feedback Framework: Better Rates for

  100. [109]

    Error Feedback Fixes

    Karimireddy, Sai Praneeth and Rebjock, Quentin and Stich, Sebastian and Jaggi, Martin , booktitle =. Error Feedback Fixes

  101. [110]

    9th International Conference on Learning Representations (ICLR) , year=

    A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning , author=. 9th International Conference on Learning Representations (ICLR) , year=

  102. [111]

    Hanlin Tang and Xiangru Lian and Chen Yu and Tong Zhang and Ji Liu , booktitle =

  103. [112]

    and Cordonnier, J.-B

    Stich, Sebastian U. and Cordonnier, J.-B. and Jaggi, Martin , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  104. [113]

    Artificial Intelligence and Statistics , pages=

    Communication-efficient learning of deep networks from decentralized data , author=. Artificial Intelligence and Statistics , pages=. 2017 , organization=

    2017

  105. [114]

    34th Conference on Neural Information Processing Systems (NeurIPS) , title =

    Gorbunov, Eduard and Kovalev, Dmitry and Makarenko, Dmitry and Richt\'. 34th Conference on Neural Information Processing Systems (NeurIPS) , title =

  106. [115]

    International Conference on Learning Representations (ICLR) , year=

    Decentralized Deep Learning with Arbitrary Communication Compression , author=. International Conference on Learning Representations (ICLR) , year=

  107. [116]

    Advances in Neural Information Processing Systems (NIPS) , year =

    Alistarh, Dan and Grubic, Demjan and Li, Jerry and Tomioka, Ryota and Vojnovic, Milan , title =. Advances in Neural Information Processing Systems (NIPS) , year =

  108. [117]

    Debraj Basu and Deepesh Data and Can Karakus and Suhas Diggavi , booktitle=

  109. [118]

    Cong Xie and Shuai Zheng and Oluwasanmi Koyejo and Indranil Gupta and Mu Li and Haibin Lin , booktitle=

  110. [119]

    9th International Conference on Learning Representations (ICLR) , title =

    Horv\'. 9th International Conference on Learning Representations (ICLR) , title =

  111. [120]

    2011 , publisher=

    Chang, Chih-Chung and Lin, Chih-Jen , journal=. 2011 , publisher=

    2011

  112. [121]

    arXiv preprint arXiv:1905.05920 , year=

    Hybrid stochastic gradient descent algorithms for stochastic nonconvex optimization , author=. arXiv preprint arXiv:1905.05920 , year=

    1905 arXiv

  113. [122]

    2018 , publisher=

    Lectures on Convex Optimization , author=. 2018 , publisher=

    2018

  114. [123]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

  115. [124]

    2009 , jnumber =

    Learning multiple layers of features from tiny images , author=. 2009 , jnumber =

    2009

  116. [125]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Pytorch: An imperative style, high-performance deep learning library , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  117. [126]

    ATT Labs [Online]

    MNIST handwritten digit database , author=. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , volume=

  118. [127]

    , title =

    Knuth, Donald E. , title =. 1997 , isbn =

    1997

  119. [128]

    1938 , publisher=

    Statistical tables for biological, agricultural aad medical research , author=. 1938 , publisher=

    1938

  120. [129]

    arXiv preprint arXiv:2005.00224 , year=

    Distributed stochastic non-convex optimization: Momentum-based variance reduction , author=. arXiv preprint arXiv:2005.00224 , year=

    2005

  121. [130]

    arXiv preprint arXiv:1912.06036 , year=

    Parallel Restarted SPIDER--Communication Efficient Distributed Nonconvex Optimization with Optimal Computation Complexity , author=. arXiv preprint arXiv:1912.06036 , year=

    1912

  122. [131]

    Momentum-based variance reduction in non-convex

    Cutkosky, Ashok and Orabona, Francesco , journal=. Momentum-based variance reduction in non-convex

  123. [132]

    Journal of Machine Learning Research , year =

    Dongruo Zhou and Pan Xu and Quanquan Gu , title =. Journal of Machine Learning Research , year =

  124. [133]

    Compressed communication for distributed deep learning: Survey and quantitative evaluation , author=

  125. [134]

    In Neural Information Processing Systems , year=

    Richt. In Neural Information Processing Systems , year=

  126. [135]

    Optimization Methods and Software , year=

    Stochastic distributed learning with gradient quantization and variance reduction , author=. Optimization Methods and Software , year=

  127. [136]

    International Conference on Artificial Intelligence and Statistics , pages=

    Federated learning with compression: Unified analysis and sharp guarantees , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2021 , organization=

    2021

  128. [137]

    arXiv e-prints , pages=

    Improved convergence rates for non-convex federated learning with compression , author=. arXiv e-prints , pages=

  129. [138]

    arXiv preprint arXiv:2102.07845 , year=

    MARINA: Faster non-convex distributed learning with compression , author=. arXiv preprint arXiv:2102.07845 , year=

  130. [139]

    arXiv preprint arXiv:2005.14165 , year=

    Language models are few-shot learners , author=. arXiv preprint arXiv:2005.14165 , year=

    2005 arXiv

  131. [140]

    arXiv preprint arXiv:2105.12806 , year=

    A Universal Law of Robustness via Isoperimetry , author=. arXiv preprint arXiv:2105.12806 , year=

  132. [141]

    arXiv preprint arXiv:2012.06188 , year=

    Recent theoretical advances in non-convex optimization , author=. arXiv preprint arXiv:2012.06188 , year=

    2012

  133. [142]

    arXiv preprint arXiv:1712.00409 , year=

    Deep learning scaling is predictable, empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    arXiv

  134. [143]

    arXiv preprint arXiv:2102.12092 , year=

    Zero-shot text-to-image generation , author=. arXiv preprint arXiv:2102.12092 , year=

    arXiv

  135. [144]

    2016 , booktitle =

    Federated Learning: Strategies for Improving Communication Efficiency , author =. 2016 , booktitle =

    2016

  136. [145]

    arXiv preprint arXiv:1610.02527 , year=

    Federated optimization: Distributed machine learning for on-device intelligence , author=. arXiv preprint arXiv:1610.02527 , year=

    arXiv

  137. [146]

    arXiv preprint arXiv:2107.06917 , year=

    A field guide to federated optimization , author=. arXiv preprint arXiv:2107.06917 , year=

  138. [147]

    arXiv preprint arXiv:1902.01046 , year=

    Towards federated learning at scale: System design , author=. arXiv preprint arXiv:1902.01046 , year=

    1902 arXiv

  139. [148]

    Proceedings of Machine Learning and Systems , volume=

    Federated optimization in heterogeneous networks , author=. Proceedings of Machine Learning and Systems , volume=

  140. [149]

    Vogels, Thijs and He, Lie and Koloskova, Anastasiia and Karimireddy, Sai Praneeth and Lin, Tao and Stich, Sebastian U and Jaggi, Martin , journal=

  141. [150]

    IEEE Transactions on Automatic Control , volume=

    Distributed optimization over time-varying directed graphs , author=. IEEE Transactions on Automatic Control , volume=. 2014 , publisher=

    2014

  142. [151]

    Mathematical Programming , pages=

    A hybrid stochastic optimization framework for composite nonconvex optimization , author=. Mathematical Programming , pages=. 2021 , publisher=

    2021

  143. [152]

    arXiv preprint arXiv:2008.09055 , year=

    An optimal hybrid variance-reduced algorithm for stochastic composite nonconvex optimization , author=. arXiv preprint arXiv:2008.09055 , year=

    2008

  144. [153]

    Foundations and Trends

    Advances and open problems in federated learning , author=. Foundations and Trends. 2021 , publisher=

    2021

  145. [154]

    arXiv preprint arXiv:2202.01268 , year=

    Tyurin, Alexander and Richt. arXiv preprint arXiv:2202.01268 , year=

  146. [155]

    Advances in Neural Information Processing Systems , volume=

    Gradient sparsification for communication-efficient distributed optimization , author=. Advances in Neural Information Processing Systems , volume=

  147. [156]

    Mathematical and Scientific Machine Learning , note=

    Natural compression for distributed deep learning , author=. Mathematical and Scientific Machine Learning , note=

  148. [157]

    Natural compression for distributed deep learning , booktitle =

    Samuel Horv\'. Natural compression for distributed deep learning , booktitle =. 2022 , note =

    2022

  149. [158]

    Sparsified

    Stich, Sebastian U and Cordonnier, Jean-Baptiste and Jaggi, Martin , journal=. Sparsified

  150. [159]

    arXiv preprint arXiv:2110.03300 , year=

    Permutation Compressors for Provably Faster Distributed Nonconvex Optimization , author=. arXiv preprint arXiv:2110.03300 , year=

  151. [160]

    arXiv preprint arXiv:2203.04925 , year=

    Correlated quantization for distributed mean estimation and optimization , author=. arXiv preprint arXiv:2203.04925 , year=

  152. [161]

    arXiv preprint arXiv:2110.03294 , year=

    Fatkhullin, Ilyas and Sokolov, Igor and Gorbunov, Eduard and Li, Zhize and Richt. arXiv preprint arXiv:2110.03294 , year=

  153. [162]

    arXiv preprint arXiv:2003.00295 , year=

    Adaptive federated optimization , author=. arXiv preprint arXiv:2003.00295 , year=

    2003

  154. [163]

    arXiv preprint arXiv:1906.04329 , year=

    Federated learning for emoji prediction in a mobile keyboard , author=. arXiv preprint arXiv:1906.04329 , year=

    1906 arXiv

  155. [164]

    arXiv preprint arXiv:1712.01887 , year=

    Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training , author=. arXiv preprint arXiv:1712.01887 , year=

  156. [165]

    Mathematical Programming , volume=

    Lower bounds for finding stationary points I , author=. Mathematical Programming , volume=. 2020 , publisher=

    2020

  157. [166]

    International Conference on Machine Learning , pages=

    Batch normalization: Accelerating deep network training by reducing internal covariate shift , author=. International Conference on Machine Learning , pages=. 2015 , organization=

    2015

  158. [167]

    arXiv preprint arXiv:2112.13097 , year=

    Faster Rates for Compressed Federated Learning with Client-Variance Reduction , author=. arXiv preprint arXiv:2112.13097 , year=

  159. [168]

    arXiv preprint arXiv:2108.04755 , year=

    Zhao, Haoyu and Li, Zhize and Richt. arXiv preprint arXiv:2108.04755 , year=

  160. [169]

    arXiv preprint arXiv:2103.01447 , year=

    Li, Zhize and Hanzely, Slavom. arXiv preprint arXiv:2103.01447 , year=

  161. [170]

    Narayanan, Deepak and Harlap, Aaron and Phanishayee, Amar and Seshadri, Vivek and Devanur, Nikhil R and Ganger, Gregory R and Gibbons, Phillip B and Zaharia, Matei , booktitle=

  162. [171]

    arXiv preprint arXiv:1903.06701 , year=

    Scaling distributed machine learning with in-network aggregation , author=. arXiv preprint arXiv:1903.06701 , year=

    1903

  163. [172]

    2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) , pages=

    Grace: A compressed communication framework for distributed machine learning , author=. 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) , pages=. 2021 , organization=

    2021

  164. [173]

    arXiv preprint arXiv:2205.15580 , year=

    A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting , author=. arXiv preprint arXiv:2205.15580 , year=

  165. [174]

    arXiv preprint arXiv:2206.03665 , year=

    Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression , author=. arXiv preprint arXiv:2206.03665 , year=

  166. [175]

    Advances in Neural Information Processing Systems , volume=

    Preserved central model for faster bidirectional compression in distributed settings , author=. Advances in Neural Information Processing Systems , volume=

  167. [176]

    International Conference on Artificial Intelligence and Statistics , pages=

    Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

  168. [177]

    International Conference on Machine Learning , pages=

    Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  169. [178]

    arXiv preprint arXiv:2205.04180 , year=

    Condat, Laurent and Yi, Kai and Richt. arXiv preprint arXiv:2205.04180 , year=

  170. [179]

    Advances in Neural Information Processing Systems , volume=

    Rethinking gradient sparsification as total error minimization , author=. Advances in Neural Information Processing Systems , volume=

  171. [180]

    Advances in Neural Information Processing Systems , volume=

    Li, Zhize and Richt. Advances in Neural Information Processing Systems , volume=

  172. [181]

    arXiv preprint arXiv:2006.11573 , year=

    Unified analysis of stochastic gradient methods for composite convex and smooth optimization , author=. arXiv preprint arXiv:2006.11573 , year=

    2006

  173. [182]

    International Conference on Learning Representations , year=

    Decoupled weight decay regularization , author=. International Conference on Learning Representations , year=

  174. [183]

    Variance reduced

    Malinovsky, Grigory and Yi, Kai and Richt. Variance reduced. Advances in Neural Information Processing Systems , volume=

  175. [184]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Mishchenko, Konstantin and Malinovsky, Grigory and Stich, Sebastian and Richt. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  176. [185]

    Alexander Tyurin and Danil Sivtsov , booktitle=. Birch. 2026 , url=

    2026

  177. [186]

    , keywords =

    Vardhan, Harsh and Stich, Sebastian U. , keywords =. Tackling benign nonconvexity with smoothing and stochastic gradients , publisher =. 2022 , copyright =

    2022

  178. [187]

    SIAM Journal on Optimization , volume=

    Randomized smoothing for stochastic optimization , author=. SIAM Journal on Optimization , volume=. 2012 , publisher=

    2012

  179. [188]

    arXiv preprint arXiv:2202.02831 , year=

    Anticorrelated noise injection for improved generalization , author=. arXiv preprint arXiv:2202.02831 , year=

  180. [189]

    International Conference on Artificial Intelligence and Statistics , pages=

    A double residual compression algorithm for efficient distributed learning , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2020 , organization=

    2020

  181. [190]

    arXiv preprint arXiv:2006.14591 , year=

    Artemis: tight convergence guarantees for bidirectional compression in federated learning , author=. arXiv preprint arXiv:2006.14591 , year=

    2006

  182. [191]

    arXiv preprint arXiv:2106.02969 , year=

    FedNL: Making Newton-type methods applicable to federated learning , author=. arXiv preprint arXiv:2106.02969 , year=

  183. [192]

    arXiv preprint arXiv:1308.6370 , year=

    Fast convergence of stochastic gradient descent under a strong growth condition , author=. arXiv preprint arXiv:1308.6370 , year=

    arXiv

  184. [193]

    Fast and faster convergence of

    Vaswani, Sharan and Bach, Francis and Schmidt, Mark , booktitle=. Fast and faster convergence of. 2019 , organization=

    2019

  185. [194]

    The 23rd International Conference on Artificial Intelligence and Statistics , pages=

    Fast and furious convergence: Stochastic second order methods under interpolation , author=. The 23rd International Conference on Artificial Intelligence and Statistics , pages=. 2020 , organization=

    2020

  186. [195]

    Recht, Benjamin and Re, Christopher and Wright, Stephen and Niu, Feng , journal=

  187. [196]

    Bias-variance reduced local

    Murata, Tomoya and Suzuki, Taiji , journal=. Bias-variance reduced local

  188. [197]

    Sharper convergence guarantees for asynchronous

    Koloskova, Anastasiia and Stich, Sebastian U and Jaggi, Martin , journal=. Sharper convergence guarantees for asynchronous

  189. [198]

    arXiv preprint arXiv:2601.19379 , year=

    Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noise , author=. arXiv preprint arXiv:2601.19379 , year=

  190. [199]

    Advances in Neural Information Processing Systems , volume=

    DeepReduce: A Sparse-tensor Communication Framework for Federated Deep Learning , author=. Advances in Neural Information Processing Systems , volume=

  191. [200]

    A unified theory of

    Gorbunov, Eduard and Hanzely, Filip and Richt. A unified theory of. International Conference on Artificial Intelligence and Statistics , pages=. 2020 , organization=

    2020

  192. [201]

    2003 , publisher=

    Introductory lectures on convex optimization: A basic course , author=. 2003 , publisher=

    2003

  193. [202]

    International Conference on Machine Learning , pages=

    Optimal complexity in decentralized training , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  194. [203]

    2020 , publisher=

    First-order and stochastic optimization methods for machine learning , author=. 2020 , publisher=

    2020

  195. [204]

    Mathematical Programming , pages=

    Lower bounds for non-convex stochastic optimization , author=. Mathematical Programming , pages=. 2022 , publisher=

    2022

  196. [205]

    Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E , journal=

  197. [206]

    nature , volume=

    Dermatologist-level classification of skin cancer with deep neural networks , author=. nature , volume=. 2017 , publisher=

    2017

  198. [207]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

  199. [208]

    Integrated ensemble

    Rajpoot, Reenu and Gour, Mahesh and Jain, Sweta and Semwal, Vijay Bhaskar , journal=. Integrated ensemble. 2024 , publisher=

    2024

  200. [209]

    Applied Sciences , volume=

    Comparing vision transformers and convolutional neural networks for image classification: A literature review , author=. Applied Sciences , volume=. 2023 , publisher=

    2023

  201. [210]

    IEEE Transactions on Intelligent Vehicles , volume=

    Recent advancements in end-to-end autonomous driving using deep learning: A survey , author=. IEEE Transactions on Intelligent Vehicles , volume=. 2023 , publisher=

    2023

  202. [211]

    ACM computing surveys (CSUR) , volume=

    Deep learning based recommender system: A survey and new perspectives , author=. ACM computing surveys (CSUR) , volume=. 2019 , publisher=

    2019

  203. [212]

    SIAM Journal on Optimization , volume=

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming , author=. SIAM Journal on Optimization , volume=. 2013 , publisher=

    2013

  204. [213]

    Advances in Neural Information Processing Systems , volume=

    An even more optimal stochastic optimization algorithm: minibatching and interpolation learning , author=. Advances in Neural Information Processing Systems , volume=

  205. [214]

    1983 , publisher=

    Problem complexity and method efficiency in optimization , author=. 1983 , publisher=

    1983

  206. [215]

    Advances in Neural Information Processing Systems , volume=

    Better mini-batch algorithms via accelerated gradient methods , author=. Advances in Neural Information Processing Systems , volume=

  207. [216]

    International Conference on Machine Learning , pages=

    Gower, Robert Mansel and Loizou, Nicolas and Qian, Xun and Sailanbayev, Alibek and Shulgin, Egor and Richt. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  208. [217]

    Slow and stale gradients can win the race: Error-runtime trade-offs in distributed

    Dutta, Sanghamitra and Joshi, Gauri and Ghosh, Soumyadip and Dube, Parijat and Nagpurkar, Priya , booktitle=. Slow and stale gradients can win the race: Error-runtime trade-offs in distributed. 2018 , organization=

    2018

  209. [218]

    Revisiting distributed synchronous

    Chen, Jianmin and Pan, Xinghao and Monga, Rajat and Bengio, Samy and Jozefowicz, Rafal , journal=. Revisiting distributed synchronous

  210. [219]

    International Conference on Machine Learning , pages=

    Nguyen, Lam and Nguyen, Phuong Ha and Dijk, Marten and Richt. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  211. [220]

    Algorithmic Learning Theory , pages=

    A tight convergence analysis for stochastic gradient descent with delayed updates , author=. Algorithmic Learning Theory , pages=. 2020 , organization=

    2020

  212. [221]

    IEEE Transactions on Automatic Control , volume=

    An asynchronous mini-batch algorithm for regularized stochastic optimization , author=. IEEE Transactions on Automatic Control , volume=. 2016 , publisher=

    2016

  213. [222]

    Asynchronous

    Mishchenko, Konstantin and Bach, Francis and Even, Mathieu and Woodworth, Blake E , journal=. Asynchronous

  214. [223]

    Is local

    Woodworth, Blake and Patel, Kumar Kshitij and Stich, Sebastian and Dai, Zhen and Bullins, Brian and Mcmahan, Brendan and Shamir, Ohad and Srebro, Nathan , booktitle=. Is local. 2020 , organization=

    2020

  215. [224]

    arXiv preprint arXiv:2202.08550 , year=

    Delay-adaptive step-sizes for asynchronous learning , author=. arXiv preprint arXiv:2202.08550 , year=

  216. [225]

    Advances in Neural Information Processing Systems , volume=

    Graph oracle models, lower bounds, and gaps for parallel stochastic optimization , author=. Advances in Neural Information Processing Systems , volume=

  217. [226]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    arXiv

  218. [227]

    Advances in Neural Information Processing Systems , volume=

    Asynchronous Stochastic Optimization Robust to Arbitrary Delays , author=. Advances in Neural Information Processing Systems , volume=

  219. [228]

    SIAM Journal on Scientific Computing , volume=

    Arock: an algorithmic framework for asynchronous parallel coordinate updates , author=. SIAM Journal on Scientific Computing , volume=. 2016 , publisher=

    2016

  220. [229]

    International Conference on Machine Learning , pages=

    A delay-tolerant proximal-gradient algorithm for distributed learning , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  221. [230]

    arXiv preprint arXiv:1610.05507 , year=

    Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Server , author=. arXiv preprint arXiv:1610.05507 , year=

    arXiv

  222. [231]

    arXiv preprint arXiv:1607.04818 , year=

    Asynchronous parallel algorithms for nonconvex big-data optimization: Model and convergence , author=. arXiv preprint arXiv:1607.04818 , year=

    arXiv

  223. [232]

    International Conference on Artificial Intelligence and Statistics , pages=

    Federated learning with buffered asynchronous aggregation , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

  224. [233]

    Horovod: Fast and easy distributed deep learning in

    Alexander Sergeev and Mike Del Balso , Journal =. Horovod: Fast and easy distributed deep learning in

  225. [234]

    2011 , publisher=

    Convex analysis and monotone operator theory in Hilbert spaces , author=. 2011 , publisher=

    2011

  226. [235]

    Foundations and trends

    Proximal algorithms , author=. Foundations and trends. 2014 , publisher=

    2014

  227. [236]

    Carmon, Yair and Hinder, Oliver , booktitle=. Making. 2022 , organization=

    2022

  228. [237]

    Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Model , volume =

    Tyurin, Alexander and Richt. Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Model , volume =. Advances in Neural Information Processing Systems , pages =

  229. [238]

    arXiv preprint arXiv:1602.05629 , volume=

    Federated learning of deep networks using model averaging , author=. arXiv preprint arXiv:1602.05629 , volume=

  230. [239]

    2022 , eprint=

    Adaptive Stochastic Gradient Descent for Fast and Communication-Efficient Distributed Learning , author=. 2022 , eprint=

    2022

  231. [240]

    Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers , year=

    Hanna, Serge Kas and Bitar, Rawad and Parag, Parimal and Dasari, Venkat and El Rouayheb, Salim , booktitle=. Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers , year=

  232. [241]

    2024 , organization=

    Islamov, Rustem and Safaryan, Mher and Alistarh, Dan , booktitle=. 2024 , organization=

    2024

  233. [242]

    Proceedings of the IEEE , volume=

    Gradient-based learning applied to document recognition , author=. Proceedings of the IEEE , volume=. 1998 , publisher=

    1998

  234. [243]

    ACM Transactions on Mathematical Software (TOMS) , volume=

    Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , author=. ACM Transactions on Mathematical Software (TOMS) , volume=. 1997 , publisher=

    1997

  235. [244]

    2011 , publisher=

    Fundamentals of queueing theory , author=. 2011 , publisher=

    2011

  236. [245]

    arXiv preprint arXiv:1606.04487 , year=

    Hadjis, Stefan and Zhang, Ce and Mitliagkas, Ioannis and Iter, Dan and R. arXiv preprint arXiv:1606.04487 , year=

    arXiv

  237. [246]

    2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages=

    Asynchrony begets momentum, with an application to deep learning , author=. 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages=. 2016 , organization=

    2016

  238. [247]

    Concentration inequalities under sub-

    Maurer, Andreas and Pontil, Massimiliano , journal=. Concentration inequalities under sub-

  239. [248]

    2002 , publisher=

    Finite-time Analysis of the Multiarmed Bandit Problem , author=. 2002 , publisher=

    2002

  240. [249]

    Communications of the ACM , volume=

    The tail at scale , author=. Communications of the ACM , volume=. 2013 , publisher=

    2013

  241. [250]

    2010 , publisher=

    Analysis and synthesis of computer systems , author=. 2010 , publisher=

    2010

  242. [251]

    Journal of Computer and System Sciences , volume=

    Combinatorial bandits , author=. Journal of Computer and System Sciences , volume=. 2012 , publisher=

    2012

  243. [252]

    Advances in Neural Information Processing Systems , volume=

    Combinatorial bandits revisited , author=. Advances in Neural Information Processing Systems , volume=

  244. [253]

    International Conference on Machine Learning , pages=

    Combinatorial multi-armed bandit: General framework and applications , author=. International Conference on Machine Learning , pages=. 2013 , organization=

    2013

  245. [254]

    Advances in Neural Information Processing Systems , volume=

    Stochastic online greedy learning with semi-bandit feedbacks , author=. Advances in Neural Information Processing Systems , volume=

  246. [255]

    Advances in Neural Information Processing Systems , volume=

    Combinatorial multi-armed bandit with general reward functions , author=. Advances in Neural Information Processing Systems , volume=

  247. [256]

    International Conference on Machine Learning , pages=

    Thompson sampling for combinatorial semi-bandits , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  248. [257]

    IEEE/ACM Transactions on Networking , volume=

    Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations , author=. IEEE/ACM Transactions on Networking , volume=. 2012 , publisher=

    2012

  249. [258]

    Artificial Intelligence and Statistics , pages=

    Tight regret bounds for stochastic combinatorial semi-bandits , author=. Artificial Intelligence and Statistics , pages=. 2015 , organization=

    2015

  250. [259]

    2020 , publisher=

    Bandit algorithms , author=. 2020 , publisher=

    2020

  251. [260]

    Empirical

    Mnih, Volodymyr and Szepesv. Empirical. Proceedings of the 25th International Conference on Machine learning , pages=

  252. [261]

    Asynchronous distributed

    Zhang, Ruiliang and Kwok, James , booktitle=. Asynchronous distributed. 2014 , organization=

    2014

  253. [262]

    NIPS Workshop on Optimization for Machine Learning , volume=

    Distributed delayed proximal gradient methods , author=. NIPS Workshop on Optimization for Machine Learning , volume=

  254. [263]

    Advances in Neural Information Processing Systems , volume=

    Tyurin, Alexander and Pozzi, Marta and Ilin, Ivan and Richt. Advances in Neural Information Processing Systems , volume=

  255. [264]

    Tyurin, Alexander and Gruntkowska, Kaja and Richt. Freya. Advances in Neural Information Processing Systems , volume=

  256. [265]

    Advances in Neural Information Processing Systems , volume=

    On the Optimal Time Complexities in Decentralized Stochastic Asynchronous Optimization , author=. Advances in Neural Information Processing Systems , volume=

  257. [266]

    The error-feedback framework:

    Stich, Sebastian U and Karimireddy, Sai Praneeth , journal=. The error-feedback framework:

  258. [267]

    International Conference on Machine Learning , pages=

    Asynchronous decentralized parallel stochastic gradient descent , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  259. [268]

    International Conference on Machine Learning , pages=

    Optimal algorithms for smooth and strongly convex distributed optimization in networks , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  260. [269]

    Moment inequalities for functions of independent random variables , author=

  261. [270]

    2019 , editor =

    Yurochkin, Mikhail and Agarwal, Mayank and Ghosh, Soumya and Greenewald, Kristjan and Hoang, Nghia and Khazaeni, Yasaman , booktitle =. 2019 , editor =

    2019

  262. [271]

    Fashion-

    Xiao, Han and Rasul, Kashif and Vollgraf, Roland , journal=. Fashion-

  263. [272]

    2017 , editor =

    Leblond, Rémi and Pedregosa, Fabian and Lacoste-Julien, Simon , booktitle =. 2017 , editor =

    2017

  264. [273]

    Journal of Machine Learning Research , year =

    Remi Leblond and Fabian Pedregosa and Simon Lacoste-Julien , title =. Journal of Machine Learning Research , year =

  265. [274]

    Proceedings of the IEEE , volume=

    Advances in asynchronous parallel and distributed optimization , author=. Proceedings of the IEEE , volume=. 2020 , publisher=

    2020

  266. [275]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Fast asynchronous parallel stochastic gradient descent: A lock-free approach with convergence guarantee , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  267. [276]

    Advances in Neural Information Processing Systems , volume=

    On variance reduction in stochastic gradient descent and its asynchronous variants , author=. Advances in Neural Information Processing Systems , volume=

  268. [277]

    International Conference on Machine Learning , pages=

    A simple stochastic variance reduced algorithm with fast convergence rates , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  269. [278]

    Data centres will use twice as much energy by 2030-driven by

    Chen, Sophia , journal=. Data centres will use twice as much energy by 2030-driven by

  270. [279]

    Journal of Machine Learning Research , volume=

    Palm: Scaling language modeling with pathways , author=. Journal of Machine Learning Research , volume=

  271. [280]

    2025 , organization=

    Yuan, Huizhuo and Liu, Yifeng and Wu, Shuang and Xun, Zhou and Gu, Quanquan , booktitle=. 2025 , organization=

    2025

  272. [281]

    International Conference on Machine Learning , pages=

    Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  273. [282]

    Khirirat, Sarit and Sadiev, Abdurakhmon and Demidovich, Yury and Richt. Better. arXiv preprint arXiv:2512.13227 , year=

  274. [283]

    Methods for Convex

    Gorbunov, Eduard and Tupitsa, Nazarii and Choudhury, Sayantan and Aliev, Alen and Richt. Methods for Convex. The Thirteenth International Conference on Learning Representations , year=

  275. [284]

    Training Deep Learning Models with Norm-Constrained

    Pethick, Thomas and Xie, Wanyun and Antonakopoulos, Kimon and Zhu, Zhenyu and Silveti-Falls, Antonio and Cevher, Volkan , booktitle =. Training Deep Learning Models with Norm-Constrained. 2025 , editor =

    2025

  276. [285]

    Generalized Gradient Norm Clipping & Non-Euclidean

    Thomas Pethick and Wanyun Xie and Mete Erdogan and Kimon Antonakopoulos and Tony Silveti-Falls and Volkan Cevher , booktitle=. Generalized Gradient Norm Clipping & Non-Euclidean. 2025 , url=

    2025

  277. [286]

    Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages =

    Parameter-Agnostic Optimization under Relaxed Smoothness , author =. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages =. 2024 , editor =

    2024

  278. [287]

    Beyond the Ideal: Analyzing the Inexact

    Egor Shulgin and Sultan AlRashed and Peter Richt. Beyond the Ideal: Analyzing the Inexact. The 29th International Conference on Artificial Intelligence and Statistics , year=

  279. [288]

    2024 , url =

    Keller Jordan and Jeremy Bernstein and Brendan Rappazzo and @fernbear.bsky.social and Boza Vlado and You Jiacheng and Franz Cesista and Braden Koszarsky and @Grad62304977 , title =. 2024 , url =

    2024

  280. [289]

    Muon is scalable for

    Liu, Jingyuan and Su, Jianlin and Yao, Xingcheng and Jiang, Zhejun and Lai, Guokun and Du, Yulun and Qin, Yidao and Xu, Weixin and Lu, Enzhe and Yan, Junjie and others , journal=. Muon is scalable for

  281. [290]

    Practical efficiency of

    Shah, Ishaan and Polloreno, Anthony M and Stratos, Karl and Monk, Philip and Chaluvaraju, Adarsh and Hojel, Andrew and Ma, Andrew and Thomas, Anil and Tanwer, Ashish and Shah, Darsh J and others , journal=. Practical efficiency of

  282. [291]

    The Fourteenth International Conference on Learning Representations , year=

    Fantastic Pretraining Optimizers and Where to Find Them , author=. The Fourteenth International Conference on Learning Representations , year=

  283. [292]

    arXiv preprint arXiv:2503.12645 , year=

    Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization , author=. arXiv preprint arXiv:2503.12645 , year=

  284. [293]

    Beyond Convexity: Stochastic Quasi-Convex Optimization , url =

    Hazan, Elad and Levy, Kfir and Shalev-Shwartz, Shai , booktitle =. Beyond Convexity: Stochastic Quasi-Convex Optimization , url =

  285. [294]

    Momentum improves normalized

    Cutkosky, Ashok and Mehta, Harsh , booktitle=. Momentum improves normalized. 2020 , organization=

    2020

  286. [295]

    arXiv preprint arXiv:2409.20325 , year=

    Old Optimizer, New Norm: An Anthology , author=. arXiv preprint arXiv:2409.20325 , year=

    arXiv

  287. [296]

    Naval Research Logistics Quarterly , volume =

    Frank, Marguerite and Wolfe, Philip , title =. Naval Research Logistics Quarterly , volume =. doi:https://doi.org/10.1002/nav.3800030109 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1002/nav.3800030109 , year =

    doi

  288. [297]

    LATIN 2008: Theoretical Informatics , year=

    Hazan, Elad , title=. LATIN 2008: Theoretical Informatics , year=

    2008

  289. [298]

    , title =

    Clarkson, Kenneth L. , title =. ACM Trans. Algorithms , month = sep, articleno =. 2010 , issue_date =. doi:10.1145/1824777.1824783 , abstract =

    2010 doi

  290. [299]

    Revisiting

    Jaggi, Martin , booktitle =. Revisiting. 2013 , editor =

    2013

  291. [300]

    arXiv preprint arXiv:2505.13416 , year=

    Gluon: Making muon & scion great again! (bridging theory and practice of lmo-based optimizers for llms) , author=. arXiv preprint arXiv:2505.13416 , year=

Pith tools

Reviewed May 21, 2026 · model on record in the stance chip above.