Pith. sign in

REVIEW 3 major objections 4 minor 8 cited by

Overcoming the Communication-Performance Tradeoff in LLM Pretraining

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SparseLoCo replaces DiLoCo's global Nesterov momentum with a per-replica error-feedback accumulator, allowing Top-k sparsification and 2-bit quantization down to 1–3% communication density while achieving lower final loss than dense DiLoCo.

desk verdict SparseLoCo is a real algorithmic contribution with solid empirical support, though the abstract overpromises MoE-scale validation and the momentum-approximation story needs a sharper ablation. read the letter →

arxiv 2508.15706 v3 pith:DW7ZUEYW submitted 2025-08-21 cs.LG

Amir Sarfi , Benjamin Thérien , Joel Lidin , Eugene Belilovsky This is my paper
classification cs.LG
keywords communication-efficientpretrainingDiLoCoerrorfeedbackTop-ksparsification2-bitquantizationlocaloutermomentumdistributedLLMtrainingsparseaggregation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed LLM training typically pays a communication cost proportional to the model size at each synchronization, even when synchronizations are infrequent. SparseLoCo is a method that cuts that cost to a few percent of the model: each worker sends only the largest entries of its pseudo-gradient, quantized to 2 bits, and the paper shows this does not hurt—it helps. The key move is to replace DiLoCo's global outer momentum (a smoothed running average of the aggregate update) with a per-worker error-feedback buffer that naturally accumulates what hasn't been sent yet. At 3.12% communication density on a 512M-parameter model, SparseLoCo reaches 2.70 final validation loss versus 2.76 for dense full-precision DiLoCo, while sending roughly one-thirtieth the data; at 0.78% density the loss is 2.79, still better than DiLoCo. The improvement holds across model sizes up to 2B parameters, up to 32 workers, and communication intervals from 15 to 250 inner steps. If correct, this means the presumed tradeoff between communication compression and model quality is absent in this regime: aggressive sparsity with error feedback is a better update rule, not just a cheaper one.

What carries the argument

Central machinery: the OuterEF error-feedback accumulator (Algorithm 1, lines 10–12) together with a chunked Top-k sparsifier and 2-bit quantizer. The accumulator e_r on each replica stores the running sum of what the worker has computed but not yet successfully communicated; each outer step transmits Q(TOP-k(e_r)) and immediately subtracts the transmitted vector from e_r. When k is small, this subtraction removes only the largest entries, so the buffer behaves approximately like the local momentum of DiLoCo-LOM, which the paper proves is exactly equivalent to DiLoCo's global momentum (Proposition 1). The chunked Top-k operation (inspired by Xu et al. 2021) selects the largest values within

What would settle it

Train a 7B-parameter model with SparseLoCo at H=15 and 3.12% density, and compare final validation loss to dense DiLoCo; also compute, over the full run, the cosine similarity between each replica's error-feedback buffer and the global-momentum reference of a DiLoCo run. If that similarity falls below the 0.75 the paper measures in the first 20 steps, or if the loss gap flips, the momentum-replacement claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that DiLoCo's global Nesterov outer momentum is not needed when pseudo-gradients are aggressively Top-k sparsified with error feedback; a single per-replica error-feedback buffer (OuterEF) acts as a local stand-in for that momentum. The buffer is updated as e_r ← β e_r + Δ_r, then the quantized Top-k of the buffer is transmitted, and the buffer is reduced by exactly what was sent. When k is small, few entries are subtracted, so the buffer retains almost its full history, closely matching the local-momentum recursion that the paper proves identical to DiLoCo (Proposition 1). The method therefore unifies infrequent communication with high compression, and in experi

Load-bearing premise

The whole method rests on the assumption that a worker's error-feedback buffer — a running sum of what has not yet been sent — closely mimics DiLoCo's global momentum when only the top few entries are sent each round; if that mimicry fails at larger scales or different data, SparseLoCo collapses to the no-momentum DiLoCo variant that the paper shows is much worse (2.868 vs 2.760).

Editorial extensions

If this is right

  • Communication volume per synchronization drops by roughly an order of magnitude: at H=15, SparseLoCo sends 4.25 MB at 0.78% density or 17.01 MB at 3.12%, versus DiLoCo's 512.40 MB, and still beats DiLoCo's final loss.
  • The optimal sparsity level grows with the communication interval H, so longer local runs want denser messages; this gives a simple rule for tuning SparseLoCo.
  • SparseLoCo scales better than DiLoCo with more workers (R=16 and 32), where DiLoCo's loss degrades, and can be combined with streaming DiLoCo to reduce peak bandwidth by a factor of 3.
  • Adding Nesterov outer momentum on top of SparseLoCo hurts badly at high sparsity, so the right design is to drop global momentum entirely rather than to combine it with error feedback.
  • Sparse aggregation is beneficial, not merely tolerable: at matched density SparseLoCo outperforms dense DiLoCo, suggesting sparsity acts as an implicit regularizer or interference reducer in pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof that local averaged momentum equals global momentum (Proposition 1) is exact, but its link to the error-feedback buffer is empirical; an interesting extension would be to derive a bound on how close OuterEF is to DiLoCo-LOM as a function of k, which the paper does not provide.
  • The observed dependence of optimal density on H suggests an adaptive-density schedule over training, which the paper does not test; such a schedule could recover the best of both extremes.
  • The 'sparse aggregation helps' result, which the authors connect to model merging, opens the door to testing other sparse-aggregation operators (e.g., sign-consistent Top-k or variance-weighted selection) in pretraining; the paper only studies magnitude-based Top-k.
  • A live deployment in the appendix reports communication times for 8B and 70B models, but those are single measurements; a controlled multi-peer benchmark with heterogeneous bandwidth would be needed to confirm the practical speedup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SparseLoCo, an algorithm for communication-efficient LLM pretraining that combines DiLoCo's infrequent synchronization with Top-k sparsification and 2-bit quantization of pseudo-gradients. The key methodological move is to replace DiLoCo's global Nesterov outer momentum with a single local error-feedback accumulator (Algorithm 1, lines 10-12), motivated by the claim that under aggressive sparsity the error-feedback buffer closely approximates local outer momentum. The paper provides a clean equivalence proof for a local-momentum variant (DiLoCo-LOM, Proposition 1) and extensive experiments at 178M, 512M, and 2B scales, with varying communication intervals H, worker counts R, and densities. The main empirical claim is that SparseLoCo at carefully chosen densities (e.g., 3.12% at H=15) achieves lower final loss than dense DiLoCo while communicating far fewer bits. The abstract additionally claims validation on an MoE transformer (645M-A273M), but no such experiment appears in the body or appendices.

Significance. If the central claims hold, the paper would be a practical contribution to bandwidth-constrained LLM training: it demonstrates that aggressive sparsification plus quantization can be combined with local-update methods without the typical performance penalty, and it offers a plausible explanation rooted in the interaction between error feedback and outer momentum. The paper's strengths include a machine-checkable linearity proof for the LOM equivalence (Appendix I), a public code repository, a broad sweep over communication intervals and worker counts, and concrete communication-volume analysis including real-world deployment notes. However, the significance is conditional on two load-bearing points: (i) the abstract's MoE-scale validation must actually be present, and (ii) the claimed approximation of outer momentum by the OuterEF buffer must be supported in the operating regime of SparseLoCo, not only in a related but different ablation.

major comments (3)
  1. [Abstract and Section 4] The abstract states that SparseLoCo's improvements are maintained across dense model scales (178M-2B), an MoE transformer (645M-A273M), increasing workers, and increasing communication intervals. However, I could not find any experimental result, table, or even a mention of an MoE model in the main text or any appendix. This is a direct discrepancy between the claimed empirical scope and the evidence supplied. Either the MoE experiments must be added (with architecture, hyperparameters, and loss/benchmark numbers) or the abstract must be revised to remove this claim.
  2. [Section 3.3, Algorithm 1, and Section 4.1] The central algorithmic claim is that OuterEF 'closely approximates' DiLoCo's outer momentum when k is small, permitting the outer momentum to be dropped. The only quantitative support is the cosine-similarity measurement in Section 4.1, which reports >=0.75 averaged over the first 20 outer steps for DiLoCo-LOM-Sub-k at 25% subtraction. This is not the operating regime of SparseLoCo: the main result (Table 2) uses 3.12% density, and the full run has 163 outer steps. The accumulator update e_r <- beta e_r + Delta - hatDelta (Algorithm 1, line 12) subtracts the largest selected components, so the buffer is not simply a momentum buffer; its behavior at 3.12% density over a full run is untested. If the approximation degenerates, SparseLoCo may approach DiLoCo without outer momentum, which Table 1 reports as much worse (2.868 vs 2.760). I ask the authors to report the cosine similarity (or a
  3. [Table 2 and Figure 1] The main comparison in Table 2 reports SparseLoCo at density 3.12% with loss 2.70 versus DiLoCo's 2.76 and AdamW DDP's 2.69. This supports the claim of beating DiLoCo, but the margin over DiLoCo is modest and the method remains slightly worse than the un-compressed DDP baseline. Figure 1 shows that across H the advantage over DiLoCo is achieved only in certain density ranges, and the optimal density shifts with H. This is not itself a flaw, but the paper should be more careful in the abstract and conclusion when saying SparseLoCo 'outperforms' DiLoCo without noting that the claim requires selecting the right density, which is a tuned hyperparameter. The current text does acknowledge this in Figure 1, but the abstract's phrasing is stronger than the evidence.
minor comments (4)
  1. [Table 2 and Appendix G] The 'Pseudo-Grad Size' column is ambiguous: it is not clear whether the reported sizes include the Top-k index overhead. Appendix G discusses index compression and gives bit-per-value numbers, but Table 2 should state explicitly whether indices are included, since this materially affects the communication-volume comparison.
  2. [Algorithm 1] The notation e_r^{(t)} is used in line 10 as a temporary variable and then e_r^{(t+1)} in line 12, which can confuse the reader. A clearer notation such as e_r^{temp} or a re-indexed update would help.
  3. [Appendix C and Appendix A] There are minor typos: 'DILOCO' should be 'DiLoCo' in Appendix C, and 'Hellaswag' should be 'HellaSwag' in Appendix A. Also, the 70B deployment results in Appendix A are described as preliminary and without a comparison baseline, so they should be labeled as anecdotal rather than as experimental evidence.
  4. [Section 4.1] The cosine-similarity result is reported only as '>=0.75' with no details on the spread over replicas or over the 20 steps. Reporting mean and standard deviation, and ideally a small plot, would make the evidence more convincing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central approximation claim is empirical and the derivations are linearity arguments, not restatements of the conclusion.

full rationale

The paper's only formal derivation (Prop. 1) shows DiLoCo-LOM equals DiLoCo by induction/linearity; this is a genuine algebraic identity, not a restatement of SparseLoCo's performance. The key claim that OuterEF approximates local outer momentum under small k is an algorithmic observation supported by Table 1 and the cosine-similarity measurement; it is not defined in terms of the result it explains, nor is any fitted parameter renamed as a prediction. The density and beta hyperparameters are tuned against validation loss, but the paper's reported losses are the outcome of that tuning, not a prediction derived from the approximation. Self-citations (Davari & Belilovsky 2024, Thérien et al. 2025, Lidin et al. 2025) are used for motivation, comparison, or deployment context and are not load-bearing for the central claim. The weakness flagged by the reader—that the momentum-approximation evidence is limited to 20 outer steps and 25% subtraction—is a limitation of evidence, not a circularity. Therefore score 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on tuned hyperparameters (density, beta, learning rates) and on two domain assumptions: that a small-k error feedback buffer approximates Nesterov outer momentum, and that sparse aggregation of salient coordinates is benign or beneficial. No new physical entities are introduced.

free parameters (5)
  • communication density (k/chunk) = 0.78% to 50%, best per H (e.g., 3.12% at H=15)
    The fraction of pseudo-gradient entries transmitted is swept per communication interval and the best-performing value is reported (Figure 1, Tables 2-5).
  • error momentum beta = 0.95 (best in sweep {0.9,0.95,0.999} for 512M SparseLoCo 0.78%)
    Tuned on validation loss; controls the EF buffer decay and the approximation of outer momentum (Table 15).
  • outer learning rate alpha_outer = 1.0 (512M SparseLoCo 0.78%, H=15)
    Tuned per method and density; part of the standard DiLoCo-style outer step (Table 15).
  • EF freeze fraction = 5% of outer steps
    Chosen after ablation (Table 11); improves final validation loss by 0.005.
  • inner learning rate alpha_inner = 1e-3 for SparseLoCo 512M
    Tuned; both SparseLoCo and DiLoCo get separate sweeps (Table 15).
assumptions (4)
  • standard math Averaging commutes with the momentum recursion (linearity).
    Used in Proposition 1 to show LOM equals global momentum (Appendix I).
  • domain assumption The error feedback update with small k behaves like the Nesterov outer momentum of DiLoCo.
    Load-bearing heuristic introduced in Section 3.3; validated empirically via cosine similarity and Table 1, not proven.
  • domain assumption Sparse aggregation of top-saliency pseudo-gradient components improves or maintains model quality relative to dense aggregation.
    Imported from model-merging literature (Yadav et al. 2023; Davari & Belilovsky 2024) and used to explain SparseLoCo's gains over DiLoCo (Section 4.2).
  • domain assumption Token budget 20x model size is compute-optimal (Chinchilla).
    Follows Hoffmann et al. 2022 to set run length; affects all losses quoted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Overcoming the Communication-Performance Tradeoff in LLM Pretraining." pith.science (2026). https://pith.science/paper/DW7ZUEYW

@misc{pith2026250815706,
  author       = {Pith},
  title        = {Pith review of: Overcoming the Communication-Performance Tradeoff in LLM Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DW7ZUEYW}},
  note         = {Machine review of arXiv:2508.15706}
}
read the original abstract

Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwidth-constrained settings, such as across datacenters and over the internet. While these local-update methods achieve communication reduction through reduced synchronization, they still require communicating dense model-sized pseudo-gradients, resulting in a communication bottleneck even for cross-datacenter links. While quantization is often applied to reduce the pseudo-gradient's size, in the context of LLM pre-training, existing approaches have not been able to leverage sparsification without incurring significant performance degradation. In this work, we introduce SparseLoCo, a communication-efficient training algorithm for LLMs that can effectively leverage Top-k sparsification and 2-bit quantization to reach extreme sparsity in the communicated pseudo-gradient, as high as 97-99%, while achieving lower final loss than dense DiLoCo. In our empirical study of language model pre-training, we demonstrate that SparseLoCo's improvements over DiLoCo in performance and pseudo-gradient compression are maintained across dense model scales (178M-2B), an MoE transformer (645M-A273M), increasing the number of workers, and increasing communication intervals.

Figures

Figures reproduced from arXiv: 2508.15706 by the authors.

Figure 1
Figure 1. SparseLoCo outperforms DiLoCo for H ∈ {15, 30, 50, 100} communication inter￾vals. We evaluate SparseLoCo, DiLoCo, and DiLoCo without Nesterov for different communica￾tion intervals and at different sparsity levels for SparseLoCo. We report the best performance in each case. Crucially, SparseLoCo can outperform DiLoCo while communicating significantly less. We also observe that the optimal density grows with higher c… view at source ↗
Figure 2
Figure 2. SparseLoCo lies on the Pareto frontier between loss and communication volume. We report communication volume (outbound) for two settings (A) ring communication topology (ring all-gather for SparseLoCo and DeMo, ring all-reduce for DiLoCo) (B) Parameter server. The points consider different H for DiLoCo, different densities for DeMo, and combinations of both for SparseLoCo using 512M models. We observe that, in both … view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    FoMoE partitions expert layers across workers in MoE LLMs, skips non-resident experts, and reports up to 1.42x lower communication than baselines plus 1.4x throughput gains while maintaining stable routing.

  2. Unifying Local Communications and Local Updates for LLM Pretraining

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    GASLoC generalizes communication acceleration to the outer optimizer to enable gossip-based decentralized LLM pretraining that supports adaptive optimizers, local steps, and outperforms prior decentralized methods on ...

  3. Learned Subspace Compression for Communication-Efficient Pipeline Parallelism

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    MAPL learns task-specific orthogonal compression subspaces per pipeline stage via manifold-constrained optimization and recovers signals with low-overhead anchors, yielding better compression-performance tradeoffs tha...

  4. ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    ROSE is a system for cooperative elasticity that co-locates serving and rollout models on shared GPUs, delivering 1.3-3.3x higher end-to-end throughput than fixed-resource baselines while preserving serving SLOs.

  5. ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    ROSE delivers 1.2-3.3x higher end-to-end throughput for agentic RL by safely co-using underutilized serving GPUs for rollouts while meeting serving SLOs.

  6. ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    ResBM achieves 128x activation compression in pipeline-parallel transformer training by adding a residual bottleneck module that preserves a low-rank identity path, with no major loss in convergence or added overhead.

  7. LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LoRDO combines global low-rank projections with full-rank quasi-hyperbolic momentum to let infrequent-synchronization distributed training match low-rank DDP at roughly 10x less communication.

  8. Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    PULSE exploits BF16-invisible sparsity in weight updates to enable over 100x lower communication in distributed RL post-training via compute-visible sparsification.

Reference graph

Works this paper leans on

32 extracted references · 13 canonical work pages · cited by 7 Pith papers

  1. [4]

    16 Preprint Table 12: Final validation loss for the 178M model while varying the number of workers (R∈ {8,16,32}) and the communication interval (H∈ {15,50,100}).Bestis bold

    We observe that SparseLoCo outperforms DiLoCo with higher number of parallel workers. 16 Preprint Table 12: Final validation loss for the 178M model while varying the number of workers (R∈ {8,16,32}) and the communication interval (H∈ {15,50,100}).Bestis bold. H=15 Method Density Loss R=8 R=32 AdamW 100.00% 2.91 2.91 DiLoCo 100.00% 2.99 3.10 SparseLoCo 0....

    work page

  2. [6]

    Train- ing compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    arXiv

  3. [8]

    Charles- ´Etienne Joseph, Benjamin Th ´erien, Abhinav Moudgil, Boris Knyazev, and Eugene Belilovsky

    URLhttps:// doi.org/10.48550/arXiv.2412.01152. Charles- ´Etienne Joseph, Benjamin Th ´erien, Abhinav Moudgil, Boris Knyazev, and Eugene Belilovsky. Meta-learning optimizers for communication-efficient learning.Trans. Mach. Learn. Res., 2025,

    arXiv doi:10.48550/arxiv.2412.01152 2025

  4. [11]

    Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be

    Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. arXiv preprint arXiv:2304.13960,

    arXiv

  5. [12]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hri- tik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh,...

    work page 2024

  6. [13]

    Shigang Li and Torsten Hoefler

    URLhttp://papers.nips.cc/paper_files/paper/2024/ hash/19e4ea30dded58259665db375885e412-Abstract-Datasets_and_ Benchmarks_Track.html. Shigang Li and Torsten Hoefler. Near-optimal sparse allreduce for distributed deep learning. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Pro- gramming, pp. 135–149,

    work page 2024

  7. [14]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dim- itris S. Papailiopoulos, and Vivienne Sze (eds.),Proceedings of the Third Conference on Machine Learning and Systems, MLSys 2020, Austin, TX, USA, March 2-4,

    work page 2020

  8. [15]

    Incentivizing Permissionless Distributed Learning of LLMs

    URLhttps://proceedings.mlsys.org/paper_files/paper/ 2020/hash/1f5fe83998a09396ebe6477d9475ba0c-Abstract.html. Joel Lidin, Amir Sarfi, Evangelos Pappas, Samuel Dare, Eugene Belilovsky, and Jacob Steeves. Incentivizing permissionless distributed learning of llms.arXiv preprint arXiv:2505.21684,

    work page Pith review arXiv 2020

Show all 32 references
  1. [16]

    Stich, and Martin Jaggi

    Tao Lin, Sebastian U. Stich, and Martin Jaggi. Don’t use large mini-batches, use local SGD.CoRR, abs/1808.07217,

    arXiv

  2. [18]

    Trade- offs of local sgd at scale: An empirical study.arXiv preprint arXiv:2110.08133,

    Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat, Ari Morcos, and Nicolas Ballas. Trade- offs of local sgd at scale: An empirical study.arXiv preprint arXiv:2110.08133,

    arXiv

  3. [20]

    Adaptive federated optimization.arXiv preprint arXiv:2003.00295,

    11 Preprint Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Kone ˇcn`y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization.arXiv preprint arXiv:2003.00295,

    2003 arXiv

  4. [21]

    Fedpaq: A communication-efficient federated learning method with periodic averaging and quan- tization

    Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quan- tization. In Silvia Chiappa and Roberto Calandra (eds.),The 23rd International Conference o...

    2020

  5. [22]

    Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora

    URLhttp://proceedings.mlr.press/v108/reisizadeh20a.html. Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-efficient federated learning with sketching. InProceedings of the 37...

    2020

  6. [23]

    Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu

    URLhttp://proceedings.mlr.press/v119/ rothchild20a.html. Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Haizhou Li, Helen M. Meng, Bin Ma, Engsiong Chng, an...

    2014

  7. [25]

    Local sgd converges fast and communicates little.arXiv preprint arXiv:1805.09767,

    Sebastian U Stich. Local sgd converges fast and communicates little.arXiv preprint arXiv:1805.09767,

    arXiv

  8. [27]

    Benjamin Th´erien, Xiaolong Huang, Irina Rish, and Eugene Belilovsky

    URL http://arxiv.org/abs/1909.05350. Benjamin Th´erien, Xiaolong Huang, Irina Rish, and Eugene Belilovsky. Muloco: Muon is a practi- cal inner optimizer for diloco,

    1909 arXiv

  9. [28]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al

    URLhttps://arxiv.org/abs/2505.23725. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:...

    arXiv

  10. [29]

    Powersgd: Practical low-rank gra- dient compression for distributed optimization

    Thijs V ogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gra- dient compression for distributed optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch ´e-Buc, Emily B. Fox, and Roman Garnett (eds.),Ad- vances in Ne...

    2019

  11. [30]

    Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael G

    URLhttps://proceedings.neurips.cc/paper/2019/ hash/d9fbed9da256e344c1fa46bb46c34c5f-Abstract.html. Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael G. Rabbat. Slowmo: Improving communication-efficient distributed SGD with slow momentum.CoRR, abs/1910.00643,

    2019 arXiv

  12. [31]

    12 Preprint Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal

    doi: 10.1109/ICDCS51616.2021.00060. 12 Preprint Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.arXiv preprint arXiv:2306.01708,

    2021

  13. [32]

    Hanzhen Zhao, Xingyu Xie, Cong Fang, and Zhouchen Lin

    URL https://arxiv.org/abs/2306.01708. Hanzhen Zhao, Xingyu Xie, Cong Fang, and Zhouchen Lin. Separate: A simple low-rank projec- tion for gradient compression in modern large-scale model training process. InThe Thirteenth International Conference on Learning Representations. 1...

    2025 arXiv

  14. [33]

    Here, the TOP-kEF baseline is a simplified DeMo that applies TOP-kselection globally to the entire tensor (rather than within chunks) while still utilizing sign descent. We observe that in the setting with no local steps (TOP-kEF, DeMo) the impact of chunking is very significa...

    2024

  15. [2014]

    2014-274

    URLhttps://doi.org/10.21437/Interspeech. 2014-274. Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. Understanding top-k sparsification in distributed deep learning.CoRR, abs/1911.08772,

    2014 arXiv

  16. [2016]

    Ilyas Fatkhullin, Alexander Tyurin, and Peter Richt´arik

    URLhttps: //openreview.net/pdf?id=OM0jvwB8jIp57ZJjtNEZ. Ilyas Fatkhullin, Alexander Tyurin, and Peter Richt´arik. Momentum provably improves error feed- back! In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.),Advances in Neural I...

    2023

  17. [2017]

    Optimizing the communication-accuracy trade-off in federated learning with rate-distortion theory.arXiv preprint arXiv:2201.02664,

    Nicole Mitchell, Johannes Ball ´e, Zachary Charles, and Jakub Kone ˇcn`y. Optimizing the communication-accuracy trade-off in federated learning with rate-distortion theory.arXiv preprint arXiv:2201.02664,

    arXiv

  18. [2019]

    org/abs/1901.09847

    URLhttps://arxiv. org/abs/1901.09847. 10 Preprint Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning...

    1901 arXiv

  19. [2020]

    Federated optimization: Distributed machine learning for on-device intelligence.arXiv preprint arXiv:1610.02527,

    Jakub Koneˇcn`y, H Brendan McMahan, Daniel Ramage, and Peter Richt´arik. Federated optimization: Distributed machine learning for on-device intelligence.arXiv preprint arXiv:1610.02527,

    arXiv

  20. [2021]

    Decoupled momentum optimization.arXiv preprint arXiv:2411.19870,

    Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Decoupled momentum optimization.arXiv preprint arXiv:2411.19870,

  21. [2022]

    INTELLECT-1 technical report.CoRR, abs/2412.01152,

    Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann. INTELLECT-1 technical report.CoRR, abs/2412.01152,

    arXiv

  22. [2023]

    Louis Fournier, Adel Nabli, Masih Aminbeidokhti, Marco Pedersoli, Eugene Belilovsky, and Edouard Oyallon

    URLhttp://papers.nips.cc/paper_files/paper/2023/ hash/f0b1515be276f6ba82b4f2b25e50bef0-Abstract-Conference.html. Louis Fournier, Adel Nabli, Masih Aminbeidokhti, Marco Pedersoli, Eugene Belilovsky, and Edouard Oyallon. Wash: Train your ensemble with communication-efficient wei...

    2023 arXiv

  23. [2024]

    Diloco: Distributed low- communication training of language models.arXiv preprint arXiv:2311.08105, 2023a

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models.arXiv preprint arXiv:2311.08105, 2023a. Arthur Douillard, Qi...

    doi arXiv

  24. [2025]

    Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi

    URLhttps://arxiv.org/abs/2504.05295. Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-sgd: Distributed sgd with quantization, sparsification, and local computations,

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.