A single global merge at the final step of decentralized SGD matches the convergence rate of parallel SGD while improving test accuracy under high data heterogeneity.
Noloco: No-all-reduce low communication training method for large models.arXiv preprint arXiv:2506.10911, 2025
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
verdicts
UNVERDICTED 4representative citing papers
GASLoC generalizes communication acceleration to the outer optimizer to enable gossip-based decentralized LLM pretraining that supports adaptive optimizers, local steps, and outperforms prior decentralized methods on standard tasks while matching DiLoCo in multi-step regimes.
Decoupled DiLoCo enables asynchronous distributed pre-training with zero global downtime under simulated failures while preserving competitive performance on text and vision tasks.
HeLoCo corrects misaligned pseudo-gradients in asynchronous low-communication training via outer momentum reference, yielding up to 7.5% better loss at fixed tokens and 22.1% over synchronous under severe heterogeneity.
citing papers explorer
-
On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning
A single global merge at the final step of decentralized SGD matches the convergence rate of parallel SGD while improving test accuracy under high data heterogeneity.
-
Unifying Local Communications and Local Updates for LLM Pretraining
GASLoC generalizes communication acceleration to the outer optimizer to enable gossip-based decentralized LLM pretraining that supports adaptive optimizers, local steps, and outperforms prior decentralized methods on standard tasks while matching DiLoCo in multi-step regimes.
-
Decoupled DiLoCo for Resilient Distributed Pre-training
Decoupled DiLoCo enables asynchronous distributed pre-training with zero global downtime under simulated failures while preserving competitive performance on text and vision tasks.
-
HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity
HeLoCo corrects misaligned pseudo-gradients in asynchronous low-communication training via outer momentum reference, yielding up to 7.5% better loss at fixed tokens and 22.1% over synchronous under severe heterogeneity.