Pith. sign in

REVIEW 3 major objections 4 minor 44 references

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EDiT claims a sharded Local SGD recipe that beats full-sync LLM training on loss with a 19ms sync delay.

desk verdict Worth reading for the systems design; don't rely on the convergence proof — the theorem's bound is unsupported by its own equations. read the letter →

arxiv 2412.07210 v2 pith:HYVBSZPY submitted 2024-12-10 cs.DC cs.AI

Jialiang Cheng , Ning Gao , Yun Yue , Zhiling Ye , Jiadi Jiang , Jian Sha This is my paper · ORCID
classification cs.DCcs.AI MSC 68T0768W15
keywords EDiTLocalSGDdistributedtrainingmodelshardinglargelanguagemodelsasynchronouspseudogradientpenaltyelastic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EDiT's aim is to make Local SGD practical for pre-training large language models, where earlier local-update methods fail because they keep full model copies in memory, add non-overlapped communication, and destabilize on noisy web-scale data. The paper combines Local SGD with parameter sharding on a two-dimensional device mesh: workers synchronize sharded parameters layer by layer during the forward pass, overlapping sync with computation, and keep all extra optimizer state sharded and CPU-offloadable. At each synchronization point a pseudo-gradient penalty rejects outlier workers by an EMA z-test, reweights the rest by their pseudo-gradient norms, and clips the aggregated update, which the authors argue suppresses loss spikes and even turns worker diversity into better updates. In Llama experiments up to 7B parameters, EDiT posts the lowest training loss on two datasets and the lowest validation perplexity on FineWeb-Edu, beating the fully synchronous baseline, with only 19ms of added sync delay, and the asynchronous A-EDiT variant keeps throughput high under a consistent straggler. If these results hold, EDiT is a drop-in replacement for synchronous LLM pre-training that needs far less communication, tolerates stragglers, and supports elastic resizing without learning-rate retuning.

What carries the argument

The load-bearing mechanism is the pseudo-gradient penalty applied inside the Sync() routine at each synchronization step. Every worker computes the pseudo-gradient $\Delta^{(i,l)}_t = \theta^{(i,l)}_{t,\tau} - \theta^{(i,l)}_t$; its norm $G^{(i,l)}_t$ is screened by an EMA z-score (Eq. 1) with fixed threshold $\delta = 3$ and decay $\alpha = 0.02$; flagged workers are eliminated; the survivors are combined with weights $w_{t,i} = \exp(-G^{(i,l)}_t) / \sum_j \exp(-G^{(j,l)}_t)$ (Eq. 2), clipped by $\beta_t = \min(\phi/(\bar{G}_t + \epsilon), 1)$ (Eq. 4), and fed to the outer Nesterov optimizer, with a rollback to the last synchronized parameters if every worker is flagged. Around this sits the engineering core: the $M \times N$ device mesh with model-shard groups handling all-gather/reduce-scatter for each layer and model-sync groups exchanging only their shard of the parameters during the forward pass, which is what converts synchronization from a batch-wide stall into an overlapped, memory-cheap operation. The convergence theorem then analyzes this Sync operation as a clipped, weighted average of inner SGD updates.

What would settle it

Log the per-worker pseudo-gradient norms over a real pre-training run and compare how often the EMA z-test flags workers against the outlier rate a Gaussian tail at $\delta=3$ implies: if healthy workers are flagged more than a fraction of a percent of the time, or genuine anomalies are missed, the anomaly elimination is miscalibrated and the stability gains do not follow. A separate check targets the proof: record the sequence of clip coefficients $\beta_t$ across steps; the appendix's Equation 12 treats $\beta_t$ as deterministic, so if measured $\beta_t$ shows non-negligible variance under identical conditioning, the stated $O(\log T/\sqrt{T})$ rate is not established by the argument given.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two things that kept Local SGD out of LLM training—memory blowup from full-model copies and synchronization overhead—can both be removed by sharding parameters along one mesh dimension and synchronizing periodically along the other, so that each worker holds exactly its shard at all times. Synchronization then happens per layer at the start of the forward pass with a prefetch of the next layer, making the periodic sync nearly free (19ms versus 160ms for Post Local SGD) and letting the method scale to 7B parameters where Post Local SGD, DiLoCo, and CO2 run out of memory. The second claim is that training stability on noisy corpora can be bought explicitly: the pseudo-gradient penalty filters each worker's accumulated parameter change at sync time, eliminating statistical outliers, down-weighting large-norm workers, and clipping the combined update, so that loss spikes are suppressed and EDiT recovers faster than DiLoCo, whose workers spike together. The paper reports EDiT reaching the lowest training loss on both the FineWeb-Edu and in-house datasets and the lowest validation perplexity on FineWeb-Edu (17.38, below the Baseline's 17.42), with Theorem 1 giving a convergence rate of $O(\log T/\sqrt{T})$ to a stationary point.

Load-bearing premise

The load-bearing premise is that the EMA z-test with fixed threshold $\delta=3$ and decay $\alpha=0.02$ reliably separates anomalous workers from healthy ones using only pseudo-gradient norms; the paper assumes this separation works on real, noisy, non-stationary gradient distributions and offers no sensitivity analysis of the thresholds.

Editorial extensions

If this is right

  • EDiT extends the reach of Local SGD to model scales that defeat its predecessors: Post Local SGD, DiLoCo, and CO2 all run out of memory at 3B or below, while EDiT trains 3B and 7B Llama models at higher throughput than the synchronous baseline.
  • Because the optimal learning rate stays fixed as workers are added (Fig. 6), EDiT supports elastic training: scaling workers from 1 to 16 or back does not require re-tuning, and in simulated 1-2-4-8 and 8-4-2-1 schedules EDiT beats Baseline PPL by 4.5% and 2.6%.
  • The pseudo-gradient penalty makes Local SGD robust to low-quality corpus shards: on the noisy in-house dataset, where DiLoCo's performance visibly declines, EDiT stays within noise of the Baseline on loss and beats it on average benchmark score.
  • A-EDiT, which synchronizes on wall-clock time rather than step count, keeps TFLOPS near the straggler-free level when one node is consistently slow, while the step-synchronous EDiT and Baseline degrade sharply.
  • Theorem 1 places EDiT's convergence at $O(\log T/\sqrt{T})$ under $L$-smoothness, bounded gradients, and unbiased noise, matching the standard Local SGD rate despite the additional penalty, weighting, and clipping.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the EMA z-test is well calibrated, the per-layer anomaly flags and weights it produces could serve as a free training-time diagnostic for data quality: workers repeatedly flagged as anomalous are likely consuming low-quality shards, which suggests using EDiT's penalty statistics to drive data curation or resharding.
  • The fixed thresholds ($\delta=3$, $\alpha=0.02$) are the fragile constants of the method; on corpora with heavy-tailed or non-stationary gradient norms—say, during a learning-rate change or a domain shift—the z-test will misclassify healthy workers, so an adaptive or distribution-free anomaly rule is the natural next experiment.
  • Because A-EDiT switches the sync trigger from steps to wall-clock time, lagging workers accumulate systematically different numbers of local updates; an untested consequence is that their effective learning rates diverge, and the anomaly filter may end up masking exactly the stale updates that time-based sync creates.
  • The elasticity result hints at a cheaper hyperparameter search protocol for sharded Local SGD generally: tune on a single worker, then scale out without changing the learning rate, a recipe the paper demonstrates only for EDiT and that would be worth testing with other outer optimizers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes EDiT, a Local-SGD-based distributed training method for large language models that combines model sharding with layer-wise parameter synchronization during the forward pass, a pseudo-gradient penalty strategy (anomaly elimination, weighted averaging, and gradient clipping), and a fully asynchronous variant A-EDiT. The authors report experiments on Llama models from 350M to 7B parameters on FineWeb-Edu and an in-house dataset, comparing against Baseline, Post Local SGD, DiLoCo, and CO2/CO2*. They claim lower training loss and validation perplexity than the baselines, higher throughput, better robustness to stragglers and bandwidth constraints, and favorable elastic-training behavior. Section 5 states a convergence theorem with rate O(log T / sqrt(T)) for EDiT, proved in Appendix A.4.

Significance. If the empirical results are reproducible and the convergence claim is valid, EDiT is a practically relevant contribution: it addresses a real gap in making Local SGD compatible with model sharding for LLM training, releases code, evaluates at scales up to 7B, and includes a useful ablation of its stability mechanisms. The main theoretical claim, however, is currently unsupported because of a clear error in the proof, and the empirical superiority over the Baseline rests on very small differences without variance estimates. The significance is therefore conditional: the engineering contribution is credible, but the paper's stronger claims need correction and additional statistical support.

major comments (3)
  1. [Appendix A.4, Eqs. (4) and (17)-(18)] The proof of Theorem 1 contains a reversed inequality that invalidates the argument. Equation (4) defines beta_t = min(phi / (Gbar_t + epsilon), 1), so by construction 0 < beta_t <= 1, yet the proof states 'Since from Equation 4, we have 1 <= beta_t <= phi/epsilon'. The lower bound is false, and beta_t may be arbitrarily close to zero when the synchronized pseudo-gradient norm is large. The step from Eq. (17) to Eq. (18) drops beta_t from the left-hand side under the sum, which requires beta_t >= 1, not beta_t <= 1. Consequently the displayed O(log T / sqrt(T)) bound does not follow from the preceding inequalities. The theorem must either be reproved with a valid lower bound on beta_t or replaced by a weaker statement that accounts for the clipping factor.
  2. [Appendix A.4, Eq. (12)] The equality E[beta_t sum_j w_{t,j} g_{t,p}^{(j)}] = beta_t E[g_{t,p}] is not justified. The coefficient beta_t and the weights w_{t,j} depend on pseudo-gradients accumulated over the last tau local steps, which include g_{t,p} itself; they are not constants or independent of the current gradient. The expectation therefore does not factor as written. This is a second load-bearing gap in the proof of Theorem 1, and it must be fixed before the convergence claim can be accepted.
  3. [Section 4.2, Figure 4] The claim that EDiT 'even surpasses the Baseline' on FineWeb-Edu rests on differences of 0.004 in training loss (2.848 vs. 2.852) and 0.04 in validation PPL (17.38 vs. 17.42). These values are reported as averages of the last 10 points, with no standard deviations, no multiple seeds, and no significance test. At LLM pretraining scale, such differences are within typical run-to-run noise. The authors should either provide repeated-run statistics or temper the claim to 'comparable to Baseline' for these metrics.
minor comments (4)
  1. [Section 3.2] The anomaly elimination mechanism relies on fixed hyperparameters delta=3 and alpha=0.02, and the gradient clip threshold is set to phi=10, but no sensitivity analysis is provided. The paper should report how performance varies with these values, since the stability gains claimed for the pseudo-gradient penalty depend on them.
  2. [Section 5, Theorem 1] The notation in Theorem 1 and Appendix A.4 suppresses the dependence of beta_t, w_{t,j}, and the pseudo-gradient norms on the module index l and on the worker index i. This obscures the fact that the clipping coefficient is computed per module per synchronization event. The theorem should make these dependencies explicit.
  3. [Section 4.3, Table 2] Throughput and TFLOPS are reported as averages over 100 steps without variance or repeated measurements. A brief statement about measurement noise would strengthen the acceleration claims, especially for the small differences between EDiT and CO2 on the 350M model.
  4. [Section 4.4, Figure 6] The elastic-training conclusion is based on single-curve comparisons of validation PPL for Baseline and EDiT. Because the final gaps are modest and no repeated experiments are reported, the claim that EDiT 'validates' the learning-rate invariance hypothesis should be phrased more cautiously.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical results are benchmarked externally and the convergence proof, although flawed, does not reduce to its own inputs.

full rationale

The paper's central empirical claims are direct measurements against external baselines (Baseline, Post Local SGD, DiLoCo, CO2) on independent datasets, not values recovered from parameters fitted to the same targets. The pseudo-gradient penalty hyperparameters (delta=3, alpha=0.02, phi=10) are hand-set and ablated component-wise, so there is no fitted-input-called-prediction pattern. The convergence theorem in Appendix A.4 is derived from stated assumptions with an explicit proof, and it relies on the framework of Wang et al. (2019), an external work with no author overlap with the present paper; no load-bearing self-citation appears. The proof does contain a serious mathematical error: Eq. (4) defines beta_t <= 1, while the proof asserts 'Since from Equation 4, we have 1 <= beta_t <= phi/epsilon', which is false. However, that is an invalid inference, not a circular reduction: the claimed O(log T / sqrt(T)) rate is not the theorem's input restated as its output. Therefore no step of the derivation chain is equivalent by construction to its own inputs, and the appropriate circularity finding is none.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims depend on a handful of hand-set thresholds (delta, alpha, phi) and the synchronization interval, plus unproven assumptions that pseudo-gradient norms identify anomalous workers and that the Wang et al. convergence framework applies. No new physical or mathematical entities are introduced.

free parameters (5)
  • synchronization interval tau = 128 (600s for A-EDiT)
    Chosen by hyperparameter search over 16, 64, 128, 256, 512; controls the communication frequency and quality tradeoff. Used in all main experiments and in Theorem 1.
  • anomaly z-score threshold delta = 3
    Hand-selected; flags a worker as anomalous when its pseudo-gradient norm exceeds the EMA mean by 3 EMA standard deviations. Central to the anomaly elimination component.
  • EMA coefficient alpha = 0.02
    Hand-selected; controls how quickly the anomaly detector adapts to changing gradient norms.
  • pseudo-gradient clip threshold phi = 10
    Hand-selected; caps the synchronized pseudo-gradient norm before the outer optimizer update. Appears in Theorem 1 and in the proof.
  • EMA warm-up period and warmup steps t_warm = not reported
    Algorithm 1 switches from standard mini-batch SGD to local SGD after t_warm steps, and the EMA anomaly detector has a preliminary period with no flagging. The exact step counts are never stated, so these are unstated free implementation choices that affect the results.
assumptions (4)
  • standard math The loss is L-smooth and lower bounded; gradients are bounded, unbiased, and independent across steps (Theorem 1 assumptions 1-3).
    Imported from the Wang et al. (2019) framework; unverifiable for LLM losses but standard for convergence proofs.
  • domain assumption Pseudo-gradient norms are reliable indicators of update quality, so z-test filtering and exp(-norm) weighting improve the update direction.
    Section 3.2 assumes anomalies manifest as large pseudo-gradient norms; the paper provides no theoretical justification, only empirical ablation.
  • domain assumption Intra-node communication is faster than inter-node, so organizing the device mesh with shard groups inside nodes and sync groups across nodes is beneficial.
    Section 3.1 relies on this cluster topology assumption to justify the M x N device mesh.
  • domain assumption The outer optimizer with pseudo-gradient clipping preserves convergence under the Wang et al. framework.
    Theorem 1 extends the framework to EDiT, but the proof is flawed because beta_t is treated as deterministic, so this axiom is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models." pith.science (2026). https://pith.science/paper/HYVBSZPY

@misc{pith2026241207210,
  author       = {Pith},
  title        = {Pith review of: EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HYVBSZPY}},
  note         = {Machine review of arXiv:2412.07210}
}
read the original abstract

Distributed training methods are crucial for large language models (LLMs). However, existing distributed training methods often suffer from communication bottlenecks, stragglers, and limited elasticity, particularly in heterogeneous or large-scale environments. Local SGD methods have been proposed to address these issues, but their effectiveness remains limited to small-scale training due to additional memory overhead and lack of concerns on efficiency and stability. To tackle these issues, we propose EDiT, an innovative Efficient Distributed Training method that combines a tailored Local SGD approach with model sharding techniques to enhance large-scale training efficiency. EDiT performs layer-wise parameter synchronization during forward pass, reducing communication and memory overhead and enabling overlap. Besides, EDiT employs a pseudo gradient penalty strategy to suppress loss spikes, which ensures training stability and improves performance. Additionally, we introduce A-EDiT, a fully asynchronous variant of EDiT that accommodates heterogeneous clusters. Building on EDiT/A-EDiT, we conduct a series of experiments to validate large-scale asynchronous training for LLMs, accompanied by comprehensive analyses. Experimental results demonstrate the superior performance of EDiT/A-EDiT, establishing them as robust solutions for distributed LLM training in diverse computational ecosystems. The code is available at Atorch codebase: https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/local_sgd.

Figures

Figures reproduced from arXiv: 2412.07210 by the authors.

Figure 1
Figure 1. The schematic illustration of our proposed EDiT method with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of model synchronization and our proposed pseudo gradient penalty method, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A comparison of the synchronization scheme of EDiT and A-EDiT. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The loss and PPL curves of different methods on the (a) & (b) FineWeb-Edu dataset and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The TFLOPS of different methods under different training scenarios. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: (a) & (b) The PPL results against learning rates ( [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: (a) The validation PPL curves of different versions of EDiT with the final PPL values [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: The training loss and validation PPL curves for the 350M, 1B, 3B, and 7B models trained [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The profiling results of Baseline, Post Local SGD, CO2, and EDiT during synchronization [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The training loss curves of the Baseline and EDiT methods under different numbers of [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 20 canonical work pages

  1. [1]

    A survey on data selection for language models

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024. https://arxiv.org/abs/2402.16827

    arXiv 2024

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    On the choice of learning rate for local sgd

    Lukas Balles, Cedric Archambeau, et al. On the choice of learning rate for local sgd. Transactions on Machine Learning Research, 2023

    work page 2023

  4. [4]

    Multi-level local sgd: Distributed sgd for heterogeneous hierarchical networks

    Timothy Castiglia, Anirban Das, and Stacy Patterson. Multi-level local sgd: Distributed sgd for heterogeneous hierarchical networks. In International Conference on Learning Representations, 2020

    work page 2020

  5. [5]

    Accelerating gossip sgd with periodic global averaging

    Yiming Chen, Kun Yuan, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Accelerating gossip sgd with periodic global averaging. In International Conference on Machine Learning, pp.\ 1791--1802. PMLR, 2021

    work page 2021

  6. [6]

    Large scale distributed deep networks

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  7. [7]

    Local sgd optimizes overparameterized neural networks in polynomial time

    Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Local sgd optimizes overparameterized neural networks in polynomial time. In International Conference on Artificial Intelligence and Statistics, pp.\ 6840--6861. PMLR, 2022

    work page 2022

  8. [8]

    Diloco: Distributed low-communication training of language models

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023

    arXiv 2023

Show all 44 references
  1. [9]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/lighteval

    2023

  2. [10]

    Why (and when) does local sgd generalize better than sgd? In The Eleventh International Conference on Learning Representations, 2022

    Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local sgd generalize better than sgd? In The Eleventh International Conference on Learning Representations, 2022

    2022

  3. [11]

    Tighter theory for local sgd on identical and heterogeneous data

    Ahmed Khaled, Konstantin Mishchenko, and Peter Richt \'a rik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pp.\ 4519--4529. PMLR, 2020

    2020

  4. [12]

    Lyra: Elastic scheduling for deep learning clusters

    Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, and Cong Wang. Lyra: Elastic scheduling for deep learning clusters. In Proceedings of the Eighteenth European Conference on Computer Systems, pp.\ 835--850, 2023

    2023

  5. [13]

    Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent

    Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017

    2017

  6. [14]

    Asynchronous decentralized parallel stochastic gradient descent

    Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, pp.\ 3043--3052. PMLR, 2018

    2018

  7. [15]

    Don't use large mini-batches, use local sgd

    Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. In International Conference on Learning Representations, 2019

    2019

  8. [16]

    Asynchronous local-sgd training for language modeling

    Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A Rusu, Jiajun Shen, Arthur Szlam, and Marc'Aurelio Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.09135, 2024

    2024 arXiv

  9. [17]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  10. [18]

    Fineweb-edu, May 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu, May 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    2024

  11. [19]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedin...

    2021

  12. [20]

    Y. E. Nesterov. A method for solving the convex programming problem with convergence rate O ( 1 k^2 ) . Proceedings of the USSR Academy of Sciences, 269: 0 543--547, 1 1983. URL https://ci.nii.ac.jp/naid/10029946121/

    1983

  13. [21]

    Federated learning with buffered asynchronous aggregation

    John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics, pp.\ 3581--3607. PMLR, 2022

    2022

  14. [22]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors . Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  15. [23]

    Local sgd accelerates convergence by exploiting second order information of the loss function

    Linxuan Pan and Shenghui Song. Local sgd accelerates convergence by exploiting second order information of the loss function. arXiv preprint arXiv:2305.15013, 2023

    2023 arXiv

  16. [24]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  17. [25]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  18. [26]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp.\ 400--407, 1951

    1951

  19. [27]

    Stl-sgd: Speeding up local sgd with stagewise communication period

    Shuheng Shen, Yifei Cheng, Jingchang Liu, and Linli Xu. Stl-sgd: Speeding up local sgd with stagewise communication period. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 9576--9584, 2021

    2021

  20. [28]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  21. [29]

    Local sgd with a communication overhead depending only on the number of workers

    Artin Spiridonoff, Alex Olshevsky, and Ioannis Ch Paschalidis. Local sgd with a communication overhead depending only on the number of workers. arXiv preprint arXiv:2006.02582, 2020

    2006 arXiv

  22. [30]

    Efficient distributed training with full communication-computation overlap

    Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Efficient distributed training with full communication-computation overlap. In The Twelfth International Conference on Learning Representations, 2023

    2023

  23. [31]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp.\ 1139--1147. PMLR, 2013

    2013

  24. [32]

    Self-influence guided data reweighting for language model pre-training

    Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. arXiv preprint arXiv:2311.00913, 2023

    2023 arXiv

  25. [33]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  26. [34]

    Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd

    Jianyu Wang and Gauri Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd. Proceedings of Machine Learning and Systems, 1: 0 212--229, 2019

    2019

  27. [35]

    Slowmo: Improving communication-efficient distributed sgd with slow momentum

    Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-efficient distributed sgd with slow momentum. In International Conference on Learning Representations, 2019

    2019

  28. [36]

    Asynchronous federated optimization

    Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934, 2019

    1903 arXiv

  29. [37]

    Tuning large neural networks via zero-shot hyperparameter transfer

    Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34: 0 17084--17097, 2021

    2021

  30. [38]

    Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning

    Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 5693--5700, 2019

    2019

  31. [39]

    Parallel sgd: When does averaging help? arXiv preprint arXiv:1606.07365, 2016

    Jian Zhang, Christopher De Sa, Ioannis Mitliagkas, and Christopher R \'e . Parallel sgd: When does averaging help? arXiv preprint arXiv:1606.07365, 2016

    2016 arXiv

  32. [40]

    Timelyfl: Heterogeneity-aware asynchronous federated learning with adaptive partial training

    Tuo Zhang, Lei Gao, Sunwoo Lee, Mi Zhang, and Salman Avestimehr. Timelyfl: Heterogeneity-aware asynchronous federated learning with adaptive partial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5064--5073, 2023

    2023

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  34. [42]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  35. [43]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  36. [44]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.