Pith. sign in

REVIEW 2 major objections 4 minor 62 references

DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distributed training can synchronize model parameters and optimizer momenta on independent schedules, cutting communication by 2× versus Local Adam and 170× versus DDP while preserving convergence on models up to 1.7B parameters.

desk verdict DES-LOC's empirical result—about 2x communication savings over Local Adam at equal quality—is credible and useful, but its flagship convergence proof covers a randomized proxy, not the deterministic algorithm in the paper. read the letter →

arxiv 2505.22549 v1 pith:NQKCPAZT submitted 2025-05-28 cs.LG

Alex Iacob , Lorenzo Sani , Mher Safaryan , Paris Giampouras , Samuel Horváth , Andrej Jovanovic , Meghdad Kurmanji , Preslav Aleksandrov
show 3 more authors
William F. Shen Xinchi Qiu Nicholas D. Lane
This is my paper
classification cs.LG MSC 68W1590C2668T05
keywords DES-LOCdistributedtrainingcommunication-efficientoptimizationadaptiveoptimizersLocalAdammomentumsynchronizationfederatedlearningconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that distributed training of foundation models can synchronize model parameters and the optimizer's momentum states independently, on separate clocks, without losing convergence. It introduces DES-LOC, a family of optimizers that assigns each state — parameters, first momentum, second momentum — its own synchronization period, and proves convergence for the resulting algorithm. On language models up to 1.7B parameters, DES-LOC communicates about 2× less than the previous state-of-the-art Local Adam and 170× less than standard DDP at comparable quality. If the claims hold, large-scale training can roughly halve its bandwidth demand without switching optimizers or sacrificing convergence, which matters because communication is the main scaling bottleneck in distributed training.

What carries the argument

The carrying object is the half-life of an optimizer state, $\tau_{0.5}(\beta) = \ln(0.5)/\ln(\beta)$ — the number of local steps until an exponential-moving-average state's weight decays to half — paired with the drift bound $\|s_{t+K} - s_t\|_\infty \le 2\rho(1-\beta^K)$ for gradients clipped to radius $\rho$. These two quantities turn a heuristic (slow states need rare sync) into a scheduling rule: sync each state at a period comparable to its half-life. The proof machinery that carries the convergence argument is the factor $\psi = \frac{4(1-p_x)}{p_x^2}\cdot\frac{(1-\beta)(1-p_u)}{1-(1-p_u)\beta}$ in Theorem 1, which separates the squared-inverse dependence on the parameter sync probability $p_x$ from the mild dependence on the momentum sync probability $p_u$, showing that model averaging dominates the error while momentum averaging only relaxes the step-size restriction.

What would settle it

Run the same 135M-model training under the deterministic schedule of Algorithm 1 and under Bernoulli-random synchronization with matched probabilities $p_x = 1/K_x$, $p_u = 1/K_u$ and compare convergence in perplexity or average squared gradient norm: material divergence would show Theorem 1 does not cover the implemented algorithm. A complementary check sets $K_u = K_v = \infty$, so momentum is never synchronized; the theory predicts the asymptotic rate survives while heuristic persistent-state baselines fail, so if DES-LOC collapses at scale in this regime the finite-horizon claim would not hold.

Watch

Extended reading notes

Core claim

The central discovery is a synchronization hierarchy rooted in how fast each optimizer state changes. With gradient clipping, the drift of a momentum state over $K$ local steps is bounded by $2\rho(1-\beta^K)$, so a state with large decay $\beta$ moves little over many steps; the second momentum of Adam or ADOPT at $\beta_2 = 0.9999$ evolves roughly 100× slower than the first momentum. DES-LOC exploits this by synchronizing parameters every $K_x$ steps, the first momentum every $K_u$, and the second momentum every $K_v$, with $K_u = 3K_x$ and $K_v = 6K_x$ as a robust default that halves communication relative to Local Adam. The paper proves for SGD with momentum that the convergence rate's leading term is asymptotically optimal and untouched by these frequencies: the momentum sync probability appears only in the higher-order term, while the parameter sync probability enters through a factor $\psi = O(1/p_x^2)$, so insufficient parameter sync breaks the rate but momentum sync can even be turned off. Experiments confirm the hierarchy, showing that perplexity collapses when $K_x$ grows large while $K_u$ and $K_v$ can be varied widely with little effect.

Load-bearing premise

The convergence proof replaces the deterministic every-$K$-steps synchronization of Algorithm 1 with independent random synchronization events of the same average frequency and asserts the two are statistically equivalent; if the periodic schedule drifts differently from the randomized one, Theorem 1 does not prove convergence of the algorithm as implemented, and the benefit of skipping momentum sync rests on the empirical half-life heuristic.

Editorial extensions

If this is right

  • A practitioner can keep the parameter sync period $K$ from Local Adam and set momentum periods to $K_u = 3K$, $K_v = 6K$ (when $\beta_1 \ll \beta_2$), halving communication at matched perplexity on models up to 1.7B parameters.
  • Because every state eventually synchronizes, DES-LOC keeps Local Adam's ability to absorb newly added workers and survive system failures, an advantage the persistent-local-state heuristics lack.
  • The SGDM theory says momentum averaging can be disabled entirely ($p_u = 0$) without changing the asymptotic $O(1/\sqrt{T})$ rate, only the step size that theory permits.
  • The measured communication savings persist when workers hold heterogeneous data, consistent with the paper's heterogeneous-loss analysis for DES-LOC-SGDM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is to set each state's sync period from its measured drift rather than from fixed multipliers, which would make per-layer or per-tensor synchronization schedules the next step.
  • The paper relaxes only the momentum side of the communication budget; layering drift-correction or gradient compression onto the parameter channel could plausibly push total traffic below the demonstrated 2× reduction.
  • Theorem 1 is proved for Bernoulli-random synchronization while Algorithm 1 uses the deterministic periodic schedule; an empirical check of the two at matched frequencies would test whether the guarantee covers the implemented algorithm.
  • The regime with momentum sync fully disabled ($p_u = 0$) is proved but not stress-tested at scale; a large-model run with $K_u = K_v = \infty$ would probe whether the asymptotic argument survives practical horizons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes DES-LOC, a family of distributed adaptive optimizers that assign independent synchronization periods to model parameters, first momentum, and second momentum. The central claims are that momentum states can be synchronized less often than parameters without losing convergence, that this yields a 2x communication reduction over Local Adam and a 170x reduction over DDP, and that the method remains robust to worker addition and system failures. The authors provide a non-convex convergence theorem for DES-LOC-SGDM, an informal Adam convergence result, and extensive experiments on 135M and 1.7B parameter language models using ADOPT and Adam, including non-IID data, ICL evaluation, and wall-clock modeling.

Significance. If the convergence claim were established for the actual algorithm, this would be a practically valuable contribution: it would reduce communication payloads for distributed adaptive optimizers by roughly half relative to Local Adam at comparable quality, with a simple design principle tied to optimizer-state half-lives. The empirical core is credible: the training setup, hyperparameter sweeps, synchronization-frequency ablations, heterogeneous-data experiments, and billion-scale evaluation are described in enough detail to be re-implemented, and the reported 2x/170x communication reductions are consistent with the stated periods. The SGDM derivation is internally consistent under its probabilistic model, and I found no circularity; the gap is in transferring that model to the deterministic algorithm actually run. The paper's headline 'provable convergence' claim is therefore not currently supported, which makes the significance conditional on a repairable but load-bearing theoretical fix.

major comments (2)
  1. [Sec. 3 / Alg. 1 vs Alg. 4 / Lemma 4 (App. E.2)] The advertised convergence guarantee does not bound the algorithm that is run. Theorem 1 is proved for Algorithm 4, where the whole candidate update is averaged at each step with independent Bernoulli probabilities p_x = 1/K_x and p_u = 1/K_u; the paragraph before Theorem 1 asserts this is 'statistically equivalent' to the periodic schedule of Algorithm 1. That equivalence fails at the pathwise level: in Algorithm 1 synchronization occurs before the update (lines 13-18), so after a sync step workers still differ through their current local gradients, whereas in Algorithm 4 a sync event makes the state identical across workers. It also fails at the second-moment level used in Lemma 4: the recurrence E||x_{t+1} - x^m_{t+1}||^2 = (1 - p_x) E||x_t - x^m_t - eta(...)||^2 holds only because a Bernoulli reset makes the deviation zero with probability p_x; under a deterministic t mod K_x schedule the deviation is reset only at multiples of K_x and follows a non-memoryless recurrence. Thus Theorem 1's bound (Eq. 9) is not established for Algorithm 1, and the central 'provable convergence' claim needs either a direct deterministic-drift analysis or a restatement that the theorem covers only the randomized variant.
  2. [Sec. 3, Eq. (10) / Appendix F] The Adam result is an lcm-reduction and does not support the communication-efficiency claim. Theorem 2 (and Theorem 6 in Appendix F) analyzes DES-LOC-Adam by reducing it to a uniform schedule with period K_lcm = lcm(K_x, K_u, K_v), so all states are synchronized together every K_lcm steps. The proof does not use the more frequent parameter synchronization of the actual schedule K_x = K, K_u = 3K_x, K_v = 6K_x, and the rate has K_lcm in the denominator. This establishes convergence for a less-communicating uniform schedule, not the empirical DES-LOC schedule, and therefore cannot justify the advertised 2x communication reduction over Local Adam at equal parameter-sync frequency. The caveat in Appendix F that this reduction is 'weak and not tight' should be reflected in the main-text claims.
minor comments (4)
  1. [Appendix E, Algorithm 4] The Ensure line lists a final variable v_{T-1} that is never defined in the SGDM algorithm; this appears to be a copy-paste artifact and should be removed.
  2. [Fig. 6 / Appendix C.4 / Table 2] Several captions refer to a '1B-model' or '1B-update norms', while the experimental section and Table 2 describe a 1.7B model; the captions should be made consistent with the stated model size.
  3. [Appendix E.2, Eq. (12)] In the proof of Lemma 4, the displayed factor ((1 - p)(1 + s))^{t - tau} uses an undefined p; it should be p_x to match the definition of q_1.
  4. [Section 2, Eqs. (5)-(6)] The main text should state explicitly that the maximal drift bounds apply to the clipped gradients and coordinate-wise clipped momentum, not to the raw stochastic gradient, since the derivation in Appendix G relies on the infinity-norm bound supplied by clipping.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence derivation is self-contained under standard assumptions, the communication-ratio claims are arithmetic properties of the chosen periods, and the quality claims are validated against external baselines.

full rationale

The paper's central convergence result (Theorem 1, Section 3) is derived from standard smoothness, bounded-variance, and bounded-heterogeneity assumptions using a descent lemma and drift bounds (Lemmas 3-5 in Appendix E). None of these lemmas assumes the theorem being proved, and the bound is not obtained by fitting a parameter to the convergence target. The step-size restriction and the final O(1/sqrt(T)) rate are consequences of the analysis, not inputs renamed as outputs. The empirical claim of 2x lower communication than Local Adam is a deterministic arithmetic consequence of the chosen synchronization periods (Kx=K, Ku=3Kx, Kv=6Kx), while the substantive experimental claim is that perplexity and downstream ICL quality are maintained at those periods; that quality claim is benchmarked externally against DDP, Local Adam, and federated baselines. Self-citations such as [33] and [34] appear only as background and baseline descriptions, and the proof does not rely on them. The paper's assertion that Bernoulli synchronization with probability px=1/Kx is 'statistically equivalent' to the deterministic t mod Kx schedule is a possible rigor gap rather than a circular step, because the proof does not define the deterministic algorithm's convergence as its conclusion; under the review rules, a validity concern without an equation-level reduction to the inputs is not scored as circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its design depends on empirically chosen sync ratios and on the unproved probabilistic-equivalence axiom in the SGDM proof; the Adam guarantee reuses prior assumptions and a worst-case lcm drift bound.

free parameters (2)
  • Synchronization periods (Kx, Ku, Kv) = Kx=256, Ku=768, Kv=1536 for the 1.7B experiments; Ku=3Kx, Kv=6Kx in most ablations
    Chosen empirically to achieve a 2x communication reduction while matching Local Adam perplexity; the values are not derived from the convergence theorem.
  • Learning rate eta* = e.g., 0.0021 for ADOPT with beta1=0.95, beta2=0.9999
    Grid-searched under DDP and Local Adam in Appendix B.2 and then applied to DES-LOC; all reported curves depend on these tuned rates.
assumptions (4)
  • standard math Assumptions 1-3: L-smoothness, unbiased bounded-variance stochastic gradients, and bounded heterogeneity
    Standard distributed non-convex optimization assumptions used in the Theorem 1 proof (Section 3).
  • ad hoc to paper Probabilistic synchronization with probability px=1/Kx is statistically equivalent to deterministic periodic synchronization every Kx steps
    Stated without proof in Section 3 before Theorem 1; the convergence proof analyzes the randomized schedule in Algorithm 4, not the deterministic t mod Kx schedule in Algorithm 1.
  • domain assumption Bounded gradients and homogeneous data distributions for the Adam analysis (Assumptions 4, 7, 8)
    Needed to import the Cheng-Glasgow Local Adam high-probability analysis through the lcm drift argument in Appendix F.
  • domain assumption Coordinate-wise clipped gradients satisfy |g_i| <= rho
    Used to derive the momentum drift bounds in Eqs. (5) and (6), which motivate the half-life heuristic in Section 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models." pith.science (2026). https://pith.science/paper/NQKCPAZT

@misc{pith2026250522549,
  author       = {Pith},
  title        = {Pith review of: DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQKCPAZT}},
  note         = {Machine review of arXiv:2505.22549}
}
read the original abstract

Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Current approaches extending Local SGD either lack convergence guarantees or require synchronizing all optimizer states, tripling communication costs. We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Through extensive experiments on language models of up to 1.7B, we show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local ADAM. Furthermore, unlike previous heuristic approaches, DES-LOC is suited for practical training scenarios prone to system failures. DES-LOC offers a scalable, bandwidth-efficient, and fault-tolerant solution for foundation model training.

Figures

Figures reproduced from arXiv: 2505.22549 by the authors.

Figure 2
Figure 2. We present a toy problem where DES-LOC (Kx = 192, Ku = 192, Kv = 692) and Local Adam (K = Kx) both converge to the optimum (overlapping in Fig. 2a). Methods keeping optimizer states local [12, 34] fail, causing oscillations without convergence. Periodically resetting states [33, 16] similarly stalls due to repeated oscillations. We optimize the non-convex function f(x1, x2) = (1 − x1) 2 + 100(x2 − x 2 1 ) 2 with M =… view at source ↗
Figure 3
Figure 3. Relative rates of change for first and second momenta across rounds using standard [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Model perplexity for DES-LOC (ADOPT, β1 = 0.95, β2 = 0.9999), varying synchronization periods independently (others fixed at Kb). Parameter synchronization (a) is critical, with sharp degradation at higher periods. Second-momentum synchronization (b) minimally affects performance due to its large half-life (τ0.5(β2) ≫ Kb). First-momentum synchronization significantly improves perplexity (c) only when the baseline ma… view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Setting Kx = K, Ku = 3Kx, and Kv = 6Kx, DES-LOC achieves a 2× communication reduction over Local Adam, matching performance at high (a) and low (b) frequencies for Local Adam and heuristic baselines (see Section 4.1). We demonstrate robustness to the addition of new wo…
Figure 6
Figure 6. Figure 6: shows that DES-LOC reliably scales to billion-scale models and extensive training workloads. Evaluating the billion-scale models on the ICL tasks ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: We present a toy problem in a Non-IID setting, where DES-LOC (with synchronization periods Kx = 192, Ku = 192, Kv = 692) and Local Adam (with K = Kx) converge to a superior solution compared to methods that keep optimizer states local [12, 34] or periodically reset the…
Figure 8
Figure 8. Figure 8: examines the effects of independently varying synchronization periods (Kx, Kv) for pa￾rameters and second momentum under DES-LOC-ADOPT in the high-frequency regime (Kb = 16), chosen based on the first momentum’s half-life (τ0.5 ≈ 13.5). Similar to the low-frequency res…
Figure 9
Figure 9. Figure 9: Model perplexity for DES-LOC-Adam (β1 = β2 = 0.95) when independently varying sync periods (Kx, Ku, Kv) while fixing others at baseline Kb. Parameter synchronization (a,b) influences performance in both high (Kb = 16) and low (Kb = 256) frequency regimes. Momenta synch…
Figure 10
Figure 10. Figure 10: Comparison of perplexity under Non-IID conditions for DES-LOC, Local Adam (Kx = Ku = Kv), and heuristic baselines (defined in Section 4.1) at high (a) and low (b) synchronization frequencies. Due to higher cross-worker variance caused by heterogeneous data, parameters…
Figure 11
Figure 11. Figure 11: Configurations of DES-LOC targeting 2× lower communication than Local Adam (Kx = Ku = Kv), setting Ku, Kv as multiples of Kx. In both high (a) and low-frequency (b) regimes, performance depends on how communication is split between momenta for β1 ≪ β2. Syncing the fir…
Figure 12
Figure 12. Figure 12: Configurations of DES-LOC targeting 2× lower communication than Local Adam (Kx = Ku = Kv), using Adam (β1 = β2 = 0.95). In contrast to DES-LOC-ADOPT (where β1 ≪ β2 yields an advantage for Ku < Kv as shown in [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Setting Kx = K, Ku = 3Kx, and Kv = 6Kx, DES-LOC-Adam achieves a 2× commu￾nication reduction over Local Adam, matching performance at high (a) and low (b) frequencies for Local Adam and heuristic baselines (see Section 4.1). Takeaway: DES-LOC-Adam achieves a similar 2×…
Figure 14
Figure 14. Figure 14: Comparison of update (a) and parameter norms (b) for billion-scale models trained [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Estimated wall-clock time for training the [PITH_FULL_IMAGE:figures/full_fig_p042_15.png]
Figure 16
Figure 16. Figure 16: Communication overhead (a) and GPU utilization (b) for training the [PITH_FULL_IMAGE:figures/full_fig_p042_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 47 canonical work pages

  1. [1]

    L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafi- oti, H. Kydlícek, A. P. Lajarín, V . Srivastav, J. Lochner, C. Fahlgren, X. Nguyen, C. Four- rier, B. Burtenshaw, H. Larcher, H. Zhao, C. Zakka, M. Morlon, C. Raffel, and T. Wolf. Smollm2: When smol goes big - data-centric training of a small language model. arXiv prep...

    arXiv 2025

  2. [2]

    Arjevani, Y

    Y . Arjevani, Y . Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1-2):165–214, 2023

    work page 2023

  3. [3]

    Balles and P

    L. Balles and P. Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning (ICML), 2018

    work page 2018

  4. [4]

    Ben Allal, A

    L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra. Cosmopedia, February 2024

    work page 2024

  5. [5]

    Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi. PIQA: reasoning about physical common- sense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Inte...

    work page 2020

  6. [6]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei...

    work page 2020

  7. [7]

    Charles, G

    Z. Charles, G. Teston, L. Dery, K. Rush, N. Fallen, Z. Garrett, A. Szlam, and A. Douillard. Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco. arXiv preprint arXiv:2503.09799, 2025

    arXiv 2025

  8. [8]

    X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C. Hsieh, Y . Lu, and Q. V . Le. Symbolic discovery of optimization algorithms. InConference on Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

Show all 62 references
  1. [9]

    Cheng and M

    Z. Cheng and M. Glasgow. Convergence of distributed adaptive optimization with local updates. In International Conference on Learning Representations (ICLR), 2025

    2025

  2. [10]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y . Tay, N. Shazeer, V . Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. ...

    2023

  3. [11]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018

    2018 arXiv

  4. [12]

    Douillard, Q

    A. Douillard, Q. Feng, A. A. Rusu, R. Chhaparia, Y . Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023

    2023 arXiv

  5. [13]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozière, B. Biron, B. Ta...

    2024 arXiv

  6. [14]

    Hägele, E

    A. Hägele, E. Bakouch, A. Kosson, L. B. Allal, L. von Werra, and M. Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  7. [15]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifr...

    2022 arXiv

  8. [16]

    Iacob, L

    A. Iacob, L. Sani, M. Kurmanji, W. F. Shen, X. Qiu, D. Cai, Y . Gao, and N. D. Lane. DEPT: Decoupled embeddings for pre-training language models. In International Conference on Learning Representations (ICLR), 2025

    2025

  9. [17]

    Kairouz, H

    P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. A. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D’Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. H...

    2021

  10. [18]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020

    2001 arXiv

  11. [19]

    S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning (ICML), 2020

    2020

  12. [20]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015

    2015

  13. [21]

    Kunstner, J

    F. Kunstner, J. Chen, J. W. Lavington, and M. Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In International Conference on Learning Representations (ICLR), 2023. 11

    2023

  14. [22]

    S. Li, Y . Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow., 2020

    2020

  15. [23]

    X. Li, B. Karimi, and P. Li. On distributed adaptive optimization with gradient compression. arXiv preprint arXiv:2205.05632, 2022

    2022 arXiv

  16. [24]

    B. Liu, R. Chhaparia, A. Douillard, S. Kale, A. A. Rusu, J. Shen, A. Szlam, and M. Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.09135, 2024

    2024 arXiv

  17. [25]

    Y . Liu, Y . Gao, and W. Yin. An improved analysis of stochastic gradient descent with momentum. arXiv preprint arXiv:2007.07989, 2020

    2007 arXiv

  18. [26]

    McMahan, E

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2017

    2017

  19. [27]

    Pagliardini, P

    M. Pagliardini, P. Ablin, and D. Grangier. The adEMAMix optimizer: Better, faster, older. In International Conference on Learning Representations (ICLR), 2025

    2025

  20. [28]

    Pascanu, T

    R. Pascanu, T. Mikolov, and Y . Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning (ICML), 2013

    2013

  21. [29]

    Penedo, H

    G. Penedo, H. Kydlícek, L. B. Allal, A. Lozhkov, M. Mitchell, C. A. Raffel, L. von Werra, and T. Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  22. [30]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020

    2020

  23. [31]

    S. J. Reddi, S. Kale, and S. Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations (ICLR), 2018

    2018

  24. [32]

    Romero, J

    J. Romero, J. Yin, N. Laanait, B. Xie, M. T. Young, S. Treichler, V . Starchenko, A. Y . Borisevich, A. Sergeev, and M. A. Matheson. Accelerating collective communication in data parallel training across deep learning frameworks. In NSDI, pages 1027–1040. USENIX Association, 2022

    2022

  25. [33]

    L. Sani, A. Iacob, Z. Cao, B. Marino, Y . Gao, T. Paulik, W. Zhao, W. F. Shen, P. Aleksandrov, X. Qiu, and N. D. Lane. The future of large language model pre-training is federated. arXiv preprint arXiv:2405.10853, 2024

    2024 arXiv

  26. [34]

    L. Sani, A. Iacob, R. L. Zeyu Cao, B. Marino, Y . Gao, W. Zhao, D. Cai, Z. Li, X. Qiu, and N. D. Lane. Photon: Federated llm pre-training. In Eighth Conference on Machine Learning and Systems, 2025

    2025

  27. [35]

    Sardana, J

    N. Sardana, J. P. Portes, S. Doubov, and J. Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In International Conference on Machine Learning (ICML), 2024

    2024

  28. [36]

    T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V . del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan- Maj...

    2022 arXiv

  29. [37]

    Sergeev and M

    A. Sergeev and M. D. Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018. 12

    2018 arXiv

  30. [38]

    Shoeybi, M

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Train- ing multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019

    1909 arXiv

  31. [39]

    S. L. Smith, P. Kindermans, C. Ying, and Q. V . Le. Don’t decay the learning rate, increase the batch size. In International Conference on Learning Representations (ICLR), 2018

    2018

  32. [40]

    S. U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations (ICLR), 2019

    2019

  33. [41]

    J. Su, M. H. M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  34. [42]

    Sutskever, J

    I. Sutskever, J. Martens, G. E. Dahl, and G. E. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning (ICML), 2013

    2013

  35. [43]

    Taniguchi, K

    S. Taniguchi, K. Harada, G. Minegishi, Y . Oshima, S. C. Jeong, G. Nagahara, T. Iiyama, M. Suzuki, Y . Iwasawa, and Y . Matsuo. ADOPT: modified adam can converge with anyβ2 with the optimal rate. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  36. [44]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023

  37. [45]

    J. Wang, Z. Charles, Z. Xu, G. Joshi, H. B. McMahan, B. A. y Arcas, M. Al-Shedivat, G. Andrew, S. Avestimehr, K. Daly, D. Data, S. Diggavi, H. Eichner, A. Gadhikar, Z. Garrett, A. M. Girgis, F. Hanzely, A. Hard, C. He, S. Horvath, Z. Huo, A. Ingerman, M. Jaggi, T. Javidi, P. K...

    2021 arXiv

  38. [46]

    Wortsman, T

    M. Wortsman, T. Dettmers, L. Zettlemoyer, A. Morcos, A. Farhadi, and L. Schmidt. Stable and low-precision training for large-scale vision-language models. In NeurIPS, 2023

    2023

  39. [47]

    Y . You, J. Li, S. J. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C. Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations (ICLR), 2020

    2020

  40. [48]

    H. Yu, R. Jin, and S. Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. arXiv preprint arXiv:1905.03817, 2019

    1905 arXiv

  41. [49]

    K. Yuan, X. Huang, Y . Chen, X. Zhang, Y . Zhang, and P. Pan. Revisiting optimal conver- gence rate for smooth and non-convex stochastic decentralized optimization. arXiv preprint arXiv:2210.07863, 2022

    2022 arXiv

  42. [50]

    Zellers, A

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi. Hellaswag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Ita...

    2019

  43. [51]

    Zhang, D

    H. Zhang, D. Morwani, N. Vyas, J. Wu, D. Zou, U. Ghai, D. Foster, and S. M. Kakade. How does critical batch size scale in pre-training? In The Thirteenth International Conference on Learning Representations, 2025. 13

    2025

  44. [52]

    Zhang, C

    Y . Zhang, C. Chen, N. Shi, R. Sun, and Z. Luo. Adam can converge without any modification on update rules. In Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  45. [53]

    Y . Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y . Hao, A. Math- ews, and S. Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endo...

    2023

  46. [54]

    Starting from the recommended baseline learning rate (η0) from Allal et al

    Stage 1: Tuning η for DDP. Starting from the recommended baseline learning rate (η0) from Allal et al. [1], we conduct a grid search as outlined by Charles et al. [7]: {. . . , √ 2 −2 η0, √ 2 −1 η0, η0, √ 2η0, √ 2 2 η0, . . .} We expand this search until perplexity stops impro...

  47. [55]

    We then repeat this procedure for Local Adam , using η∗ DDP as the new baseline

    Stage 2: Tuning η for Local Adam . We then repeat this procedure for Local Adam , using η∗ DDP as the new baseline. To balance generalizability and computational cost, we set the synchronization period to an intermediate value of K = 64, between high-frequency (K = 16) and low...

  48. [56]

    2a by including results on the heterogeneous data dis- tribution described in Section 4.1

    Appendix C.1 complements Fig. 2a by including results on the heterogeneous data dis- tribution described in Section 4.1. This highlights DES-LOC’s robustness under imperfect sampling or strongly Non-IID federated scenarios [see 17, Sec 3.1]

  49. [57]

    4 by showing the separate impact of varying synchro- nization frequencies for parameters and the second momentum when the base frequency is Kb = 16

    Appendix C.2.1 complements Fig. 4 by showing the separate impact of varying synchro- nization frequencies for parameters and the second momentum when the base frequency is Kb = 16. It supports our claim that parameters and second momentum exhibit similar behavior across differ...

  50. [58]

    4 by evaluatingDES-LOC-Adam

    Appendix C.2.2extends Fig. 4 by evaluatingDES-LOC-Adam. We confirm that the parameter synchronization frequency is the most important, as predicted by our theory. In contrast, the momenta sync frequency is far less impactful, especially for low parameter sync frequencies

  51. [59]

    5 by showing DES-LOC-ADOPT’s perplexity against baseline methods on heterogeneous data (as defined in Section 4.1)

    Appendix C.3.1 complements Fig. 5 by showing DES-LOC-ADOPT’s perplexity against baseline methods on heterogeneous data (as defined in Section 4.1). This validates our claim from Contribution 2 regarding DES-LOC’s effectiveness on heterogeneous datasets

  52. [60]

    Appendix C.3.2 presents an ablation study examining alternative low-communication configurations of DES-LOC, justifying our choice of Ku = 3Kx, Kv = 6Kx used in Fig. 5

  53. [61]

    5 for DES-LOC-Adam, demon- strating that DES-LOC achieves similar communication reductions and performance when using Adam instead of ADOPT

    Appendix C.3.3 repeats the baseline comparison from Fig. 5 for DES-LOC-Adam, demon- strating that DES-LOC achieves similar communication reductions and performance when using Adam instead of ADOPT

  54. [62]

    standard

    Appendix C.4 provides additional metrics illustrating training instabilities for theFAVG+OPT baseline, including rapidly growing parameter norms, supporting observations in Fig. 6.b. C.1 Toy Problem on Non-IID Data (See Fig. 2a) Toy Example Non-IID: Fig. 7 simulates the scenar...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.