Pith. sign in

REVIEW 3 major objections 5 minor 34 references

This paper claims that LoRDO, a distributed low-rank optimizer with a full-rank momentum correction, matches synchronous low-rank DDP in language modeling (perplexity gap under 1%) while reducing communication by roughly 10x.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:37 UTC pith:CCIRIJXF

load-bearing objection A genuinely interesting stagnation analysis and fix, but the headline communication savings are computed for the wrong variant and never actually measured. the 3 major comments →

arxiv 2602.04396 v2 pith:CCIRIJXF submitted 2026-02-04 cs.LG cs.AI

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

Andrej Jovanovi\'c , Alex Iacob , Mher Safaryan , Ionut-Vlad Modoranu , Lorenzo Sani , William F. Shen , Xinchi Qiu , Dan Alistarh
show 1 more author
Nicholas D. Lane
This is my paper
classification cs.LG cs.AI
keywords low-rank optimizationdistributed traininglocal updatescommunication efficiencyquasi-hyperbolic momentumsubspace projectionlanguage model pretrainingoptimizer state compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that low-rank distributed optimization and infrequent communication can be combined without the performance loss that has blocked the combination. It identifies two failure modes: local per-worker projections are too noisy when each worker sees a small batch, while a globally shared projection computed from the aggregated pseudo-gradient is stable but permanently traps training in a fixed rank-r subspace. The proposed fix is LoRDO, which injects a full-rank quasi-hyperbolic momentum signal into each worker's update so the global projection can keep exploring the full parameter space. If correct, the method gives near-parity with low-rank DDP at 125M-720M scales while cutting communication about 10x, and it can even beat DDP in very low-memory regimes with small rank and batch size.

Core claim

LoRDO's central claim is that a globally shared low-rank projection, computed by SVD from the aggregated pseudo-gradient at synchronization points, is theoretically superior to per-worker local projections because it uses the full multi-worker batch and produces a more stable basis. However, without correction, that same global projection confines all subsequent updates to one fixed rank-r subspace, so the model stagnates. LoRDO restores full subspace exploration by adding a full-rank quasi-hyperbolic momentum term to each worker's local update, making the aggregated pseudo-gradient full rank again. With this single design change, LoRDO-Global reaches near-parity with synchronous low-rank DD

What carries the argument

The key object is the global projection matrix Q_t, a rank-r orthonormal basis obtained via SVD from the aggregated pseudo-gradient after each K-step synchronization window, applied inside a local Adam-style update. The companion piece is the full-rank quasi-hyperbolic momentum (QHM) term, which adds (1-omega)*G_hat / (mu(sqrt(v)+epsilon)) after the low-rank projection rather than inside the low-rank space. The global projection unifies the workers and lowers the variance of the basis estimate (effective batch size M*B), while the full-rank QHM keeps the pseudo-gradient full rank so the next SVD can refresh the subspace. Momentum rotation into the new basis and local error feedback are suppo

Load-bearing premise

The projection-quality argument assumes transformer gradients have a power-law singular-value spectrum and gradient noise that shrinks like the square root of batch size; if either assumption fails, the claimed advantage of global pseudo-gradient projections over local ones is not established.

What would settle it

Train a 125M model with LoRDO-Global and no full-rank QHM, tracking the mean squared singular value of Q_{t+1}^T Q_t at each synchronization; the paper predicts this stays near 1 and perplexity stagnates, so if the basis keeps rotating and loss keeps decreasing, the stagnation mechanism is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, low-rank distributed training can reduce communication by about 10x at 125M-720M scale while keeping perplexity within 1% of low-rank DDP, making pretraining feasible on slower interconnects.
  • At low ranks and small per-worker batch sizes, LoRDO-Global is more stable than local projection methods and can outperform DDP by 3.36-4.7% in perplexity, extending feasible training to very memory-constrained hardware.
  • Lower ranks are more sensitive to longer synchronization intervals; the full-rank QHM term and higher beta_1 mitigate this sensitivity, giving a practical rule for choosing rank and synchronization frequency together.
  • Error feedback and momentum rotation are necessary components: removing either one degrades performance across all tested ranks, so practical implementations must retain both.
  • Per-payload communication for LoRDO-Global drops to O(rq) on the uplink with an additional O(pr) for the projection matrix, so the rank controls a concrete memory-communication trade-off.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The subspace-stagnation mechanism, if general, suggests that the same full-rank injection trick could be applied to other compressed optimizers (e.g., FFT-based or block-wise subspace estimators), not just SVD-based ones.
  • The paper's claim that local projections cause interference under non-IID data is plausible but untested; a natural extension is to compare LoRDO-Global and LoRDO-Local on skewed data shards to see whether the unified basis prevents divergence.
  • Because LoRDO-Global's downlink already includes the O(pr) projection matrix, combining it with quantization or sparsification of the full-rank QHM signal could push communication savings further.
  • LoRDO-Local's aggregation can recover full rank from worker-specific subspaces, suggesting that diversity across workers may itself aid exploration; isolating that factor would be a useful follow-up.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LoRDO, a framework for low-rank adaptive optimization in the local-update distributed setting. It introduces global projection matrices computed from aggregated pseudo-gradients at synchronization boundaries, argues that purely global low-rank projections restrict the optimization trajectory to a rank-r subspace, and adds a full-rank quasi-hyperbolic momentum term to restore subspace exploration. Experiments on 16M–720M parameter transformers trained on SmolLM2 report near-parity with low-rank DDP (perplexity gap <1%) and claim ~10× communication reduction, plus improved performance in low-memory/low-rank regimes. The paper includes extensive ablations, hyperparameter sweeps, and a discussion of limitations.

Significance. The idea of unifying low-rank optimizers with infrequent synchronization is timely and the stagnation pathology is clearly identified and empirically demonstrated. The full-rank QHM fix is simple and well-motivated, and the 720M experiments suggest the method can match a strong low-rank DDP baseline while reducing synchronization frequency. The paper also provides many ablations (error feedback, momenta rotation, sync frequency, rank sensitivity) which strengthen the empirical contribution. However, the headline communication-efficiency claim is not measured and the provided formulas do not apply to the full-rank QHM variant actually used; the parity claim rests on single runs. If these issues are addressed, this would be a solid contribution.

major comments (3)
  1. [§3.2, Appendix B.5.3, Table 2] The communication-benefit formulas (Eqs. 22–25 in B.5.3 and the ratios in §3.2) are derived for the no-QHM/low-rank-QHM branches. The main experimental configuration LoRDO-Global uses full-rank QHM (Appendix G.1), which per Table 2 requires up-link O(pq+2rq) and down-link O(pq+pr+2rq). Section B.5.1 even states that with full-rank QHM the communication cost reverts to dense pseudo-gradients. Consequently the abstract's ≈10× reduction is not supported by the stated formulas. The correct per-sync total for full-rank QHM is 2pq+pr+4rq vs. 2pq per step for low-rank DDP; with K=32 this may still yield a large reduction, but it should be derived explicitly and, given Appendix A's admission that no empirical communication results are provided, should be validated by byte-level measurements or at least a careful calculation for the actual variant. The statement in Contribution 2 that this is ach
  2. [§5.5, Table 1] The parity claim is based on a single run per configuration. The perplexity gap between DDP r=256 (10.34) and LoRDO-Global r=256 (10.41) is 0.07, and downstream accuracy differences are within one point. No error bars, multiple seeds, or statistical significance tests are reported. Since the <1% gap is a headline claim, the authors should either provide repeat runs (at least at 125M/16M scale) to estimate run-to-run variance or explicitly caveat that the difference is within expected noise.
  3. [§3.1, Eq. (1), Appendix B.8] The claimed theoretical superiority of global projections rests on a power-law singular-value model and noise scaling κ/√B, imported from Xie et al. (2023) and McCandlish et al. (2018). These assumptions are not verified on the transformer gradients in this paper. The empirical Figure 4 is consistent with the predicted trend, which mitigates the concern, but the text should distinguish a heuristic derivation from a proven result, and ideally measure the gradient spectrum / noise scaling to support the 'theoretically superior' claim.
minor comments (5)
  1. [§5.5] The text states LoRDO reduces the communication overhead of full-rank DDP by ≈25×, while elsewhere the claim is ≈10× relative to low-rank DDP. Specify the baseline for each number to avoid ambiguity.
  2. [Algorithm 1, line 16] The function µ is used in the full-rank QHM branch but defined only later in §3.2. Define it near the algorithm or add a comment.
  3. [Figure 2] The caption describes both panels as 125M results, but the text in §5.2 and §5.5 refers to them as if they correspond to different settings. Recheck figure/panel references.
  4. [Appendix G.3 and G.4] Typo: 'r ∈ {8,16,32,54,128}' should be 'r ∈ {8,16,32,64,128}' in both captions.
  5. [Table 3] The 'Uplink Time Buffer' column uses '/' for full-rank QHM rows; explain in the caption why the buffer is not needed in that case.

Circularity Check

0 steps flagged

No significant circularity: the central rank-stagnation and full-rank-QHM claims follow from the paper's own definitions and are empirically benchmarked; the unconfirmed ≈10x communication figure is a validation gap, not a circular reduction.

full rationale

The paper's main theoretical claims are derived from its own equations rather than fitted to its results. The claim that LoRDO-Global restricts the pseudo-gradient to an r-rank subspace follows directly from the definition of the global projection update (Section 3.2, Eq. 21 and Appendix B.3): because Δ_t = Q_t · (1/M) Σ η α, the aggregated signal is in the column space of Q_t, so SVD-based projection updates cannot leave that subspace. This is a mathematical consequence, not a fitted parameter renamed as a prediction. The proposed remedy, full-rank quasi-hyperbolic momentum, is also defined explicitly (Algorithm 1, line 16) and is supported by controlled comparisons against no-QHM and low-rank-QHM variants in Figures 2, 5, and G.1, so the central design claim does not reduce to a self-citation. The instability bound in Eq. (1) relies on explicitly stated external assumptions (power-law singular values from Xie et al., 2023; κ/√B noise scaling from McCandlish et al., 2018) and is used conditionally rather than as an empirical fit. The paper does cite its own prior work (MT-DAO, DES-LOC, Iacob et al. 2026a,b) for the communication-frequency framework and for quasi-hyperbolic momentum context, and these citations are load-bearing in the sense that the paper builds on that framework; however, the formulas are re-derived in Appendix B.5 and the QHM mechanism is independently ablated, so this is not a circular dependence. The abstract's ≈10x communication reduction is computed from the infrequent-communication ratio (Section 3.2, Appendix B.5.3) and is explicitly not empirically confirmed (Appendix A: 'we do not provide empirical results confirming these bounds'). Moreover, Table 2 shows that the full-rank-QHM variant, which is the variant used for the parity results, has up-link O(pq+2rq) and down-link O(pq+pr+2rq), so the claimed reduction may be inconsistent with the actual algorithm's communication pattern. This is a substantive correctness/verification concern, but it is not a case of a prediction being equivalent to its input by construction, so it does not raise the circularity score beyond the minor-self-citation level.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical entities. Its claims rest on tuned hyperparameters, imported statistical assumptions about gradient spectra and noise, and a linear algebra stagnation argument.

free parameters (4)
  • rank r = r=8..256 for ablations, r=256 for the 720M run, r=2048 for full-rank baseline
    Low-rank dimension chosen per experiment; memory, communication, and performance conclusions all depend on it.
  • QHM mixing coefficient ω = Best global full-rank QHM ω≈0.97-0.99; best local low-rank QHM ω≈0.90-0.94, per rank
    Tuned by grid sweep on 16M models; controls the strength of the full-rank signal that is claimed to restore subspace exploration.
  • learning rate η = Swept over powers of two, transferred via CompleteP parametrization
    Tuned per method on 16M; parity claims assume optimal transfer to 125M/720M.
  • synchronization period K = K = Kx = Ku = Kv = 32 by default
    Communication interval determines the claimed 10x reduction; lower-rank sensitivity to K is one of the paper's own findings.
axioms (6)
  • domain assumption Gradient singular values follow a power law σ_k = C k^{-α}, α>0
    Used to derive Eq. (1); not verified on the models trained here.
  • domain assumption Gradient noise is additive with norm scaling κ/√B
    Underpins the comparison between local (batch B) and global (batch MB) projection stability.
  • standard math Davis-Kahan sin Θ theorem
    Imported to bound subspace perturbation; standard linear algebra result.
  • standard math For a fixed basis Q, any update Qα lies in range(Q), so SVD of the aggregated pseudo-gradient cannot escape range(Q)
    Linear algebra fact used for the stagnation claim; correct for generic full-row-rank aggregated updates.
  • domain assumption Pseudo-gradients carry curvature information that makes them more informative than local stochastic gradients
    Speculative explanation in §3.1 and E.3; not proven and explicitly left for future work.
  • domain assumption Empirical trends at 16M/125M transfer to 720M
    Stated in Limitations A1; only one 720M configuration (r=256) was tested.

pith-pipeline@v1.3.0-alltime-deepseek · 32842 in / 14577 out tokens · 150223 ms · 2026-08-03T04:37:15.787426+00:00 · methodology

0 comments
read the original abstract

Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose $\texttt{LoRDO}$, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. $\texttt{LoRDO}$ achieves near-parity with low-rank $\texttt{DDP}$ in language modeling and downstream tasks at model scales of $125$M--$720$M, while reducing communication by $\approx 10 \times$. Finally, we show that $\texttt{LoRDO}$ improves performance even more in very low-memory settings with small rank/batch size.

Figures

Figures reproduced from arXiv: 2602.04396 by Alex Iacob, Andrej Jovanovi\'c, Dan Alistarh, Ionut-Vlad Modoranu, Lorenzo Sani, Mher Safaryan, Nicholas D. Lane, William F. Shen, Xinchi Qiu.

Figure 1
Figure 1. Figure 1: Global projection matrix pathologies. LoRDO-Global fails to learn when quasi-hyperbolic momentum terms have not been applied due to the projection bases failing to update throughout the duration of local training. performs its local variant, and more readily matches the per￾formance of the DDP counterpart, where LoRDO-Global and LoRDO-Local recover MT-DAO (Iacob et al., 2026a) when a full-rank representati… view at source ↗
Figure 2
Figure 2. Figure 2: LoRDO with global projections offers superior resilience to small-batch regimes compared to the local projection method. Particularly under heavy memory constraints, which necessitate low ranks, LoRDO surpasses DDP. 5.3. LoRDO-Global Improves Projection Quality (RQ3) In [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation across number of workers and local batch size (M × B) for 16M parameter experiments where the global batch size (or effective batch size) is 64. We present this ablation for both LoRDO-Global and Local and the difference ∆ = P P XLocal − P P XGlobal. As predicted, we find that LoRDO-Local is more sensitive to changes in the local batch size. Global QHM Local QHM Global Non-QHM Local Non-QHM 34.9 3… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation across synchronization frequency for LoRDO variants and QHM terms. Lower ranks are more sensitive to delays in synchronization. In addition to offering more stable perfor￾mance, QHM terms reduce this sensitivity with an increased β1. Low Ranks Are Sensitive to Sync Freq: Lower ranks are more sensitive to infrequent synchronization. Quasi￾hyperbolic momentum mitigates this instability, maintain￾ing… view at source ↗
Figure 6
Figure 6. Figure 6: Impact of error feedback used during local optimization for DDP and the global and local variants of LoRDO across r ∈ {8, 16, 32, 64, 128, 256}, where r = 256 is the full-rank Adam baseline. Across all ranks and model classes, error feedback is essential for optimal performance, corroborating the findings of Seide et al. (2014); Robert et al. (2025). E.3. Ablation: DDP Gradient versus LoRDO-Global In Secti… view at source ↗
Figure 7
Figure 7. Figure 7: Importance of ensuring well-positioned momenta. Both local and global variants of LoRDO fail to learn effectively when optimizer states are not rotated into the new basis. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablation across increasing the synchronization frequency, and thus the frequency of projection matrix updates. We find that the performance of LoRDO-Local remains relatively consistent across this axis, while LoRDO-Global’s performance displays a dependence. This indicates that increasing the number of local steps (effectively increasing the history baked into the pseudo-gradient) is beneficial for determi… view at source ↗
Figure 9
Figure 9. Figure 9: Ablation across sparsity levels for LoRDO-Global and Local, reporting the final perplexity for each, along with the difference ∆ = P P XLocal − P P XGlobal. We observe that while both methods are affected by increasing sparsity, the addition of the full-rank quasi-hyperbolic momentum signal allows for improved performance relative to LoRDO-Local, despite using a potentially damaged projection matrix due to… view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of LoRDO and DDP at the 720M scale. Figure 10a shows that LoRDO is consistent with its DDP counterpart, which is reflected in all downstream tasks. While a gap between low-rank and full-rank optimizers exists, we posit that with a longer training duration, LoRDO would tend toward the performance of the full-rank DDP. F. Extended Related Work Memory-Efficient Optimization and Communication Compr… view at source ↗
Figure 11
Figure 11. Figure 11: Hyperparameter sweep across η ∈ {0.0005, 0.001, 0.002, 0.004, 0.008, 0.016} for 16M models to determine optimal learning rate for warmed up model training starting point. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Hyperparameter sweep for 16M models trained with DDP, where β1 = β2 = 0.999 across r ∈ {8, 16, 32, 54, 128} for combinations of ω ∈ [0.90.99] and different multiples (switch scale) of the learning rate as per (Iacob et al., 2026a). For sweep combination, we show the effect of applying the quasi-hyperbolic formulation in its full-rank or low-rank form. Due to the approximation that the full-rank QHM method… view at source ↗
Figure 15
Figure 15. Figure 15: Hyperparameter sweep for 16M models trained with LoRDO, where β1 = β2 = 0.999 across r ∈ {8, 16, 32, 54, 128} for combinations of ω ∈ [0.90.99] and different multiples (switch scale) of the learning rate as per (Iacob et al., 2026a). For sweep combination, we show the effect of applying the quasi-hyperbolic formulation in its full-rank or low-rank form. As in [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 15 linked inside Pith

  1. [1]

    Dion: Dis- tributed orthonormalized updates.arXiv preprint arXiv: 2504.05295,

    Ahn, K., Xu, B., Abreu, N., and Langford, J. Dion: Dis- tributed orthonormalized updates.arXiv preprint arXiv: 2504.05295,

    arXiv

  2. [2]

    (Qm t ⊤Qm t−1)2(ˆ¯vm t−1 −( ˆ¯um t−1)2) + (Qm t ⊤Qm t−1 ˆ¯um t−1)2 Rotate the second moment locally 16:else 17:Q m t =Q m t−1 Maintain stale projection 18:ˆg m t ←Q m⊤ t ( ˆGm t +E m t−1)Low-rank gradient signal with error-feedback 19:E t m ← ˆGm t +E m t−1 −Q m t ˆgm t Compute error feedback 20:u m t ←β 1 ¯ut−1 + (1−β 1)ˆgm t 21:v m t ←β 2¯vt−1 + (1−β 2)...

    2024

  3. [3]

    For full-rank methods, this structure is not possible; as such, it achieves a lower memory overhead traded for an increase in the communication payload size

    Specifically, in order to realize the up-link cost savings in Section B.5.1, for the non- and low-rank quasihyperbolic momentum variants, each worker m incurs an additional O(rq) cost to store the accumulated buffer across time between parameter synchronization periods. For full-rank methods, this structure is not possible; as such, it achieves a lower me...

    2025

  4. [4]

    We leave a thorough investigation on the interplay between projecting gradients and pseudo-gradients to future work. E.4. Ablation: Impact on Sparse Communication Although methods implementing communication compression at synchronization intervals are orthogonal to our work, we consider the effect of sparsification to additionally improve the communicatio...

    2025

  5. [6]

    A., Chhaparia, R., Donchev, Y ., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J

    Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y ., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J. Diloco: Distributed low-communication training 9 LoRDO: Distributed Low-Rank Optimization with Infrequent Communication of language models.arXiv preprint arXiv:2311.08105,

    Pith/arXiv arXiv

  6. [8]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozi `ere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., M...

    Pith/arXiv arXiv

  7. [11]

    Karimireddy, S

    URL https: //kellerjordan.github.io/posts/muon/. Karimireddy, S. P., Jaggi, M., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03606,

    Pith/arXiv arXiv 2008

  8. [12]

    Kim, J., Lee, B., Park, C., Oh, Y ., Kim, B., Yoo, T., Shin, S., Han, D., Shin, J., and Yoo, K. M. Peri-ln: Revisiting layer normalization in the transformer architecture.arXiv preprint arXiv: 2502.02732,

    Pith/arXiv arXiv

  9. [14]

    Fft-based dynamic subspace selection for low-rank adaptive optimization of large language models.arXiv preprint arXiv:2505.17967,

    10 LoRDO: Distributed Low-Rank Optimization with Infrequent Communication Modoranu, I.-V ., Safaryan, M., Schultheis, E., Ryabinin, M., Chumachenko, A., and Alistarh, D. Fft-based dynamic subspace selection for low-rank adaptive optimization of large language models.arXiv preprint arXiv:2505.17967,

    arXiv

  10. [15]

    Peng, B., Quesnelle, J., and Kingma, D. P. Demo: Decoupled momentum optimization.arXiv preprint arXiv:2411.19870,

    arXiv

  11. [16]

    Dilocox: A low- communication large-scale training framework for de- centralized cluster.arXiv preprint arXiv:2506.21263,

    Qi, J., Zhu, W., Li, L., Wu, M., Wu, Y ., He, W., Gao, X., Zeng, J., and Heinrich, M. Dilocox: A low- communication large-scale training framework for de- centralized cluster.arXiv preprint arXiv:2506.21263,

    Pith/arXiv arXiv

  12. [17]

    Com- munication efficient LLM pre-training with sparseloco

    Sarfi, A., Th ´erien, B., Lidin, J., and Belilovsky, E. Com- munication efficient LLM pre-training with sparseloco. arXiv preprint arXiv:2508.15706,

    Pith/arXiv arXiv

  13. [19]

    Muloco: Muon is a practical inner optimizer for diloco.arXiv preprint arXiv:2505.23725,

    Th´erien, B., Huang, X., Rish, I., and Belilovsky, E. Muloco: Muon is a practical inner optimizer for diloco.arXiv preprint arXiv:2505.23725,

    Pith/arXiv arXiv

  14. [20]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V ., Kha...

    Pith/arXiv arXiv

  15. [24]

    with α= 1.0 . This configuration enables one-shot hyperparameter transfer from small to large models, allowing us to conduct extensive sweeps on the16M variant and reserve compute-intensive scaling experiments for baseline comparisons. Batch sizes and training durations follow contemporary best practices (Zhang et al., 2025). For the 125M and 720M models,...

    2025

  16. [26]

    In Figure 6, we conduct an ablation on using error feedback across ranks and method types for our 16M model scales

    during the local optimization procedures for each worker to account for information lost during low-rank compression. In Figure 6, we conduct an ablation on using error feedback across ranks and method types for our 16M model scales. Across all settings, we find error feedback is essential to ensuring good performance, consistent with Seide et al. (2014);...

    2014

  17. [28]

    2000 3000 4000 5000 6000 Steps 30 40 50 60 70 80 90 100Perplexity r = 32 DDP w/o EF (avg 43.7 @ step

    2000

  18. [29]

    2000 3000 4000 5000 6000 Steps 30 40 50 60 70 80Perplexity r = 64 DDP w/o EF (avg 40.0 @ step

    2000

  19. [30]

    2000 3000 4000 5000 6000 Steps 30 35 40 45 50 55 60 65 70Perplexity r = 128 DDP w/o EF (avg 37.2 @ step

    2000

  20. [31]

    2000 3000 4000 5000 6000 Steps 30 35 40 45 50 55 60 65Perplexity r = 256 DDP (avg 35.7 @ step

    2000

  21. [32]

    Across all ranks and model classes, error feedback is essential for optimal performance, corroborating the findings of Seide et al

    Figure 6.Impact of error feedback used during local optimization for DDP and theglobalandlocalvariants of LoRDO across r∈ {8,16,32,64,128,256} , where r= 256 is the full-rank Adam baseline. Across all ranks and model classes, error feedback is essential for optimal performance, corroborating the findings of Seide et al. (2014); Robert et al. (2025). E.3. ...

    2014

  22. [34]

    also falls into this category by enabling rank-r orthogonalization; however, it typically relies on QR decomposition, which requires execution at every step, incurring significant computational costs that scale with rank (Modoranu et al., 2025). While we primarily discuss SVD-based projections, our framework’s core contribution—handling subspace misalignm...

    2025

  23. [64]

    All models utilize the Warmup-Stable-Decay (WSD) scheduler (H¨agele et al., 2024)

    Training steps T are set as multiples of the compute-optimal token budget (Hoffmann et al., 2022): the 16M model is trained for ≈2× the compute-optimal duration, while the 125M and 720M models are trained for ≈2× and ≈1× their respective budgets. All models utilize the Warmup-Stable-Decay (WSD) scheduler (H¨agele et al., 2024). Warmup lengths follow recom...

    2022

  24. [1970]

    DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., ...

    Pith/arXiv arXiv

  25. [1985]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,

    2022

  26. [2014]

    Benchmarking optimizers for large language model pretraining.arXiv preprint arXiv: 2509.01440,

    Semenov, A., Pagliardini, M., and Jaggi, M. Benchmarking optimizers for large language model pretraining.arXiv preprint arXiv: 2509.01440,

    Pith/arXiv arXiv

  27. [2017]

    B., Lozhkov, A., Bakouch, E., Bl ´azquez, G

    Allal, L. B., Lozhkov, A., Bakouch, E., Bl ´azquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydl ´ıcek, H., Lajar´ın, A. P., Srivastav, V ., Lochner, J., Fahlgren, C., Nguyen, X., Fourrier, C., Burtenshaw, B., Larcher, H., Zhao, H., Zakka, C., Morlon, M., Raffel, C., and Wolf, T. Smollm2: When smol goes big - data-centric training of a small langu...

    Pith/arXiv arXiv

  28. [2019]

    McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training.arXiv preprint arXiv:1812.06162,

    Pith/arXiv arXiv

  29. [2020]

    Communication- efficient language model training scales reliably and robustly: Scaling laws for diloco.arXiv preprint arXiv:2503.09799,

    Charles, Z., Teston, G., Dery, L., Rush, K., Fallen, N., Gar- rett, Z., Szlam, A., and Douillard, A. Communication- efficient language model training scales reliably and robustly: Scaling laws for diloco.arXiv preprint arXiv:2503.09799,

    Pith/arXiv arXiv

  30. [2021]

    At first, we consider the setting where we set the number of local stepsKto one

    setting). At first, we consider the setting where we set the number of local stepsKto one. This gives the following local update computation when using a low-rank adaptive optimizer: θm t+1 ←θ t −η m t Qm t αm t (2) where θt is the model received by the worker at the previous synchronization boundary. Computing the per-worker pseudo-gradients: ∆m t ←θ t −...

    2024

  31. [2023]

    Streaming diloco with overlapping communica- tion: Towards a distributed free lunch.arXiv preprint arXiv:2501.18512,

    Douillard, A., Donchev, Y ., Rush, K., Kale, S., Charles, Z., Garrett, Z., Teston, G., Lacey, D., McIlroy, R., Shen, J., Ram ´e, A., Szlam, A., Ranzato, M., and Barham, P. Streaming diloco with overlapping communica- tion: Towards a distributed free lunch.arXiv preprint arXiv:2501.18512,

    Pith/arXiv arXiv

  32. [2024]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models.arXiv pre...

    Pith/arXiv arXiv

  33. [2025]

    C., Noci, L., Li, M

    Dey, N., Zhang, B. C., Noci, L., Li, M. B., Bordelon, B., Bergsma, S., Pehlevan, C., Hanin, B., and Hestness, J. Don’t be lazy: Completep enables compute-efficient deep transformers.arXiv preprint arXiv: 2505.01618,

    arXiv

  34. [6144]

    2000 3000 4000 5000 6000 Steps 30 40 50 60 70 80 90 100 110Perplexity r = 16 DDP w/o EF (avg 47.2 @ step

    2000