Pith. sign in

REVIEW 3 major objections 6 minor 41 references

PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PLUMAGE is a fixed-rank, unbiased, minimum-variance gradient estimator for LLM training; it samples singular components without replacement and realigns Adam moments across projection updates.

desk verdict Useful empirical low-rank gradient method; the reuse-unbiasedness claim in §3.3.2 is not established and should be corrected, but the paper deserves a serious referee. read the letter →

arxiv 2505.18313 v1 pith:25SB56GM submitted 2025-05-23 cs.LG stat.ML

Matan Haroush , Daniel Soudry This is my paper · ORCID
classification cs.LGstat.ML
keywords low-rankgradientestimationminimum-varianceunbiasedestimatorLLMpre-trainingoptimizerstatealignmentAdammomentssamplingwithoutreplacementmemory-efficienttrainingcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that low-rank gradient training need not sacrifice unbiasedness or pay high variance: it constructs a k-sparse estimator that samples singular components without replacement with inclusion probabilities chosen to minimize variance, then proves the one-sided projection version preserves those properties. If correct, PLUMAGE is a drop-in replacement for GaLore-style low-rank optimizers that closes about a third of the gap between low-rank and full-rank pre-training loss at the same memory footprint, and about 28% of the GLUE training-loss gap. The paper also claims to fix a subtle failure mode: when the projection is periodically updated, Adam's first- and second-moment statistics live in different subspaces, and it provides a realignment rule that prevents spurious weight updates.

What carries the argument

The load-bearing object is the probability vector $p$ and the without-replacement wheel-of-fortune sampler: top $r^*$ singular modes are always kept, and the remaining $k-r^*$ modes are drawn with probability proportional to their singular values, so that the estimator's expected value equals $G$ and its expected squared error is minimized under the k-sparse constraint. The one-sided projection $P D^{-1} P^\top G$ makes the estimator usable with Adam by storing a single $m\times k$ projection per weight, and the realignment matrix $B=P_2^\top P_1$ with $V_2 \approx B^{\circ 2} V_1$ is the mechanism that transfers first- and second-moment statistics between consecutive projection subspaces.

What would settle it

Measure the per-step bias $E[P_t D_t^{-1} P_t^\top G_t] - G_t$ when $P_t$ is sampled at step $t_0 < t$ and the gradient's top subspace rotates; train a small model on data with a known drifting subspace and compare PLUMAGE's update direction with the full-batch gradient. If the angular error grows with subspace drift, then the claimed unbiasedness does not hold in the amortized training loop.

Watch

Extended reading notes

Core claim

On the paper's own terms, PLUMAGE's central claim is that a fixed-rank estimator of the form $\hat{G}=\sum_{i=1}^{n} \frac{1}{p_i} I_i \sigma_i u_i v_i^\top$, with inclusion probabilities $p_i=1$ for the top $r^*$ modes and $p_i=(k-r^*)\sigma_i / \sum_{j>r^*} \sigma_j$ for the remaining modes, and with exactly $k$ indices sampled without replacement, is unbiased and has minimum variance among k-sparse estimators, and that the one-sided estimator $\hat{G}=P D^{-1} P^\top G$ inherits these properties because it reduces to the same form. It further claims that realigning Adam's moments via $M_2 \approx P_2^\top P_1 M_1$ and $V_2 \approx (P_2^\top P_1)^{\circ 2} V_1$ removes the spurious updates caused by changing projections, and reports that this shrinks the full-rank optimization gap by 33% on average in pre-training and 28% on GLUE while keeping GaLore's computational and memory footprint.

Load-bearing premise

The projection sampled from an old gradient remains a fair sample of the current gradient because the dominant gradient subspace is stable over the reuse interval.

Editorial extensions

If this is right

  • At the same memory and compute footprint as GaLore, PLUMAGE lowers pre-training validation perplexity from 30.18, 24.08, and 17.03 to 28.73, 21.81, and 16.29 for the 130M, 350M, and 1B Llama variants.
  • On GLUE fine-tuning, PLUMAGE's mean terminal accuracy is 85.66 versus GaLore's 84.89 and full-rank Adam's 85.91, with lower mean loss than GaLore.
  • PLUMAGE introduces no hyperparameters beyond the rank $r$ and update interval $\tau$, and it reuses the full-rank learning rate without retuning.
  • Realigning first- and second-moment statistics across projection updates removes spurious weight updates and makes training stable across seeds.
  • The one-sided projection halves the communication cost of two-sided ATOMO-style estimators in data-parallel settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the unbiasedness proof covers a freshly sampled projection; the amortized claim that staleness does not matter relies on subspace stability, so the method's advantage should be tested under explicit subspace drift.
  • Inference: the same sampling scheme could be applied right-sided to activations, trading forward-pass projection cost for activation-memory savings; the paper mentions this direction but does not test it.
  • Inference: the principal-angle adaptive interval controller depends on manually chosen thresholds, so the reported gains may be sensitive to those settings.
  • Inference: the second-moment realignment approximates the off-diagonal entries of the gradient's second-moment matrix as zero; when gradients are strongly correlated across coordinates, this approximation could degrade alignment quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes PLUMAGE, a low-rank gradient estimator for training large models. In §3.1 the authors derive inclusion probabilities for a k-sparse estimator of the form (7), solve the variance-minimization problem (13)-(18), and in §3.2 show that the one-sided projection form (20) is equivalent to the two-sided form. Section 3.3 integrates the estimator with Adam: Eqs. (22)-(24) define low-rank moments, Eqs. (25)-(27) introduce projection reuse over τ steps, and Eqs. (28)-(29) realign first and second moments when the projection changes. Experiments compare PLUMAGE with Adam, GaLORE, and FLORA on LLaMA pretraining (C4) and RoBERTa fine-tuning (GLUE), reporting smaller gaps to full-rank Adam.

Significance. The derivation of the inclusion probabilities is mostly sound, the wheel-of-fortune sampling is efficient, and the empirical comparisons are relevant and consistently reported. If the unbiasedness claim held, PLUMAGE would be a valuable contribution to memory-efficient LLM training. The main empirical result—PLUMAGE reduces the gap to full-rank Adam relative to GaLORE by roughly a third in perplexity and by about 28% in GLUE loss—is a useful practical finding. However, the theoretical claim that the deployed algorithm remains unbiased is not established; this directly affects the paper's title and abstract. With that claim reframed or proved under explicit assumptions, the work would be publishable.

major comments (3)
  1. [§3.3.2, Eqs. (25)-(27)] The sentence 'the estimate remains unbiased so long as #total_training_steps >> τ' is asserted without proof and is not correct as stated. Once P_kt is sampled, it is fixed for τ steps; at any later step t the projection is measurable with respect to the past, so the conditional expectation of the update is E[P D^{-1} P^T G_t | F_{t-1}] = P D^{-1} P^T G_t, not G_t. The identity E[P D^{-1} P^T] = I used in §3.1 requires an independent redraw of P at the same G_t, which is not what Algorithm 5 does. The condition '#steps >> τ' only describes refresh frequency and does not remove the dependence between the stale projection and future gradients. The unbiasedness guarantee of §3.1 therefore does not transfer to the deployed loop. Please either derive a corrected statement (e.g., a bias bound under an explicit subspace-stability assumption) or present PLUMAGE as a heuristic and remove 'unbiased' from the title and abstract.
  2. [Algorithm 5] The pseudocode is internally inconsistent in its handling of moment realignment. In the branch where the projection is refreshed, M_t and V_t are overwritten by update_state(...) before the moving-average updates 'M_t <- β1 M_{t-1} + ...' and 'V_t <- β2 V_{t-1} + ...', so the old-subspace moments that should feed the EMA are no longer available. This makes the described algorithm non-reproducible. Please clarify the intended ordering (e.g., compute R_t, update the EMA in the new subspace, and realign the previous moments before combining, or explicitly store the old moments).
  3. [§3.1 / title] The minimum-variance claim is established only within the restricted family of estimators in Eq. (7) with fixed inclusion probabilities p_i. The title's 'Minimum-vAriance' and the term 'MVUE' in §3.2 are stronger than what is proven; no optimality over all unbiased estimators is shown. Please qualify the claim (e.g., 'minimum variance within the class of component-wise estimators of the form (7)').
minor comments (6)
  1. [Eq. (11)] The orthogonality relation should read u_i^T u_j = δ_ij (or equivalently Tr(v_i v_j^T) = δ_ij); v_i^T u_j is not defined when m ≠ n.
  2. [Eqs. (25)-(26)] The subscripts are garbled; the right-hand side should reference M^{kt}_{t-1} and V^{kt}_{t-1}, not M^{kt}_t and V^{kt}_t.
  3. [Abstract and §3.3.2] The claim that no hyperparameters are introduced beyond r and τ is contradicted by κ in §3.3.2 and by γ_shrink, γ_expand, γ_reset in Appendix A; please clarify that these are optional and were set to fixed values in the main experiments.
  4. [Table 2] The 'Mean' row reports the average terminal loss, but the caption does not define whether these are training or validation losses; the abstract's 28% claim should reference this explicitly.
  5. [Algorithm 2] Notation is inconsistent with §3.1 (d vs n, r* = d-c unexplained); please align the pseudocode with the main text.
  6. [General] Inconsistent capitalization of GaLore/GALORE appears throughout; please choose one spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PLUMAGE's MVUE derivation is self-contained, and the §3.3.2 reuse-bias issue is a correctness gap rather than a circular step.

full rationale

The central derivation chain is self-contained. Section 3.1 defines the estimator in Eq. (7), imposes unbiasedness via Eq. (8) (p_i = E[I_i]), enforces the exact-rank constraint in Eq. (9), minimizes the trace variance in Eq. (12), and solves the resulting KKT conditions to obtain r* and p_i in Eqs. (17)–(18). The one-sided estimator in Eq. (20) is then reduced algebraically to the two-sided estimator using the SVD and orthonormality in Eq. (21), so the MVUE claim is derived rather than imported from a citation or fitted to the reported losses. The empirical results in Tables 1–4 are measurements taken under fixed hyperparameters, not predictions generated from fitted constants, so there is no fitted-input-called-prediction pattern. The only self-citations ([6] and [7], sharing author D. Soudry) support background statements about gradient bias and sparse-gradient design; the paper re-derives the sampling probabilities instead of relying on those papers, so the self-citations are not load-bearing. One passage does assert a missing proof: Section 3.3.2 states 'the estimate remains unbiased so long as #total_training_steps≫τ'. With P_kt sampled once and fixed for τ steps, P_kt is part of the conditioning information at later steps, and E[P D^{-1}P^T G_t | F_{t-1}] = P D^{-1}P^T G_t, not G_t, so the unconditional unbiasedness from Section 3.1 does not transfer automatically. This is an unsupported correctness claim about the deployed loop, but it is not a reduction of a claimed result to its own inputs, and it does not make the derivation circular. Verdict: no significant circularity; score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central derivation has no free parameters of its own; the hyperparameters r, tau (and optional kappa) are user choices, and the adaptive-controller thresholds in the appendix are hand-tuned. The main load-bearing assumption is that a stale projection remains an approximately unbiased sample over the update interval; this is a domain assumption, not proven in the paper.

free parameters (4)
  • rank r = 128 (130M/350M pretrain), 512 (1B), 8 (GLUE)
    Chosen by hand following GaLORE's budget; determines the number of sampled singular directions and the memory/compute trade-off. Not fitted to the target loss.
  • update interval tau = 200 (pretrain), 100 (GLUE)
    Interval between SVD refreshes; set to values from prior GaLORE experiments, not tuned per method.
  • resample interval kappa = set equal to tau in experiments
    Introduced in Section 3.3.2 as an optional third hyperparameter; set to tau throughout, so it does not add effective tuning in the reported runs.
  • adaptive thresholds gamma_shrink, gamma_expand, gamma_reset = 0.4, 0.6, 0.3
    Hand-set in Appendix A 'by observing the statistics during training'; only used in the adaptive-interval variant reported in Table 4, not in the main results.
assumptions (4)
  • standard math Singular vectors of G are orthonormal, so cross terms vanish in the variance computation.
    Used in Eqs. (11)-(12) to reduce the variance objective to a sum over marginal probabilities; standard linear algebra property of SVD.
  • domain assumption Gradient second-moment matrix is diagonal in the projected coordinate frame (E[G_kj G_lj] ≈ 0 for k != l).
    Invoked in Eq. (29) to approximate V2 realignment as (B∘2)V1. Same diagonal assumption Adam makes for its second-moment statistic; common but an approximation.
  • domain assumption The gradient's dominant subspace changes slowly enough that a projection sampled from G_kt remains an approximately valid sample for G_t over tau steps.
    Section 3.3.2: 'the estimate remains unbiased so long as #total_training_steps >> tau'. This unproven training-dynamics assumption is required for the reuse-unbiasedness claim and is not established; it is a domain assumption about LLM loss landscapes.
  • standard math The wheel-of-fortune systematic sampling algorithm draws exactly k distinct indices with the prescribed marginal inclusion probabilities.
    Algorithm 3 relies on this known property of systematic sampling (cited from MathOverflow [12]); used to satisfy the exact-k condition in Eq. (9).

how reviews work

0 comments
Cite this review

Pith. "Pith review of PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training." pith.science (2026). https://pith.science/paper/25SB56GM

@misc{pith2026250518313,
  author       = {Pith},
  title        = {Pith review of: PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25SB56GM}},
  note         = {Machine review of arXiv:2505.18313}
}
read the original abstract

Accelerator memory and networking constraints have emerged as dominant bottlenecks when training large language models LLMs with billions of parameters. Existing low rank gradient estimators such as GaLoRE and FLORA compress gradients and optimizer tensors by projecting weight gradients onto a rank r subspace, enabling LLM training on consumer hardware. Yet, these methods are either biased or subject to high estimator variance. Moreover, the optimizer state based on the first and second moments estimates expressed in the previous subspace becomes misaligned whenever the projection is updated, leading to instabilities during training. We propose PLUMAGE: Probabilistic Low rank Unbiased Minimum vAriance Gradient Estimator. PLUMAGE is a drop in replacement for existing low rank gradient estimators. It does not introduce new hyperparameters beyond the chosen rank r and the update interval. In addition, we resolve optimizer state misalignment issues to prevent spurious weight updates and enhance training stability. We empirically demonstrate that PLUMAGE shrinks the full rank optimization's gap over the pre training evaluation loss by 33% on average across models and the average training loss across the GLUE benchmark by 28% within a similar computational and memory footprint as GaloRE.

Figures

Figures reproduced from arXiv: 2505.18313 by the authors.

Figure 1
Figure 1. “Wheel-of-Fortune" sampling with k = 5 arms It is important to note that Ii need not be independent. Meaning, that we can sample the singular vectors however we want, just as long as we satisfy Eq. (18) (e.g. to satisfy the k-sparse condition in Eq. (9)). For example, we can first sample i = 1 . . . r∗ (deterministically), then sample a (k − r ∗ ) sparse solution from the rest of the compo￾nents. Specifically, let k… view at source ↗
Figure 3
Figure 3. Ablation studies comparing validation loss Llama 130m on C4 [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Tracking r∗ (deterministic rank) and ρ (mean cosine principal angle) during pretraining of Llama 1B on C4 with PLUMAGES/MP/A C Algorithms C.1 Auxilary algorithms In this subsection, we present the algorithms for computing the PLUMAGE probabilities (Al￾gorithm 2) and sampling from it exactly k indices (Algorithm 3) and constructing the projection matrices (Algorithm 4) as discussed in Sections 3 and 3.2. Note that al… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Mean ρ observed during LLama2-1B pretraining on C4 with rank=512 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Mean r ∗ observed during LLama2-1B pretraining on C4 with rank=512 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Pretraining loss plots of Llama on C4 datasts. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 24 canonical work pages

  1. [1]

    A. F. Aji and K. Heafield. Sparse communication for distributed gradient descent. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for 9 Computational Linguistics, 2017. doi: 10.18653/v1/d17-1045. URL http://dx.doi.org/ 10.18653/v1/D17-1045

    doi:10.18653/v1/d17-1045 2017

  2. [2]

    Alain, A

    G. Alain, A. Lamb, C. Sankar, A. Courville, and Y . Bengio. Variance reduction in sgd by distributed importance sampling, 2015

    work page 2015

  3. [3]

    Banner, I

    R. Banner, I. Hubara, E. Hoffer, and D. Soudry. Scalable methods for 8-bit training of neural networks, 2018

    work page 2018

  4. [4]

    Blumenfeld, I

    Y . Blumenfeld, I. Hubara, and D. Soudry. Towards cheaper inference in deep networks with lower bit-width accumulators, 2024

    work page 2024

  5. [5]

    X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y . Liu, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y . Lu, and Q. V . Le. Symbolic discovery of optimization algorithms, 2023

    2023

  6. [6]

    Chmiel, L

    B. Chmiel, L. Ben-Uri, M. Shkolnik, E. Hoffer, R. Banner, and D. Soudry. Neural gradients are near-lognormal: improved quantized and sparse training, 2020

    work page 2020

  7. [7]

    Chmiel, I

    B. Chmiel, I. Hubara, R. Banner, and D. Soudry. Minimum variance unbiased n:m sparsity for the neural gradients, 2022

    work page 2022

  8. [8]

    Courbariaux, I

    M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y . Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1, 2016

    work page 2016

Show all 41 references
  1. [9]

    Dettmers, M

    T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer. 8-bit optimizers via block-wise quantization, 2021

    2021

  2. [10]

    Dettmers, A

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URLhttps://arxiv.org/abs/2305.14314

    2023 arXiv

  3. [11]

    Eckart and G

    C. Eckart and G. Young. The approximation of one matrix by another of lower rank.Psychome- trika, 1(3):211–218, 1936

    1936

  4. [12]

    K. Fabian. How to sample exactly k indices given the inclusion probabilities of all indices?, 2024. URL https://mathoverflow.net/q/475554. URL:https://mathoverflow.net/q/475554 (ver- sion: 2024-07-23)

    2024

  5. [13]

    A. I. for AI. C4: Colossal clean crawled corpus. Dataset available at https://huggingface. co/datasets/allenai/c4, 2020

    2020

  6. [14]

    Gholami, Z

    A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer. Ai and memory wall. IEEE Micro, 44(3):33–39, May 2024. ISSN 1937-4143. doi: 10.1109/mm.2024.3373763. URL http://dx.doi.org/10.1109/MM.2024.3373763

    2024

  7. [15]

    Gupta, T

    V . Gupta, T. Koren, and Y . Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018

    2018

  8. [16]

    Y . Hao, Y . Cao, and L. Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024

    2024

  9. [17]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106. 09685

    2021

  10. [18]

    Hubara, B

    I. Hubara, B. Chmiel, M. Island, R. Banner, J. Naor, and D. Soudry. Accelerated sparse neural training: A provable and efficient method to find n:m transposable masks. In M. Ran- zato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neu- ral Infor...

    2021

  11. [19]

    ichi Amari, R

    S. ichi Amari, R. Karakida, and M. Oizumi. Fisher information and natural gradient learning of random deep networks, 2018. URLhttps://arxiv.org/abs/1808.07172

    2018 arXiv

  12. [20]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017. URL https: //arxiv.org/abs/1412.6980. 10

    2017 arXiv

  13. [21]

    Liang, B

    K. Liang, B. Liu, L. Chen, and qiang liu. Memory-efficient LLM training with online subspace descent. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. NeurIPS, 2024. URLhttps://openreview.net/forum?id=P8rTCT6g45

    2024

  14. [22]

    X. Liao, S. Li, Y . Xu, Z. Li, Y . Liu, and Y . He. Galore+: Boosting low-rank adaptation for llms with cross-head projection, 2024

    2024

  15. [23]

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https://arxiv.org/abs/1907.11692

    2019 arXiv

  16. [24]

    Muhamed, O

    A. Muhamed, O. Li, D. P. Woodruff, M. Diab, and V . Smith. Grass: Compute efficient low- memory llm training with structured sparse gradients.CoRR, abs/2406.17660, 2024. URL https://doi.org/10.48550/arXiv.2406.17660

    2024 doi arXiv

  17. [25]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019

    2019

  18. [26]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, page 1–16. IEEE, Nov. 2020. doi: 10.1109/sc41405.2020. 000...

    2020 arXiv

  19. [27]

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors.Nature, 323(6088):533–536, 1986

    1986

  20. [28]

    Shamshoum, N

    Y . Shamshoum, N. Hodos, Y . Sieradzki, and A. Schuster. Compact: Compressed activations for memory-efficient llm training, 2024. URLhttps://arxiv.org/abs/2410.15352

    2024 arXiv

  21. [29]

    N. Shazeer. Glu variants improve transformer, 2020

    2020

  22. [30]

    Shazeer and M

    N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018

    2018

  23. [31]

    Shoeybi, M

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019

    2019

  24. [32]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023 arXiv

  25. [33]

    V ogels, S

    T. V ogels, S. P. Karimireddy, and M. Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization, 2020. URLhttps://arxiv.org/abs/1905.13727

    2020 arXiv

  26. [34]

    N. Vyas, D. Morwani, R. Zhao, M. Kwun, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade. Soap: Improving and stabi lizing shampoo using adam, 2024

    2024

  27. [35]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https: //arxiv.org/abs/1804.07461

    2019 arXiv

  28. [36]

    H. Wang, S. Sievert, Z. Charles, S. Liu, S. Wright, and D. Papailiopoulos. Atomo: Communication-efficient learning via atomic sparsification, 2018

    2018

  29. [37]

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Huggingface’s transformers: State-o...

    2019

  30. [38]

    Zhang and R

    B. Zhang and R. Sennrich. Root mean square layer normalization, 2019

    2019

  31. [39]

    J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y . Tian. GaLore: Memory- efficient LLM training by gradient low-rank projection. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors,Proceedings of the 41st Internatio...

    2024

  32. [40]

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y . Hao, A. Math- ews, and S. Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel.Pro- ceedings...

    2023

  33. [41]

    Zhu and A

    P. Zhu and A. Knyazev. Angles between subspaces and their tangents.Journal of Numerical Mathematics, 21(4), Jan. 2013. ISSN 1570-2820. doi: 10.1515/jnum-2013-0013. URL http://dx.doi.org/10.1515/jnum-2013-0013. 12 Technical Appendices and Supplementary Material A Measuring proj...

    2013 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.