Pith. sign in

REVIEW 4 major objections 3 minor 43 references

Random low-rank projections are a geometrically principled choice for LLM training because gradient subspaces are nearly flat and leak a significant residual signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:46 UTC pith:VSSQMS7Q

load-bearing objection Abstract promises a theorem the paper never delivers; the empirical gains are tiny and hinge on an unvalidated scale-ratio assumption, but the random-walk/jump subspace update is a genuinely new idea worth watching. the 4 major comments →

arxiv 2510.01878 v2 pith:VSSQMS7Q submitted 2025-10-02 cs.LG

Geometrically Principled Randomized Optimization for Efficient LLM Training

Sahar Rajabi , Nayeema Nonta , Sirisha Rambhatla This is my paper
classification cs.LG
keywords low-rank gradient optimizationGrassmannian manifoldrandom projectionsLLM pretrainingoptimizer memorysubspace trackingresidual recoverygradient geometry
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

During LLM pretraining, the gradient's dominant low-rank subspace captures most energy early but becomes less dominant over time, especially in deeper layers, and it evolves in a nearly flat landscape. The paper argues that this geometry explains why random projections work: they are not merely cheap approximations but purposeful exploration of an almost-flat manifold, and they need to be paired with two ingredients—an optimizer that rotates its state when the subspace changes, and a residual-recovery step that rescales the discarded gradient using a columnwise ratio borrowed from the low-rank optimizer output. On this basis, the authors introduce GrassWalk and GrassJump, which perform random walks and jumps on the Grassmannian manifold, and report better evaluation loss on LLaMA-1B and LLaMA-7B pretraining than structured baselines while keeping GaLore-level memory. The central claim is that randomization is a feature, not a shortcut, once the geometry of gradient subspaces is accounted for.

Core claim

The central discovery is that the gradient subspace of a transformer during pretraining is both almost flat—the singular values of its estimation-error derivative are tiny and decaying—and incomplete: a non-negligible, growing fraction of gradient energy lies in the residual bulk outside the rank-r core. From this the authors infer that randomized updates, whether random walks or jumps on the Grassmannian, can navigate the manifold as effectively as structured tracking, and that a subspace-aware Adam that rotates its moments plus a residual recovery that rescales the bulk signal are necessary to make random projections competitive. They operationalize this as GrassWalk and GrassJump, showing

What carries the argument

The central object is the Grassmannian manifold Gr(r, n), the space of r-dimensional subspaces, equipped with random-walk updates via the exponential map (GrassWalk) or fresh random orthonormal jumps (GrassJump). These are coupled with (a) a subspace-aware optimizer (AO) that rotates Adam's first and second moments onto the new basis, and (b) a recovery-scaling (RS) step that reintroduces the residual Δ_t = G_t − S_t G̃_t by columnwise rescaling with the ratio between Adam's output and the raw low-rank gradient. A supporting quantity is the energy ratio R_t = ||G̃_t||_F / ||G_t||_F, which measures how much gradient energy the core subspace retains over training.

Load-bearing premise

The load-bearing premise is that the columnwise ratio between Adam's output and the raw low-rank gradient in the core subspace can be used to rescale the residual gradient, even though the paper does not measure whether that ratio actually transfers to the residual.

What would settle it

Compute, during LLaMA pretraining, the true columnwise scale factor that would optimally rescale the residual Δ_t (for instance, by comparing against the update direction of full-rank Adam), and check whether it matches the ratio derived from the low-rank optimizer output; a large mismatch in early or late layers would invalidate the recovery step. Alternatively, run GrassJump on a synthetic optimization problem with deliberately high-curvature gradient subspaces: if random projections still win, the flat-curvature explanation is not the operative mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the geometric explanation holds, random projections become a legitimate design choice for low-rank gradient training, not a stopgap, and can be combined with any subspace-update schedule.
  • The residual-recovery mechanism becomes a required component: without it, random projections lose their advantage, as the paper's ablation shows.
  • The flat-curvature insight suggests that other stochastic exploration techniques on the optimizer manifold could yield similar benefits.
  • The method achieves memory savings comparable to existing low-rank projection methods while lowering evaluation loss, pointing toward scalable pretraining on limited hardware.
  • The decline of core-subspace dominance in deeper layers implies that rank or update frequency should be layer- and time-sensitive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the flat-curvature observation generalizes, randomized subspace exploration might act as a form of implicit regularization that could also benefit fine-tuning, not just pretraining.
  • The scale-ratio transfer in Equation 9 is assumed rather than measured; one could test it directly by comparing the columnwise ratio on the residual against the ratio derived from the core subspace, which would validate or refute the recovery step.
  • The random-walk step size and jump interval are likely critical hyperparameters; the paper leaves their adaptive tuning open, so a follow-up study could tie them to the measured curvature of the gradient subspace.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies low-rank gradient subspace methods for LLM pretraining and introduces two randomized algorithms, GrassWalk and GrassJump, that update the projection subspace by random walks/jumps on the Grassmannian, combine this with an adaptive optimizer (AO) and residual recovery scaling (RS), and report improved evaluation loss on LLaMA-1B and LLaMA-7B compared with several baselines. The abstract claims a theoretical proof that random low-rank projections preserve gradient-subspace geometry, but the main text contains no theorem or proof. The empirical gains over the strongest baseline (SubTrack++) are small (0.02–0.10 in evaluation loss), and the manuscript does not include the Qwen-1.5B experiments promised in the abstract.

Significance. If its central claims were fully established, the paper would contribute a valuable practical recipe for memory-efficient LLM pretraining: systematic ablations (Fig. 3) isolating subspace-update method, optimizer alignment, and residual recovery are a strength, and the observation that random projections become competitive only when paired with AO+RS is interesting. However, the advertised theoretical foundation is absent, the key scale-ratio assumption behind RS is not validated, and the reported margins over the main baseline are extremely small with no error bars. The work is therefore currently more a collection of promising empirical components than a demonstrated state-of-the-art method.

major comments (4)
  1. [Abstract and Section 4] The abstract states: 'we theoretically establish that random low-rank projections preserve the geometry.' The full text contains no theorem, lemma, or proof supporting this statement. Section 4 gives update rules on the Grassmannian but does not formalize what 'preserve the geometry' means or prove that random projections have this property. This is the paper's headline contribution, so the manuscript does not deliver what it advertises.
  2. [Eq. (9) and Fig. 3] The residual-recovery mechanism Λ_t = φ_t(G_t) Δ_t rescales the discarded residual by a per-column ratio computed from the core-subspace gradient and optimizer output. The paper cites prior work for the consistency of this ratio, but presents no direct measurement that the ratio transfers from the core subspace to the residual in the proposed random-projection setting. The ablation in Fig. 3 shows that random projections alone are worse than Grassmannian tracking (6.16 vs 5.97) and only become competitive (3.87 vs 3.89) after adding AO+RS. The claimed advantage over SubTrack++ therefore rests on an unvalidated assumption.
  3. [Abstract, Tables 1 and 2] The abstract claims state-of-the-art results on LLaMA-1B, LLaMA-7B, and Qwen-1.5B pretraining, but no Qwen-1.5B experiments appear in the paper. Moreover, the reported gains are very small: GrassWalk 3.86 vs SubTrack++ 3.89 in Table 1, and GrassWalk ties SubTrack++ at 4.37 in Table 2 while GrassJump is 0.10 lower. No error bars or multiple seeds are reported. The evidence does not support the 'state-of-the-art' claim, especially since Table 2 omits all baselines except SubTrack++.
  4. [Section 3, Fig. 2] The conclusion that the gradient subspace evolves in 'almost flat curvature' is drawn from the small singular values of a subspace-estimation-error derivative. The paper does not provide a formal link between these singular values and the curvature of the optimization landscape on the Grassmannian, nor does it prove that flatness makes random steps advantageous. The chain from observation to algorithm is qualitative; without a precise statement, the paper's thesis that randomization is 'geometrically principled' is not established.
minor comments (3)
  1. [Section 5 heading] Typo in heading: 'Pre-Taining Experiments' should be 'Pre-Training Experiments.'
  2. [Algorithm 1] Pseudocode contains unclear/erroneous lines: 'St ←random rank-rorthonormal matrix' is immediately followed by a full line about generating a random matrix and updating the subspace, which is redundant; also the initialization line uses S for both singular values and the subspace matrix, causing a notation conflict with Eq. (2).
  3. [References] The main baseline 'Anonymous, 2025' (SubTrack++) is an anonymous submission whose code is not available; given that it is the primary comparator and the other methods are well-established, the comparison would be much stronger if the authors provided a public implementation or detailed hyperparameter settings for this baseline.

Circularity Check

0 steps flagged

No construction-level circularity: the benchmark losses are observed training outcomes, not outputs of fitted equations; the RS scale-ratio is imported from external prior work, and no self-citation chain is shown to be load-bearing.

full rationale

I walked the derivation chain: Eqs. (4), (7)-(8), and (9)-(10) define the GrassWalk/GrassJump updates rather than fit any target; the losses in Tables 1-2 are empirical results of running LLaMA pretraining, so no equation is constructed to reproduce the benchmark numbers. The only external load-bearing assumption is the recovery-scaling ratio in Eq. (9), introduced as "Based on the observation that the scale ratio between dominant and bulk subspaces is consistent (Zhu et al., 2025; Chen et al., 2025b)" and attributed to prior work by other groups. This is an imported empirical premise and it is not directly validated in the paper, which is a correctness/evidence concern, but it is not a circular reduction because the ratio is neither fitted to the reported losses nor derived from this paper's own claims. The anonymous SubTrack++ baseline is used as a comparison and as the experimental setting, but the manuscript gives no evidence of author overlap, so I cannot treat it as a load-bearing self-citation; even if it were, it is a baseline competitor rather than a premise that forces the conclusion. I also note for the record that the abstract states "we theoretically establish that random low-rank projections preserve the geometry," while the full text contains no theorem or proof for this statement; that is an omitted proof and a verifiability gap, but not a circularity. The ablation (Fig. 3) actually shows that random projections alone are weak and that AO/RS carry most of the gain, which weakens the attribution of the improvements to the random-walk mechanism; again, this is an experimental-interpretation issue, not a definitional or fitting-based circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The method depends on hand-set η, T, ζ, and r, with only T=100 disclosed. The theoretical statement (random projections preserve geometry) is an unproved assertion, and the residual-recovery step imports an unvalidated scale-ratio assumption from prior work. No new physical entity is introduced.

free parameters (5)
  • η (Grassmannian geodesic step size, Eq. 4)
    Controls how far the random walk moves the subspace every T steps; no value or sensitivity analysis is reported.
  • T (subspace update interval) = 100
    Stated in Section 5: 'the subspace updated every 100 steps.' Global hyperparameter.
  • ζ (recovery-scaling growth-rate limiter, Eq. 10)
    Limits divergence of the residual rescaling; no value or sensitivity analysis is reported.
  • r (subspace rank)
    Rank of the projection and optimizer state; not reported for the experiments, though it directly sets the memory/performance tradeoff.
  • random direction sampling distribution for X and GrassJump basis
    The distribution used to sample random rank-r matrices is never specified; it determines how exploration behaves.
axioms (6)
  • ad hoc to paper Random low-rank projections preserve gradient-subspace geometry
    The abstract asserts this is theoretically established, but Section 4 gives no theorem or proof; it is an unsupported premise for the framing.
  • domain assumption A low-rank core subspace exists and captures most gradient energy
    Section 3 (Eq. 3, Figure 1) assumes the validity of prior subspace results and interprets R_t > 0.5 as evidence that projection is sensible.
  • domain assumption Scale ratio between dominant and residual subspaces is consistent
    Equation 9 rescales the residual Δ_t using the per-column ratio of optimizer output to raw low-rank gradient; no validation is provided in this paper.
  • domain assumption Adam moments can be rotated as statistical estimates under subspace change
    Equations 7–8 are inherited from anonymous SubTrack++/LDAdam-style reasoning; Eq. 8 is garbled and no derivation is shown.
  • ad hoc to paper Flat subspace curvature makes random steps advantageous
    Inferred from Figures 1–2 (top-20 singular values of a subspace estimation error derivative); no formal connection between this flatness and optimizer escape behavior is established.
  • standard math Grassmannian exponential map and SVD-based geodesic update are correct
    Equation 4 relies on standard Grassmannian geometry as cited in Bendokat et al. (2024); this part is standard.

pith-pipeline@v1.3.0-alltime-deepseek · 12041 in / 14263 out tokens · 114619 ms · 2026-08-04T12:46:17.817428+00:00 · methodology

0 comments
read the original abstract

Low-rank gradient optimization for large language models is currently divided into two categories: structured methods that rigorously identify subspaces, and randomized approaches employed primarily for computational efficiency. In this work, we question the intuition behind why random projections are effective. We trace this phenomenon to the geometry of the gradient subspaces, which exhibits subspace optimization landscape has a nearly flat curvature, while a significant portion of gradient information lies outside the core subspace. Leveraging these insights, and drawing on randomized linear algebra, we theoretically establish that random low-rank projections preserve the geometry, and we introduce GrassWalk and GrassJump, algorithms that navigate the Grassmannian manifold via random walks and jumps. By coupling this randomized exploration with subspace-aware optimizer and recovering the lost gradient signals, we achieve state-of-the-art results on LLaMA-1B, LLaMA-7B, and Qwen-1.5B pretraining. Our findings reframe randomization not merely as a computational shortcut, but as a geometrically principled approach to high-dimensional optimizations.

Figures

Figures reproduced from arXiv: 2510.01878 by Nayeema Nonta, Sahar Rajabi, Sirisha Rambhatla.

Figure 1
Figure 1. Figure 1: Each decoder layer stack includes seven layer types. In the 1B model, the plots show the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of the top 20 singular values of the subspace estimation error derivative across [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: We ablate (i) the subspace update method: Grassmannian tracking, Grassmannian random [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of different methods on LLaMA pretraining. (a) Wall-clock training curves [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Subtrack++ : Gradient subspace tracking for scalable LLM training

    Anonymous. Subtrack++ : Gradient subspace tracking for scalable LLM training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=6geRIdlFWJ

    2025

  3. [3]

    Online identification and tracking of subspaces from highly incomplete information, 2011

    Laura Balzano, Robert Nowak, and Benjamin Recht. Online identification and tracking of subspaces from highly incomplete information, 2011. URL https://arxiv.org/abs/1006.4046

    Pith/arXiv arXiv 2011

  4. [4]

    Thomas Bendokat, Ralf Zimmermann, and P.-A. Absil. A grassmann manifold handbook: basic geometry and computational aspects. Advances in Computational Mathematics, 50 0 (1), January 2024. ISSN 1572-9044. doi:10.1007/s10444-023-10090-8. URL http://dx.doi.org/10.1007/s10444-023-10090-8

    doi:10.1007/s10444-023-10090-8 2024

  5. [5]

    signsgd: Compressed optimisation for non-convex problems, 2018

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems, 2018. URL https://arxiv.org/abs/1802.04434

    Pith/arXiv arXiv 2018

  6. [6]

    Blocker, Haroon Raja, Jeffrey A

    Cameron J. Blocker, Haroon Raja, Jeffrey A. Fessler, and Laura Balzano. Dynamic subspace estimation with grassmannian geodesics, 2023. URL https://arxiv.org/abs/2303.14851

    Pith/arXiv arXiv 2023

  7. [7]

    Greedy low-rank gradient compression for distributed learning with convergence guarantees, 2025 a

    Chuyan Chen, Yutong He, Pengrui Li, Weichen Jia, and Kun Yuan. Greedy low-rank gradient compression for distributed learning with convergence guarantees, 2025 a . URL https://arxiv.org/abs/2507.08784

    arXiv 2025

  8. [8]

    Fira: Can we achieve full-rank training of LLM s under low-rank constraint?, 2025 b

    Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of LLM s under low-rank constraint?, 2025 b . URL https://openreview.net/forum?id=lR7rqLtsXZ

    2025

  9. [9]

    A memory efficient randomized subspace optimization method for training large language models

    Yiming Chen, Yuan Zhang, Yin Liu, Kun Yuan, and Zaiwen Wen. A memory efficient randomized subspace optimization method for training large language models. In Forty-second International Conference on Machine Learning, 2025 c . URL https://openreview.net/forum?id=XuCf87V8OF

    2025

  10. [10]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [11]

    Roberts, and Ethan Dyer

    Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace, 2018. URL https://arxiv.org/abs/1812.04754

    Pith/arXiv arXiv 2018

  12. [12]

    Flora: Low-rank adapters are secretly gradient compressors, 2024

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024. URL https://arxiv.org/abs/2402.03293

    Pith/arXiv arXiv 2024

  13. [13]

    Subspace optimiztion for large language models with convergence guarantees, 2025

    Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimiztion for large language models with convergence guarantees, 2025. URL https://openreview.net/forum?id=udtrtwkvk5

    2025

  14. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients, 2024

    Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients, 2024. URL https://arxiv.org/abs/2407.11239

    Pith/arXiv arXiv 2024

  16. [16]

    Fast online low-rank tensor subspace tracking by cp decomposition using recursive least squares from incomplete observations, 2017

    Hiroyuki Kasai. Fast online low-rank tensor subspace tracking by cp decomposition using recursive least squares from incomplete observations, 2017. URL https://arxiv.org/abs/1709.10276

    Pith/arXiv arXiv 2017

  17. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv 2017

  18. [18]

    Relora: High-rank training through low-rank updates, 2023

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates, 2023. URL https://arxiv.org/abs/2307.05695

    Pith/arXiv arXiv 2023

  19. [19]

    Memory-efficient LLM training with online subspace descent

    Kaizhao Liang, Bo Liu, Lizhang Chen, and qiang liu. Memory-efficient LLM training with online subspace descent. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=P8rTCT6g45

    2024

  20. [20]

    Badam: A memory efficient full parameter optimization method for large language models, 2024

    Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter optimization method for large language models, 2024. URL https://arxiv.org/abs/2404.02827

    Pith/arXiv arXiv 2024

  21. [21]

    Velora: Memory efficient training using rank-1 sub-token projections, 2024

    Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections, 2024. URL https://arxiv.org/abs/2405.17991

    Pith/arXiv arXiv 2024

  22. [22]

    Microadam: Accurate adaptive optimization with low space overhead and provable convergence, 2024

    Ionut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtic, Thomas Robert, Peter Richtarik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space overhead and provable convergence, 2024. URL https://arxiv.org/abs/2405.15593

    Pith/arXiv arXiv 2024

  23. [23]

    Grass: Compute efficient low-memory llm training with structured sparse gradients, 2024

    Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, and Virginia Smith. Grass: Compute efficient low-memory llm training with structured sparse gradients, 2024. URL https://arxiv.org/abs/2406.17660

    Pith/arXiv arXiv 2024

  24. [24]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024. URL https://arxiv.org/abs/2403.17919

    Pith/arXiv arXiv 2024

  25. [25]

    Laradji, and Mark Schmidt

    Amrutha Varshini Ramesh, Vignesh Ganapathiraman, Issam H. Laradji, and Mark Schmidt. Blockllm: Memory-efficient adaptation of llms by selecting and optimizing the right coordinate blocks, 2024. URL https://arxiv.org/abs/2406.17296

    Pith/arXiv arXiv 2024

  26. [26]

    Tied- L o RA : Enhancing parameter efficiency of L o RA with weight tying

    Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied- L o RA : Enhancing parameter efficiency of L o RA with weight tying. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ...

    work page doi:10.18653/v1/2024.naacl-long.481 2024

  27. [27]

    LDA dam: Adaptive optimization from low-dimensional gradient statistics

    Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. LDA dam: Adaptive optimization from low-dimensional gradient statistics. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Zkp1GuHerF

    2025

  28. [28]

    Identifying policy gradient subspaces, 2024

    Jan Schneider, Pierre Schumacher, Simon Guist, Le Chen, Daniel Häufle, Bernhard Schölkopf, and Dieter Büchler. Identifying policy gradient subspaces, 2024. URL https://arxiv.org/abs/2401.06604

    Pith/arXiv arXiv 2024

  29. [29]

    Does SGD really happen in tiny subspaces? In The Thirteenth International Conference on Learning Representations, 2025

    Minhak Song, Kwangjun Ahn, and Chulhee Yun. Does SGD really happen in tiny subspaces? In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=v6iLQBoIJw

    2025

  30. [30]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    Pith/arXiv arXiv 2023

  31. [31]

    Chain of lora: Efficient fine-tuning of language models via residual learning, 2024

    Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning, 2024. URL https://arxiv.org/abs/2401.04151

    Pith/arXiv arXiv 2024

  32. [32]

    Coap: Memory-efficient training with correlation-aware gradient projection, 2025

    Jinqi Xiao, Shen Sang, Tiancheng Zhi, Jing Liu, Qing Yan, Yuqian Zhang, Linjie Luo, and Bo Yuan. Coap: Memory-efficient training with correlation-aware gradient projection, 2025. URL https://arxiv.org/abs/2412.00071

    Pith/arXiv arXiv 2025

  33. [33]

    Invariant low-dimensional subspaces in gradient descent for learning deep matrix factorizations

    Can Yaras, Peng Wang, Wei Hu, Zhihui Zhu, Laura Balzano, and Qing Qu. Invariant low-dimensional subspaces in gradient descent for learning deep matrix factorizations. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  34. [34]

    Compressible dynamics in deep overparameterized low-rank learning & adaptation

    Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overparameterized low-rank learning & adaptation. arXiv preprint arXiv:2406.04112, 2024

    arXiv 2024

  35. [35]

    Global convergence of a grassmannian gradient descent algorithm for subspace estimation, 2016

    Dejiao Zhang and Laura Balzano. Global convergence of a grassmannian gradient descent algorithm for subspace estimation, 2016. URL https://arxiv.org/abs/1506.07405

    Pith/arXiv arXiv 2016

  36. [36]

    Adam-mini: Use fewer learning rates to gain more, 2024

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more, 2024. URL https://arxiv.org/abs/2406.16793

    Pith/arXiv arXiv 2024

  37. [37]

    Galore: Memory-efficient llm training by gradient low-rank projection, 2024 a

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024 a . URL https://arxiv.org/abs/2403.03507

    Pith/arXiv arXiv 2024

  38. [38]

    Adapprox: Adaptive approximation in adam optimization via randomized low-rank matrices, 2024 b

    Pengxiang Zhao, Ping Li, Yingjie Gu, Yi Zheng, Stephan Ludger Kölker, Zhefeng Wang, and Xiaoming Yuan. Adapprox: Adaptive approximation in adam optimization via randomized low-rank matrices, 2024 b . URL https://arxiv.org/abs/2403.14958

    Pith/arXiv arXiv 2024

  39. [39]

    Pan, Zhangyang Wang, and Jinwon Lee

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance, 2025. URL https://arxiv.org/abs/2412.05270

    Pith/arXiv arXiv 2025

  40. [40]

    FRUGAL : Memory-efficient optimization by reducing state overhead for scalable training

    Philip Zmushko, Aleksandr Beznosikov, Martin Tak \'a c , and Samuel Horv \'a th. FRUGAL : Memory-efficient optimization by reducing state overhead for scalable training. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=B4TyAILcE4

    2025

  41. [41]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  42. [42]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  43. [43]

    These findings explain both the strengths and weaknesses of existing randomized and structured approaches, and motivate our proposed algorithms

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2024