Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Randomized subspace optimization can make LLM training memory-efficient in both activations and optimizer states while matching the convergence rate of a standard stochastic first-order method.

desk verdict The activation-memory trick is genuinely new and the theory is coherent, but the theorem as stated does not cover the algorithm actually run in the experiments. read the letter →

arxiv 2502.07222 v1 pith:6NZK6ZFV submitted 2025-02-11 cs.LG

Yiming Chen , Yuan Zhang , Yin Liu , Kun Yuan , Zaiwen Wen This is my paper · ORCID
classification cs.LG MSC 65K1090C1568T07
keywords randomizedsubspaceoptimizationlargelanguagemodeltrainingmemory-efficientactivationmemoryreductionconvergenceguaranteesAdamoptimizerrandomprojectionsGaLore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper makes the case that LLM training can be reduced to a sequence of smaller random subspace problems rather than one high-dimensional problem. At each outer iteration the method samples a projection matrix $P^k$, solves a low-dimensional subproblem for coefficients $B$, and updates $W^{k+1}=W^k+P^k\tilde B^k$, so gradients, optimizer moments, and stored activations all shrink with the subspace rank. The central theoretical claim is a convergence rate matching standard stochastic first-order methods: after $K$ outer iterations, the average squared gradient norm is at most $18\hat L(\Delta_0/K+\epsilon)$, where $\hat L=\max_\ell m_\ell/r_\ell$ and $\epsilon$ is the allowed inexactness of each subproblem solve. A reader should care because this links a practical memory saving to a proof that the saving does not come with a worse convergence order. The experiments on LLaMA pre-training and GLUE fine-tuning support the practical claim that the memory saving is obtained without giving up performance relative to GaLore and Adam.

What carries the argument

The load-bearing object is the random projection $P^k$ together with the expected $\epsilon$-inexact subproblem condition. The projection shrinks the effective dimension of every object that is stored for backpropagation: gradients become $P^\top\nabla f(W)$, optimizer states become $r_\ell n_\ell$ per layer, and activations such as $Y$ can be replaced by $YP$. Lemma B.1 shows that the proximal subproblem objective $g_k(B)$ is strongly convex and smooth with constants governed by $\hat L=\max_\ell m_\ell/r_\ell$, and the proof telescopes the resulting descent inequalities to turn subproblem optimality gaps into a bound on $\|\nabla f(W^k)\|^2$ through the identity $\mathbb{E}\|P^\top\nabla f(W)\|^2=\|\nabla f(W)\|^2$.

What would settle it

Run RSO on a small smooth problem and record the actual projection statistics and subproblem errors. If $\|P_\ell^\top P_\ell-(m_\ell/r_\ell)I\|$ is not small for the projection actually used, or if the empirical average of $\|\nabla f(W^k)\|^2$ over $K$ outer iterations exceeds $18\hat L(\Delta_0/K+\epsilon)$ computed from measured quantities, then the theorem is not governing the deployed algorithm.

Watch

Extended reading notes

Core claim

The central discovery is that optimizing in a randomly selected low-dimensional subspace at each iteration preserves the convergence rate of full-space training while reducing every memory component that scales with the weight dimension. The paper proves that if the random projection $P^k$ satisfies $P_\ell^\top P_\ell=(m_\ell/r_\ell)I$ and $\mathbb{E}[P_\ell P_\ell^\top]=I$, and if each proximal subproblem $g_k(B)=f(W^k+P^kB)+\frac{1}{2\eta_k}\|B\|^2$ is solved to expected $\epsilon$-inexactness, then the squared gradient norm averaged over outer iterations is bounded by $18\hat L(\Delta_0/K+\epsilon)$. The mechanism is concrete: in a layer $Z=Y(W+PB)$, backpropagation needs only $YP$ rather than the full activation $Y$, and Adam moments live on the $r\times n$ subproblem rather than the $m\times n$ weight, which is what lowers both optimizer-state and activation memory.

Load-bearing premise

The proof requires that each random projection has exact scaling and unbiasedness and that every subproblem is solved to a verified expected $\epsilon$-inexactness; the implementation uses a Gaussian-style projection that only approximates the scaling and keeps the same projection for hundreds of steps without checking the inexactness condition.

Editorial extensions

If this is right

  • If the central claim is correct, LLM training can cut activation memory without switching to zeroth-order methods, because first-order gradients are computed inside the lower-dimensional subproblem.
  • The $O(1/K)$ bound with inexactness floor puts the memory savings on the same asymptotic footing as Adam-type methods, so the method does not trade away convergence order.
  • Lower-dimensional gradients reduce communication volume in data-parallel training, which the reported wall-clock comparisons attribute to faster iteration times that grow with model size.
  • The framework accommodates zeroth-order, first-order, and second-order subproblem solvers, so the memory-versus-convergence tradeoff is decoupled from the choice of inner optimizer.
  • Because the rate constant is $\hat L=\max_\ell m_\ell/r_\ell$, smaller subspace ranks give larger memory savings but proportionally weaken the convergence bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same layer-level identity $Z=Y(W+PB)$ suggests that the activation-saving mechanism could be applied to other memory-heavy tensors in a transformer, such as the intermediate feed-forward values, beyond what the paper's memory table counts.
  • Editorial extension: the theory implies a checkable adaptive rule for when to resample the subspace, since the proof assumes a fresh independent projection each outer iteration; the fixed 300-500 step intervals in the experiments are a practical compromise that the theorem does not directly cover.
  • Editorial extension: one could test the exact Haar or random-coordinate projections required by Assumption 5.3 against the Gaussian-style projection used in practice on a small controlled problem, and measure whether the empirical average of squared gradient norms stays inside the theorem's predicted bound.
  • Editorial extension: as the rank $r_\ell$ approaches $1$, the method approaches random coordinate descent, so the paper's bound offers a continuous interpolation between coordinate-wise and full-space optimization, with the memory cost scaling accordingly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Randomized Subspace Optimization (RSO) for training LLMs. At each outer iteration the algorithm samples projection matrices P^k, approximately solves the low-dimensional proximal subproblem (4a) in variables B, and updates W^{k+1}=W^k+P^k B~k. The authors claim this simultaneously reduces memory for optimizer states, gradients, and activations, and improves communication efficiency. The main theoretical result, Theorem 5.5, gives an averaged gradient-norm bound of order O(Lhat/K + Lhat*epsilon) under L-smoothness, Assumption 5.3 on the projection distribution, and an expected epsilon-inexact subproblem oracle (Definition 5.1); Table 2 reports subproblem and total sample complexities for zeroth-, first-, and second-order solvers. Experiments pre-train LLaMA models (60M-7B) and fine-tune RoBERTa, LLaMA, and OPT, reporting validation perplexity, GLUE scores, peak memory, and iteration time against Adam, GaLore, LoRA, and ReLoRA.

Significance. The proposed framework is timely and potentially useful: if the convergence result holds, RSO would be among the few subspace methods that reduce activation memory as well as optimizer-state memory while retaining a first-order-style rate. The paper's strengths include a relatively self-contained proof of Theorem 5.5, an explicit memory-counting appendix (Appendix A), and large-scale experiments including LLaMA-7B. However, the practical algorithm described in Section 6 does not implement the assumptions of the theorem: projections are reused for 300-500 steps, the epsilon-inexactness of the Adam subproblem solves is never checked, and the projection distribution is not specified. Together with an invalid inequality in Lemma B.1, this means the theoretical guarantee currently does not cover the empirical protocol. The empirical comparison in Table 6 also uses different batch sizes for Adam versus RSO/GaLore, weakening the reported head-to-head result.

major comments (4)
  1. [Section 5, Theorem 5.5; Section 4, Algorithm 1; Tables 9-10] The convergence bound Eq. (10) presupposes that each outer iteration k uses a freshly sampled projection P^k and that every subproblem (4a) is solved to expected epsilon-inexactness in the sense of Definition 5.1. The experiments in Section 6 and Appendix D fix the same projection for 300 or 500 inner Adam steps (Tables 9-10), reuse it for hundreds of coupled updates, and do not report any estimate of g_k(B~k) - g_k(B^k_*). Algorithm 1 also does not state whether B is reset to zero at each resampling as assumed by the theorem. Consequently Theorem 5.5 does not apply to the protocol used in Tables 3-7, and the claim of comparable complexity with Adam is not established for the deployed algorithm. Please either adapt the implementation to satisfy the theorem's conditions, or provide a separate convergence analysis for the fixed-subspace interval protocol.
  2. [Section 5, Assumption 5.3 and Remark 5.4] Assumption 5.3 requires P_l^T P_l = (m_l/r_l) I and E[P_l P_l^T] = I exactly. Remark 5.4 concedes that sampling from N(0, 1/r_l) only approximates the first identity, but the paper never states which distribution was actually used to generate the projection matrices in the experiments. If the experiments used Gaussian projections, the operator-norm bound ||P^T x||^2 <= (m/r)||x||^2 that underlies the definition of Lhat in Lemma B.1 is only approximate, so the value of Lhat used in Theorem 5.5 is not justified for the reported runs. Specify the distribution and, for Gaussian projections, either prove or state a high-probability version of the required identities, or switch to Haar or coordinate matrices for which Assumption 5.3 holds exactly.
  3. [Appendix B, Lemma B.1] The proof of Lemma B.1 contains an invalid inequality: from ||P_l^T (grad_l f(W+P B_1) - grad_l f(W+P B_2))||_F^2 <= ||P_l||_F^2 ||...||_F^2 it does not follow that the sum is bounded by max_l(m_l/r_l) times the gradient difference, because under Assumption 5.3 we have ||P_l||_F^2 = m_l, not m_l/r_l. The intended Lhat-smoothness can be recovered by using the operator norm ||P_l||_op^2 = m_l/r_l and the bound ||P(B_1-B_2)||^2 <= max_l(m_l/r_l)||B_1-B_2||^2, but as written the lemma does not prove Theorem 5.5. Please correct this step.
  4. [Appendix C.1, Table 6] The LLaMA-7B pre-training comparison is not controlled: RSO and GaLore use batch size 16, while Adam uses batch size 8. Because batch size changes both the optimization trajectory and the memory footprint, the reported perplexity and memory numbers are not an apples-to-apples comparison with Adam. Rerun Adam at batch size 16, or run all methods at the same batch size, or explicitly justify why the comparison remains valid.
minor comments (5)
  1. [Section 3.3] GaLore is cited as [Zhang et al., 2023b], but that reference is 'Fine-tuning happens in tiny subspaces', not the GaLore paper; the correct citation appears to be [Zhao et al., 2024a].
  2. [Appendix B, proof of Theorem B.2] In the telescoping step, the displayed '+ epsilon' before dividing by K should be '+ K epsilon' if each of the K subproblems contributes an epsilon term; after dividing by K this yields the stated epsilon in the final bound. Please correct the intermediate display.
  3. [Theorem 5.5] The definition of Lhat contains a stray superscript and reads 'Lhat := max_l{m_l/r_l}^L'; it should be Lhat := max_{l=1,...,L} m_l/r_l, with L denoting the number of layers.
  4. [Algorithm 1 and Section 6] Algorithm 1 should specify how each subproblem is initialized (the theorem assumes B0 = 0) and how many inner Adam steps are taken per outer iteration; currently this information is not given, which prevents reproducibility of the theoretical protocol.
  5. [Section 6.3, Table 5] The note that GaLore's SVD decomposition time is excluded appears only in the caption; the main text should state this limitation when claiming that RSO requires less time per iteration.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence proof is self-contained, and the empirical comparisons are against external baselines, not derived from the paper's own assumptions.

full rationale

The paper's central theoretical claim, Theorem 5.5 (Eq. 10), is proved in Appendix B directly from Assumptions 5.2 and 5.3 together with Definition 5.1. The proof establishes strong convexity and smoothness of the proximal subproblem g_k, telescopes a descent inequality, and converts E||P^T grad f(W^k)||^2 into E||grad f(W^k)||^2 using the unbiasedness condition E[P P^T] = I. No fitted constant, empirical measurement, or prior work is inserted into the derivation, so the bound is not equivalent to its inputs by construction. The expected epsilon-inexact subproblem condition is an oracle assumption on the solver; it is not a renamed version of the conclusion, and the same epsilon appears explicitly in the final bound, making the theorem a genuine conditional guarantee rather than a circular reduction. The practical concern identified in Remark 5.4—that Gaussian sampling only approximates Assumption 5.3—and the experimental fact that projections are fixed for 300-500 steps without verifying the epsilon condition are theory-practice gaps, not circularity: they question whether the theorem's hypotheses hold in the implementation, but they do not make the derivation self-referential. Self-citations to He et al. 2024, Chen et al. 2024a, and Wen et al. 2025 appear only in related-work attributions and are not load-bearing in the convergence proof; no uniqueness theorem or external result is imported from those works to force the paper's choices. Performance claims are benchmarked against Adam, GaLore, and LoRA using configurations from the public GaLore setup or independent runs, so the empirical claims are externally falsifiable rather than constructed from fitted parameters. No specific circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or dimensions are introduced; the random projection matrix P is a mathematical tool, not an invented physical entity. The free parameters are standard hyperparameters, and the axioms are the smoothness, projection identity, subproblem oracle, and bounded-below assumptions.

free parameters (4)
  • Rank r = 128, 256, 512, 1024, etc.
    User-chosen hyperparameter. The convergence constant L_hat = max_l m_l/r_l and the memory savings depend on r, but r is not fitted to make the proof work.
  • Learning rate scaling factor = 0.35 for pre-training; {8, 16, 32} for fine-tuning
    Chosen to match GaLore's scaling; affects empirical performance but not the theoretical rate.
  • Subspace update interval = 300 or 500 steps
    How long one random projection P is kept before resampling. This parameter appears in experiments but not in Algorithm 1's per-iteration resampling, creating a theory-practice gap.
  • Proximal coefficient eta_k = 1/(2 L_hat)
    Set to 1/(2 max_l m_l/r_l) in Theorem 5.5 to guarantee strong convexity of the subproblem. It is a chosen constant, not fitted to data.
assumptions (4)
  • domain assumption f is L-smooth (Assumption 5.2)
    Standard smoothness assumption for neural network losses, used throughout the proof.
  • ad hoc to paper Random projection satisfies P_l^T P_l = (m_l/r_l) I and E[P_l P_l^T] = I (Assumption 5.3)
    This exact identity is needed for the unbiasedness and operator norm estimates in the proof. Remark 5.4 acknowledges that practical Gaussian sampling only approximates it.
  • ad hoc to paper Subproblem solver returns an expected epsilon-inexact solution (Definition 5.1)
    The convergence guarantee depends on this oracle. The experiments use a fixed number of Adam steps and do not verify the epsilon condition.
  • standard math f is bounded below (f* > -infinity)
    Needed to define Delta_0 and telescope the descent inequality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models." pith.science (2026). https://pith.science/paper/6NZK6ZFV

@misc{pith2026250207222,
  author       = {Pith},
  title        = {Pith review of: A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NZK6ZFV}},
  note         = {Machine review of arXiv:2502.07222}
}
read the original abstract

The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numerous memory-efficient techniques have been proposed, with GaLore standing out as a notable example designed to reduce the memory footprint of optimizer states. However, these approaches do not alleviate the memory burden imposed by activations, rendering them unsuitable for scenarios involving long context sequences or large mini-batches. Moreover, their convergence properties are still not well-understood in the literature. In this work, we introduce a Randomized Subspace Optimization framework for pre-training and fine-tuning LLMs. Our approach decomposes the high-dimensional training problem into a series of lower-dimensional subproblems. At each iteration, a random subspace is selected, and the parameters within that subspace are optimized. This structured reduction in dimensionality allows our method to simultaneously reduce memory usage for both activations and optimizer states. We establish comprehensive convergence guarantees and derive rates for various scenarios, accommodating different optimization strategies to solve the subproblems. Extensive experiments validate the superior memory and communication efficiency of our method, achieving performance comparable to GaLore and Adam.

Figures

Figures reproduced from arXiv: 2502.07222 by the authors.

Figure 1
Figure 1. Memory components for training LLaMA-1B model with Adam optimizer [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of peak memory usage (in GB) per device for RSO and GaLore during LLaMA training with varying ranks. All hyperparameters, except rank, are consistent with [Zhao et al., 2024a]. Adam’s memory usage is reported for LLaMA-350M and LLaMA-1B but excluded for LLaMA-7B due to an out-of-memory (OOM) error. 6.3 Memory and Communication Efficiency To evaluate the memory and communication efficiency of our proposed … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed

    cs.LG 2025-06 reject novelty 6.0 of 10

    FZOO claims Adam-like zeroth-order fine-tuning via loss-std normalization and batched forward passes, but the paper's algorithm perturbs activations rather than parameters, breaking the link to its own theory.

Reference graph

Works this paper leans on

26 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165,

    arXiv 2005

  2. [7]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  3. [8]

    Chain of lora: Efficient fine-tuning of language models via residual learning

    Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.04151,

    arXiv

  4. [9]

    Subspace optimization for large language models with convergence guarantees

    Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint arXiv:2410.11289,

    arXiv

  5. [10]

    Flora: Low-rank adapters are secretly gradient compressors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293,

    arXiv

  6. [11]

    Enhancing zeroth-order fine-tuning for language models with low-rank structures

    Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.07698, 2024a. Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:...

    arXiv

  7. [12]

    Lora learns less and forgets less

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673,

    arXiv

  8. [13]

    Memory-efficient llm training with online subspace descent

    10 A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-efficient llm training with online subspace descent. arXiv preprint arXiv:2408.12857,

    arXiv

Show all 26 references
  1. [14]

    Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024b

    Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024b. Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. Ldada...

  2. [16]

    Breaking memory limits: Gradient wavelet transform enhances llms training

    Ziqing Wen, Ping Luo, Jiahuan Wang, Xiaoge Deng, Jinping Zou, Kun Yuan, Tao Sun, and Dongsheng Li. Breaking memory limits: Gradient wavelet transform enhances llms training. arXiv preprint arXiv:2501.07237,

    arXiv

  3. [18]

    Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15173, 2024b. John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono...

    arXiv

  4. [19]

    Training deep nets with sublinear memory cost

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174,

    arXiv

  5. [21]

    {Zero-offload}: Democratizing {billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. {Zero-offload}: Democratizing {billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564,

    2021

  6. [23]

    Unified convergence analysis for adaptive optimization with moving average estimator

    Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. Unified convergence analysis for adaptive optimization with moving average estimator. arXiv preprint arXiv:2104.14840,

    arXiv

  7. [24]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

    1907 arXiv

  8. [26]

    We also report the memory overhead and total training time for each method

    C More Experimental Results C.1 Pre-training on LLaMA-7B Model Table 6 compares the performance of our RSO method with GaLore and Adam on the LLaMA-7B model, where evaluations are conducted for 50K steps due to limited computational resources. We also report the memory overhea...

  9. [2014]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  10. [2016]

    Qlora: efficient finetuning of quantized llms (2023)

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized llms (2023). arXiv preprint arXiv:2305.14314, 52:3982–3992,

    2023 arXiv

  11. [2017]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024a. Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing ...

    arXiv

  12. [2018]

    Natural galore: Accelerating galore for memory-efficient llm training and fine-tuning

    Arijit Das. Natural galore: Accelerating galore for memory-efficient llm training and fine-tuning. arXiv preprint arXiv:2410.16029,

    arXiv

  13. [2019]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461,

    arXiv

  14. [2020]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  15. [2021]

    G10: Enabling an efficient unified gpu memory and storage architecture with smart tensor migrations

    Haoyang Zhang, Yirui Zhou, Yuqi Xue, Yiqi Liu, and Jian Huang. G10: Enabling an efficient unified gpu memory and storage architecture with smart tensor migrations. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pages 395–410, 2023a. Lé...

    2010

  16. [2023]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  17. [2024]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  18. [2025]

    Variance-reduced zeroth-order methods for fine-tuning language models

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.