Pith. sign in

REVIEW 4 major objections 4 minor 66 references

Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Low-rank gradient projection can be tuned by projection granularity, and finer granularity beats larger rank at the same memory cost.

desk verdict A useful new knob for low-rank gradient projection, but the headline 'finer is better' rests on an untested bfloat16-numerics explanation and a theorem with a factor error. read the letter →

arxiv 2505.01744 v1 pith:Z7YKL4BD submitted 2025-05-03 cs.LG

Yezhen Wang , Zhouhao Yang , Brian K Chen , Fanyi Pu , Bo Li , Tianyu Gao , Kenji Kawaguchi This is my paper · ORCID
classification cs.LG
keywords low-rankgradientprojectiongranularitymemory-efficientfine-tuninglargelanguagemodelsProjFactoradaptiveoptimizersforwardestimationaccumulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that low-rank gradient projection (LoRP) has a second tuning knob: how finely the gradient matrix is sliced before projection. VLoRP reshapes each gradient matrix before applying a random projection, so a granularity factor $c$ and a rank $r$ jointly determine the memory cost $M=c r$. The authors claim that for any fixed $M$, finer-grained projection (larger $c$, smaller $r$) generally performs better than coarser projection with higher rank, across commonsense reasoning, MMLU, and GSM8K on LLaMA2-7B. They also introduce ProjFactor, an Adam-style optimizer that keeps the first moment in the projected subspace and stores a factored second moment, matching Adam-like dynamics while using less memory and supporting gradient accumulation.

What carries the argument

The load-bearing object is the projection granularity factor $c$. VLoRP reshapes a gradient $G\in\mathbb{R}^{n\times m}$ into $\tilde{G}\in\mathbb{R}^{nc\times (m/c)}$, applies row-wise random projection $\tilde{G}\tilde{P}$ with $\tilde{P}\in\mathbb{R}^{(m/c)\times r}$, then reshapes back; with entries i.i.d. from $N(0,1/r)$, the estimator is $G_o = \tilde{G}\tilde{P}\tilde{P}^\top$ reshaped to $n\times m$. The memory budget $M=c r$ fixes the stored size $ncr=nM$, and Proposition 3.2 gives $\mathbb{E}[G_o]=G$ and $\mathbb{E}\|G_o-G\|^2 = \frac{m+c}{M}\|G\|^2$, making $c$ and $r$ interchangeable to first order in the variance. ProjFactor keeps the first-moment state $\tilde{m}_s$ in the subspace, projects it back as $\tilde{m}_s\tilde{P}^\top$ for the update, and stores the second moment as a rank-1 row/column factorization in the Adafactor style; its continuous-time dynamics have Lyapunov function $H = L(W) + \langle \tilde{m}_s, \tilde{m}_s/\sqrt{\hat{v}_o}\rangle/(2a)$ and satisfy $dH/dt \leq 0$.

What would settle it

Run the same VLoRP configurations, for example $(c=256, r=1)$ versus $(c=1, r=256)$ with $M=256$, on LLaMA2-7B under the same tasks in float32 or float64; if the fine-grained configuration no longer outperforms the coarse one, the central claim reduces to a numerical-precision effect rather than a property of projection granularity.

Watch

Extended reading notes

Core claim

The paper's central claim is that projection granularity is a meaningful new degree of freedom in low-rank gradient projection. Concretely, instead of always projecting the $n\times m$ gradient matrix $G$ row by row into rank $r$, VLoRP reshapes $G$ into $\tilde{G}$ of shape $nc\times (m/c)$, projects rows of $\tilde{G}$ with a random Gaussian matrix $\tilde{P}$ of size $(m/c)\times r$, and reshapes the result back. Because only $\tilde{G}_s = \tilde{G}\tilde{P}$ is stored, its size is $ncr = nM$, so the memory budget $M=cr$ can be held fixed while $c$ and $r$ vary. The paper argues, and demonstrates on LLaMA2-7B across Commonsense170k, MMLU, and GSM8K, that configurations with finer granularity (larger $c$, smaller $r$) consistently outperform coarser ones at the same $M$, with the finest tested configuration $(c=256, r=1)$ best on all three benchmarks. It also proves the projected estimator stays unbiased with variance $O((m+c)/M)\|G\|^2$ and achieves $O(1/T)$ SGD convergence, and it gives ProjFactor a monotone-descent guarantee through a Hamiltonian-descent Lyapunov argument.

Load-bearing premise

The claim that finer granularity is always preferable at a fixed memory budget rests on the expectation that the gain is not just a bfloat16 rounding artifact, since the paper's numerical-error experiments are the only direct evidence for the mechanism and no float32 or float64 runs are shown.

Editorial extensions

If this is right

  • Under a fixed memory budget, choosing the finest granularity tested, such as $c=256$ with $r=1$, improves average accuracy over coarser configurations at equal memory on commonsense reasoning, MMLU, and GSM8K.
  • VLoRP with ProjFactor stores $O(mn + 2nM + n + m)$ per parameter matrix and keeps the gradient itself in projected form, so gradient accumulation does not require storing full-rank gradients.
  • The gradient estimator remains unbiased and its variance depends only on $(m+c)/M$, so changing granularity at fixed memory does not change the $O(1/T)$ SGD convergence rate.
  • ProjFactor's update dynamics have a Lyapunov function that decreases monotonically, guaranteeing convergence to a stationary point under the stated assumptions.
  • If the granularity claim holds, memory-limited fine-tuning can trade rank for granularity rather than simply lowering rank, which adds a new calibration axis for low-rank optimizers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper's numerical-error experiment gives a clean way to separate causes; rerunning the same configuration grid in float32 or float64 would show whether finer granularity remains best once bfloat16 rounding is removed.
  • Editorial inference: the variance identity suggests the granularity benefit may be layer- and scale-dependent, so an adaptive per-layer choice of $c$ could outperform any single global configuration.
  • Editorial inference: the forward-gradient viewpoint places VLoRP on a spectrum between whole-model gradient estimation and coordinate-wise estimation, so the same reshaping trick could be applied to zeroth-order optimizers or to non-Gaussian projection distributions.
  • Editorial inference: because numerical error compounds over repeated accumulation, coarse configurations might be rescued by higher-precision accumulators rather than by finer projections, which is a directly testable variant of ProjFactor.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VLoRP, a framework that generalizes low-rank gradient projection (LoRP) by introducing a granularity factor c that reshapes each gradient matrix from R^{n x m} to R^{nc x (m/c)} before projecting with rank r = M/c under a fixed memory budget M = cr. The central empirical claim is that, for a fixed M, finer projection granularity (larger c, smaller r) generally improves performance and stability. The paper also introduces ProjFactor, an Adam-style optimizer that stores first moments in the subspace and uses an Adafactor-style factorization for second moments, and it provides convergence guarantees: an O(1/T) bound for SGD (Theorem 3.3) and a monotone-descent/stationary-convergence result for ProjFactor via Hamiltonian descent (Theorem 4.1). Experiments compare VLoRP with Adam, Adafactor, LoRA, Galore, fira, and APOLLO on LLaMA2-7B, GPT2-XL, and LLaMA3.2-3B across commonsense reasoning, MMLU, and GSM8K.

Significance. If the fine-granularity advantage holds beyond the specific low-precision setting tested, VLoRP adds a simple and useful memory/performance trade-off knob to the LoRP family, and ProjFactor is a practical optimizer that reduces memory below Galore while remaining competitive. The paper's strengths include a clean formulation, extensive experiments across multiple models and memory budgets, and a falsifiable central claim: higher-precision experiments would directly test whether the benefit is a property of projection granularity or an artifact of bfloat16 arithmetic. The theoretical framework is conventional rather than machine-checked, but the estimator derivation is explicit and reproducible from the stated assumptions. The main weakness is that the paper's own Proposition 3.2 predicts a slight variance increase with finer granularity in exact arithmetic, so the only proposed mechanism for the empirical gain is reduced bfloat16 rounding error, a mechanism that is not isolated experimentally.

major comments (4)
  1. [Section 3.3 and Proposition 3.2] The headline claim that finer granularity is preferable under a fixed memory budget is not supported by the paper's own estimator analysis. Proposition 3.2 states E||G_o - G||^2 = (m+c)/M ||G||^2, so for fixed M the gradient-estimation variance slightly increases with c in exact arithmetic, not decreases. The empirical advantage is attributed in Section 3.3 ('Numerical Error') and Appendix D.3 to reduced bfloat16 rounding error, but no float32 or float64 experiments are reported. If the gain disappears in higher precision, the headline result reduces to a low-precision numerical tip rather than a property of projection granularity. Please add experiments that isolate the precision variable, and state clearly which part of the observed gain survives in float32/float64.
  2. [Theorem 3.3 and Appendix C.2] The stated convergence bound is algebraically inconsistent with the proof. With eta = C = M/((m+c+M)L), the denominator in inequality (10) is eta - (m+c+M)L eta^2/(2M) = C/2, so the resulting bound is 2/(C T) (L(W0)-L(W*)), not 2C/T as printed in Theorem 3.3. The proof's final equality '2(m+c+M)L/M = 2C' is also wrong; the left-hand side equals 2/C. This does not destroy the O(1/T) rate, but the theorem as stated and proved must be corrected.
  3. [Section 5 and Figure 12] The statement in Section 5 that 'the finest-grained VLoRP configuration (c=2^8, r=2^0) achieves the highest scores ... on ... GSM8k' is contradicted by the LLaMA3.2-3B results in Figure 12 and Appendix D.8, where c=2^4 (r=16) reaches the highest GSM8K score of 39.88 while c=2^8 (r=1) scores 36.77. The appendix softens this to a 'general trend,' but this exception should be reported in the main text and the unqualified claim amended, since it bears directly on the paper's central empirical conclusion.
  4. [Table 2 and Figure 3] MMLU and GSM8K results are reported without any measure of variability, and in Table 1 many adjacent configurations differ by less than the reported standard errors (e.g., 60.38 vs 60.39 for the two coarsest rows). Because the central empirical claim is a monotone trend in c, the absence of repeated runs or seed-dependent variability for two of the three benchmarks makes it difficult to assess whether the observed ordering is significant. Please report error bars or variance information for all headline results.
minor comments (4)
  1. [Section 5, memory analysis] The claimed memory complexity O(mn + 2nM + n + m) for VLoRP+ProjFactor does not reflect the dependence on c: the factored second-moment states have sizes nc and m/c, giving O(mn + 2nM + nc + m/c). The stated simplified form is acceptable only under an additional restriction on c that is not made explicit.
  2. [Appendix C.3, inequality (16)] The condition stated before (16) as 'setting a >= (R+1)b/4a' appears to be a typo; for the displayed inequality to hold the correct condition is a >= (R+1)b/4. Please correct it.
  3. [Figures 3 and 12] The asterisk on 'VLoRP*' in the left panels of Figures 3 and 12 is not explained in the caption or text; please define it.
  4. [Section 3.4] The symbol L is used both for the loss function and for the smoothness constant in Theorem 3.3, which is confusing; consider using L(.) for the loss and a different constant for smoothness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are derived from stated assumptions and benchmarked against external baselines, with self-citations used only as non-load-bearing motivation.

full rationale

The paper's central claim is empirical: under a fixed memory budget M=cr, finer projection granularity generally improves performance. This claim is tested against external baselines (Adam, Adafactor, LoRA, Galore, fira, APOLLO) on commonsense reasoning, MMLU, and GSM8K, so it is not a fitted quantity relabeled as a prediction. The theoretical results are derived in the appendix from explicit assumptions: Proposition 3.2 computes the mean and variance of the gradient estimator from first principles (Gaussian projection), and Theorem 3.3 gives an O(1/T) convergence bound using the stated smoothness and step-size choices. No step in this derivation is equivalent to its input by construction. The Hamiltonian-descent analysis of ProjFactor relies on stated assumptions (Assumption C.4), including the non-annihilation condition on the projection, but this is an explicit assumption rather than a hidden reuse of the conclusion, so it is a correctness/robustness concern rather than circularity. There are self-citations: Lemma C.1 references Shen et al. (2024), which shares authors, for a 'similar case' with Rademacher samples, but the Gaussian case is proven in the paper, so the citation is not load-bearing. FLoRA's equivalence theorem is cited as an independent prior result from different authors and is used for motivation, not to derive the paper's main claims. Thus, no load-bearing circular step was identified.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claims rely on standard random-projection theory and L-smoothness assumptions, plus several algorithm-specific premises (fixed projection matrix, low-rank gradient structure, bounded ratio assumption in C.4). The main free choices are hyperparameters c, r, M, learning rate, projection update gap, and warm-up; none of these is a physically invented entity.

free parameters (6)
  • learning rate eta = 2e-5 (Commonsense), 4e-5 (MMLU), 1e-4 (GSM8K)
    Set to the best values found through empirical testing (Appendix D.10); directly affects all reported results.
  • granularity factor c = powers of 2 from 2^-6 to 2^8
    The central hyperparameter; swept to establish the finer-is-better claim (Tables 5-7).
  • rank r = M/c, e.g., 1 to 2^14
    Chosen jointly with c to keep M fixed; the central claim depends on this pairing.
  • memory budget M = 256 (LLaMA2-7B), 64 (GPT2-XL)
    Arbitrary choice regulating comparison; results are also shown for M=16 and M=64 in Figure 4.
  • projection resampling gap tau = not stated as default; ablated 1-100 in D.5
    Algorithm 1 input; ablation suggests best around 20-30, but the default used in main tables is not specified.
  • warm-up steps = not stated; ablated 0, 10, 20, 50, 100 in D.6
    Affects early-training stability; the default is not reported.
assumptions (6)
  • standard math The loss is L-smooth with respect to the parameter matrix
    Used to prove Theorem 3.3 (Appendix C.2).
  • domain assumption Gradient matrices of LLMs are approximately low-rank
    The motivation for LoRP, stated in Section 2 and used to justify projecting gradients.
  • domain assumption Projection matrices are fixed for tau steps before resampling
    Algorithm 1 lines 4-6 and the ablation in D.5; the fixed-subspace update is integral to the empirical behavior.
  • ad hoc to paper Assumption C.4: stationarity of the Lyapunov function implies zero projected gradient; projection does not annihilate nonzero gradients; the ratio ||Go||^2/||v_o_r|| stays bounded by R
    These 'mild assumptions' are introduced specifically to prove Theorem 4.1 and are not verified in experiments.
  • standard math Gaussian random projections with entries N(0,1/r) preserve geometry (Johnson-Lindenstrauss-type guarantees)
    Used to connect LoRP to forward gradients and to justify random projection quality (Section 2, Appendix E.3).
  • domain assumption Reshape constraints hold: m/c and n*c are integers and c is a power of two
    The paper states this in Section 3.2 as an implementation requirement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients." pith.science (2026). https://pith.science/paper/Z7YKL4BD

@misc{pith2026250501744,
  author       = {Pith},
  title        = {Pith review of: Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7YKL4BD}},
  note         = {Machine review of arXiv:2505.01744}
}
read the original abstract

Building upon the success of low-rank adapter (LoRA), low-rank gradient projection (LoRP) has emerged as a promising solution for memory-efficient fine-tuning. However, existing LoRP methods typically treat each row of the gradient matrix as the default projection unit, leaving the role of projection granularity underexplored. In this work, we propose a novel framework, VLoRP, that extends low-rank gradient projection by introducing an additional degree of freedom for controlling the trade-off between memory efficiency and performance, beyond the rank hyper-parameter. Through this framework, we systematically explore the impact of projection granularity, demonstrating that finer-grained projections lead to enhanced stability and efficiency even under a fixed memory budget. Regarding the optimization for VLoRP, we present ProjFactor, an adaptive memory-efficient optimizer, that significantly reduces memory requirement while ensuring competitive performance, even in the presence of gradient accumulation. Additionally, we provide a theoretical analysis of VLoRP, demonstrating the descent and convergence of its optimization trajectory under both SGD and ProjFactor. Extensive experiments are conducted to validate our findings, covering tasks such as commonsense reasoning, MMLU, and GSM8K.

Figures

Figures reproduced from arXiv: 2505.01744 by the authors.

Figure 1
Figure 1. Overview of VLoRP versus standard LoRP. Bottom (gray) : In ordinary LoRP, the gradient matrix G is directly projected row-by-row from R n×m into R n×r and stored as Gs . Top (pale beige): In contrast, VLoRP reshapes the original gradient matrix G first to adjust the granularity of projection (from R n×m to R nc× m c ), and during the update, the project￾backed gradient G˜o would be reshaped back into R n×m to update… view at source ↗
Figure 2
Figure 2. Left: Schematic illustration of the Subspace Scheme (SS, green) operating in a learned subspace, and the Original Scheme (OS, blue) operating in the original space. Right (top row, panels a–c): Fine-tuning loss curves of SS (green), OS (blue), and Adam (orange) on three tasks, showing that OS outperforms SS by a large margin while has a comparable performance with Adam. Right (bottom row, panels d–f): Comparison of … view at source ↗
Figure 4
Figure 4. Performance Evaluation for Different Projection [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: Left: Performance comparison of different meth￾ods on GSM8K. Right: Performance comparison among the configurations of VLoRP with M = 256. The x-axis indicates configurations from fine to coarse (left to right). Different Memory Budgets Besides, in [PITH_FULL_IMAGE:fi…
Figure 6
Figure 6. Figure 6: Loss curves of different grained projections on the Commonsense170k dataset: (a) Training LLaMA2-7B with [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Left: Illustration of computational numerical error for a single parameter matrix during an update step. The numerical error of the projection operator is defined as the absolute difference between G˜ bf16P˜ bf16 and G˜ f loat64P˜ f loat64, averaged across all paramete…
Figure 8
Figure 8. Figure 8: Comparative Analysis of Projection Matrix Generations in LLaMA2-7B Training on the Commonsense170k [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Ablation study on the update frequency of the projection matrix: (a). Training loss statistics across different update [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Effect of different warm-up steps on training loss for VLoRP: (a). Learning rate schedules with varying warm-up [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Throughput Analysis of our method, LoRA, Galore, and Adam. Throughput, plotted on the y-axis, is defined as [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Left: Performance comparison of different methods on GSM8K with LLaMA3.2-3B. Right: Performance comparison among the configurations of VLoRP with M = 256. The x-axis indicates configurations from fine to coarse (left to right) [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 31 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S., Davis, A., Dean, J., Devin, M., et al

    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016

    arXiv 2016

  3. [3]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    G., Pearlmutter, B

    Baydin, A. G., Pearlmutter, B. A., Syme, D., Wood, F., and Torr, P. Gradients without backpropagation. arXiv preprint arXiv:2202.08587, 2022

    arXiv 2022

  5. [5]

    S., Cao, L., Choromanski, K., and Scheinberg, K

    Berahas, A. S., Cao, L., Choromanski, K., and Scheinberg, K. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22 0 (2): 0 507--560, 2022

    2022

  6. [6]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Inte...

    work page 2020

  7. [7]

    Deepzero: Scaling up zeroth-order optimization for deep model training

    Chen, A., Zhang, Y., Jia, J., Diffenderfer, J., Parasyris, K., Liu, J., Zhang, Y., Zhang, Z., Kailkhura, B., and Liu, S. Deepzero: Scaling up zeroth-order optimization for deep model training. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 , 2024 a

    work page 2024

  8. [8]

    Lion secretly solves constrained optimization: As lyapunov predicts

    Chen, L., Liu, B., Liang, K., and Liu, Q. Lion secretly solves constrained optimization: As lyapunov predicts. arXiv preprint arXiv:2310.05898, 2023

    arXiv 2023

Show all 66 references
  1. [9]

    Training deep nets with sublinear memory cost.(2016)

    Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost.(2016). arXiv preprint arXiv:1604.06174, 2016

    2016 arXiv

  2. [10]

    Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024 b

    Chen, X., Feng, K., Li, C., Lai, X., Yue, X., Yuan, Y., and Wang, G. Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024 b

    2024

  3. [11]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2019

  4. [12]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  5. [13]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [14]

    and Gupta, A

    Dasgupta, S. and Gupta, A. An elementary proof of a theorem of johnson and lindenstrauss. Random Struct. Algorithms, 22 0 (1): 0 60--65, 2003

    2003

  7. [15]

    Large scale distributed deep networks

    Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  8. [16]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10...

    2023

  9. [17]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  11. [19]

    Flora: Low-rank adapters are secretly gradient compressors

    Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , 2024

    2024

  12. [20]

    Lora+: Efficient low rank adaptation of large models

    Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , 2024

    2024

  13. [21]

    Towards a unified view of parameter-efficient transfer learning

    He, J., Zhou, C., Ma, X., Berg - Kirkpatrick, T., and Neubig, G. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022

    2022

  14. [22]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  15. [23]

    Parameter-efficient transfer learning for NLP

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for NLP . In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach...

    2019

  16. [24]

    J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022

    2022

  17. [25]

    Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E., Bing, L., Xu, X., Poria, S., and Lee, R. K. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical ...

    2023

  18. [26]

    E., and Kawaguchi, K

    Hu, Z., Yang, Z., Wang, Y., Karniadakis, G. E., and Kawaguchi, K. Bias-variance trade-off in physics-informed neural networks with randomized smoothing for high-dimensional pdes. arXiv preprint arXiv:2311.15283, 2023 b

    2023 arXiv

  19. [27]

    and Motwani, R

    Indyk, P. and Motwani, R. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Vitter, J. S. (ed.), Proceedings of the Thirtieth Annual ACM Symposium on the Theory of Computing, Dallas, Texas, USA, May 23-26, 1998 , pp.\ 604--613. ACM , 1998

    1998

  20. [28]

    From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients

    Jaiswal, A., Yin, L., Zhang, Z., Liu, S., Zhao, J., Tian, Y., and Wang, Z. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407.11239, 2024

    2024 arXiv

  21. [29]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  22. [30]

    J., Blankevoort, T., and Asano, Y

    Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 , 2024

    2024

  23. [31]

    Some extensions of liapunov's second method

    LaSalle, J. Some extensions of liapunov's second method. IRE Transactions on circuit theory, 7 0 (4): 0 520--527, 1960

    1960

  24. [32]

    The power of scale for parameter-efficient prompt tuning

    Lester, B., Al - Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pp...

    2021

  25. [33]

    Llava-onevision: Easy visual task transfer

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  26. [34]

    Memory-efficient llm training with online subspace descent

    Liang, K., Liu, B., Chen, L., and Liu, Q. Memory-efficient llm training with online subspace descent. arXiv preprint arXiv:2408.12857, 2024

    2024 arXiv

  27. [35]

    F., Cheng, K., and Chen, M

    Liu, S., Wang, C., Yin, H., Molchanov, P., Wang, Y. F., Cheng, K., and Chen, M. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , 2024

    2024

  28. [36]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019

    2019

  29. [37]

    J., Paulin, D., Teh, Y

    Maddison, C. J., Paulin, D., Teh, Y. W., O'Donoghue, B., and Doucet, A. Hamiltonian descent methods. arXiv preprint arXiv:1809.05042, 2018

    2018 arXiv

  30. [38]

    K., Ruder, S., Dehghani, M., and Henderson, J

    Mahabadi, R. K., Ruder, S., Dehghani, M., and Henderson, J. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confere...

    2021

  31. [39]

    D., Chen, D., and Arora, S

    Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36: 0 53038--53075, 2023

    2023

  32. [40]

    On variants of the johnson-lindenstrauss lemma

    Matousek, J. On variants of the johnson-lindenstrauss lemma. Random Struct. Algorithms, 33 0 (2): 0 142--156, 2008

    2008

  33. [41]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, ...

    2018

  34. [42]

    Nevel'son, M. B. and Has' minskii, R. Z. Stochastic approximation and recursive estimation, volume 47. American Mathematical Soc., 1976

    1976

  35. [43]

    H-fac: Memory-efficient optimization with factorized hamiltonian descent

    Nguyen, S., Chen, L., Liu, B., and Liu, Q. H-fac: Memory-efficient optimization with factorized hamiltonian descent. arXiv preprint arXiv:2406.09958, 2024

    2024 arXiv

  36. [44]

    Pearlmutter, B. A. Fast exact multiplication by the hessian. Neural Comput., 6 0 (1): 0 147--160, 1994

    1994

  37. [45]

    Residual prompt tuning: improving prompt tuning with residual reparameterization

    Razdaibiedina, A., Mao, Y., Khabsa, M., Lewis, M., Hou, R., Ba, J., and Almahairi, A. Residual prompt tuning: improving prompt tuning with residual reparameterization. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023 , pp...

    2023

  38. [46]

    Scaling forward gradient with local losses

    Ren, M., Kornblith, S., Liao, R., and Hinton, G. Scaling forward gradient with local losses. arXiv preprint arXiv:2210.03310, 2022

    2022 arXiv

  39. [47]

    and Monro, S

    Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, pp.\ 400--407, 1951

    1951

  40. [48]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,...

    2020

  41. [49]

    Socialiqa: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  42. [50]

    and Stern, M

    Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learnin...

    2018

  43. [51]

    Memory-efficient gradient unrolling for large-scale bi-level optimization

    Shen, Q., Wang, Y., Yang, Z., Li, X., Wang, H., Zhang, Y., Scarlett, J., Zhu, Z., and Kawaguchi, K. Memory-efficient gradient unrolling for large-scale bi-level optimization. arXiv preprint arXiv:2406.14095, 2024

    2024 arXiv

  44. [52]

    Learning by directional gradient descent

    Silver, D., Goyal, A., Danihelka, I., Hessel, M., and van Hasselt, H. Learning by directional gradient descent. In International Conference on Learning Representations, 2021

    2021

  45. [53]

    L., Kindermans, P., Ying, C., and Le, Q

    Smith, S. L., Kindermans, P., Ying, C., and Le, Q. V. Don't decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018

    2018

  46. [54]

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation

    Spall, J. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37 0 (3): 0 332--341, 1992

    1992

  47. [55]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  48. [56]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  49. [57]

    Vyas, N., Morwani, D., and Kakade, S. M. Adamem: Memory efficient momentum for adafactor. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024

    2024

  50. [58]

    J., and Xing, E

    Wang, C., Chen, X., Smola, A. J., and Xing, E. P. Variance reduction for stochastic gradient optimization. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8,...

    2013

  51. [59]

    Lora-ga: Low-rank adaptation with gradient approximation

    Wang, S., Yu, L., and Li, J. Lora-ga: Low-rank adaptation with gradient approximation. arXiv preprint arXiv:2407.05000, 2024

    2024 arXiv

  52. [60]

    Multilora: Democratizing lora for better multi-task learning

    Wang, Y., Lin, Y., Zeng, X., and Zhang, G. Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXiv:2311.11501, 2023

    2023 arXiv

  53. [61]

    Wengert, R. E. A simple automatic derivative evaluation program. Commun. ACM , 7 0 (8): 0 463--464, 1964

    1964

  54. [62]

    Williams, R. J. and Zipser, D. A learning algorithm for continually running fully recurrent neural networks. Neural Comput., 1 0 (2): 0 270--280, 1989

    1989

  55. [63]

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Paper...

    2019

  56. [64]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  57. [65]

    Galore: Memory-efficient LLM training by gradient low-rank projection

    Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , 2024

    2024

  58. [66]

    Z., Wang, Z., and Lee, J

    Zhu, H., Zhang, Z., Cong, W., Liu, X., Park, S., Chandra, V., Long, B., Pan, D. Z., Wang, Z., and Lee, J. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05270, 2024

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.