Pith. sign in

REVIEW 3 major objections 5 minor 71 references

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RefLoRA: A new fine-tuning method that picks the optimal low-rank factorization each step, balancing updates and converging faster than LoRA.

desk verdict A principled LoRA variant with a real theoretical idea, but the abstract's 'optimal per-step' claim outruns the implemented algorithm. read the letter →

arxiv 2505.18877 v4 pith:AAYZJP23 submitted 2025-05-24 cs.LG

Yilang Zhang , Bingcong Li , Georgios B. Giannakis This is my paper · ORCID
classification cs.LG MSC 68T0790C2615A4568T50
keywords low-rankadaptationparameter-efficientfine-tuningmatrixgeometricmeanlossupperboundminimizationbalancedupdateslargelanguagemodelsconvergenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LoRA's slow and unstable convergence stems from the nonuniqueness of its low-rank factorization, which causes inconsistent and imbalanced updates. To fix this, it proposes RefLoRA, which selects the optimal factorization at each step by minimizing an upper bound on the loss. The central claim is that this optimal refactoring leads to balanced updates, consistent weight changes across equivalent factorizations, and a flatter loss landscape, resulting in faster and more stable fine-tuning of large models. If true, RefLoRA offers a principled way to improve LoRA's efficiency and performance with negligible extra computational cost.

What carries the argument

The central object is the $r \times r$ symmetric positive definite matrix $S_t := P_tP_t^\top$, which parameterizes all equivalent factorizations $(A_tP_t, B_tP_t^{-\top})$ of the low-rank update. By minimizing a Lipschitz-smoothness-based upper bound on the loss with respect to $S_t$, the optimal refactoring is shown to be the matrix geometric mean $\tilde{S}_t$ of $(A_t^\top A_t)^{-1}$ and $B_t^\top B_t$, which balances the factors and guarantees consistent updates.

What would settle it

Run RefLoRA on a large model (e.g., LLaMA-7B) with a range of learning rates spanning $10^{-5}$ to $10^{-3}$ and compare its convergence and final performance against vanilla LoRA; if for small learning rates LoRA matches or exceeds RefLoRA's performance, the practical optimality of the balanced update in the small-$\eta$ regime would be disproven.

Watch

Extended reading notes

Core claim

The paper discovers that by characterizing all equivalent LoRA factorizations of a weight update with a symmetric positive definite matrix $S_t$, it can derive a closed-form optimal $S_t$ that minimizes a quadratic upper bound on the loss. This optimal choice is the matrix geometric mean $\tilde{S}_t = (A_t^\top A_t)^{-1}\#(B_t^\top B_t)$, which enforces balanced updates ($A_t^\top A_t = B_t^top B_t$) and consistency across all equivalent factorizations. The authors prove that using this refactoring accelerates convergence and empirically demonstrate improvements on matrix factorization, GLUE, commonsense reasoning, and image generation tasks.

Load-bearing premise

The practical implementation always uses the balanced update $\tilde{S}_t$, but this is theoretically justified only for larger learning rates; for typical small learning rates ($\eta \sim 10^{-4}$), the true optimal $S_t$ is a scaled version of $\tilde{S}_t$, and the paper asserts without detailed proof that the balanced choice remains effective in that regime.

Editorial extensions

If this is right

  • RefLoRA can be applied to any LoRA-based fine-tuning pipeline, providing a drop-in replacement that improves convergence speed and stability.
  • The balanced update property allows using larger learning rates without divergence, potentially reducing total fine-tuning time.
  • The lightweight variant RefLoRA-S achieves near-LoRA throughput with minimal memory overhead, making it suitable for resource-constrained settings.
  • The theoretical framework offers a new perspective on optimizing low-rank adaptations, potentially inspiring further refinements in the field.
  • Negligible computational overhead means RefLoRA can scale to models with billions of parameters without significant efficiency loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistency theorem (Theorem 6) likely holds for any optimization method that uses the same preconditioning, suggesting a broader principle for designing factorized updates.
  • The balanced update condition could be linked to convergence guarantees in non-convex low-rank optimization, potentially leading to new theoretical bounds.
  • A testable extension would be applying RefLoRA's refactoring to other PEFT methods like DoRA or AdaLoRA to see if it further improves their performance.
  • The paper's reliance on the matrix geometric mean might indicate that distance-geometric tools from matrix analysis could find more applications in neural network optimization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces RefLoRA, a variant of LoRA that at each optimization step refactors the low-rank factors (A_t, B_t) by selecting an r-by-r symmetric positive definite matrix S_t that minimizes an upper bound on the loss. The proposed optimal S_t is shown, under Lipschitz smoothness and full-rank assumptions, to be the matrix geometric mean of (A_t^T A_t)^{-1} and (B_t^T B_t) in a certain large-learning-rate regime, yielding balanced updates and consistent weight updates across equivalent factorizations. A simplified scalar variant (RefLoRA-S) is also derived. The paper reports experiments on matrix factorization, GLUE, commonsense reasoning, and DreamBooth, together with convergence and complexity comparisons.

Significance. If the theoretical derivation were fully valid for the regime in which the method is deployed, RefLoRA would be a valuable contribution: it provides a closed-form, per-step optimal factorization with O((m+n+r)r^2) extra time and O(r^2) memory, and it connects low-rank fine-tuning to Riemannian optimization on a quotient manifold. The paper is commendably self-contained: proofs are given in the appendix, the method is described by pseudocode, and a public code repository is provided. The main caveat is that the implemented algorithm uses the balanced update S_t = S̃_t universally, while the theorem certifies this choice as the exact minimizer only in a parameter regime that the paper does not show is the operating regime for typical fine-tuning learning rates. This gap affects the central claim that RefLoRA 'identifies the optimal low-rank factorization per step.'

major comments (3)
  1. [§3.2–§3.3, Theorem 3 and Algorithm 1] The implemented method does not match the theorem's optimality claim in the likely operating regime. Theorem 3 states that S_t^* = S̃_t is the exact minimizer of (8) only when η ≥ 1/(C̃_t L) or η < 0; for 0 < η < 1/(C̃_t L), the minimizers are scaled matrices c_± S̃_t with c_± = (C̃_t L η)^{-1} ± sqrt((C̃_t L η)^{-2} - 1). Algorithm 1 and the 'balanced update is adopted thereafter' in §3.3 always use S_t = S̃_t. For typical fine-tuning learning rates such as η = 10^{-4}, the small-η branch applies unless C̃_t L < 10^4, which the paper never quantifies. In that branch, S̃_t does not minimize (8); the paper's assertion that 'this adjustment only affects the region where η is tiny' is unquantified because C̃_t and L are not estimated for the experiments. Since the abstract claims RefLoRA 'identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss,' the theoretical claim does not currently cover the method as run. Please either estimate C̃_t and L for the experimental setups, implement the scaled S_t^* in the small-η branch, or revise the optimality claim to describe the balanced update as a heuristic that is optimal only in the large-η regime.
  2. [§3.2, Eq. (7) and Appendix B.2] The neglect of the O(Lη^3) term in the upper bound is not justified by the cited smallness of η alone. The derivation bounds this term by η^3 times ||∇ℓ(W_t)||_2^3 times Frobenius norms of A_t S_t A_t^T and B_t S_t^{-1} B_t^T (Eq. (19) in the appendix). The paper states that η ~ O(10^{-4}) and ||∇ℓ(W_t)||_2 = O(10^{-1}) make the term negligible, but the term also scales with L, which appears in the main retained term as Lη^2||∇ℓ||^2. No bound on L or on the product L||∇ℓ|| is provided, and L is precisely the quantity that also determines which branch of Theorem 3 applies. Without an estimate of L, the truncation in (7) is an unverified approximation, and a central step in deriving the RefLoRA objective (8) rests on an unquantified assumption.
  3. [§4.2–§4.3, Tables 2 and 3] The empirical claim that RefLoRA 'outperforms various benchmarks' is not supported by statistical evidence. Although Table 2 states that results are averaged over 5 random runs, no standard deviations, confidence intervals, or significance tests are reported anywhere in the main text or appendix. Several reported differences are small (e.g., the GLUE average of 89.52 for RefLoRA versus 88.98 for DoRA, or 89.46 for AdaLoRA), and the commonsense reasoning tables report single numbers without variance. Without a measure of variability, it is impossible to judge whether the observed gains reflect the method or run-to-run noise, especially for differences at the 0.1–0.5 point level. Please report error bars or at least state the number of seeds and provide a variance estimate for the main comparisons, or temper the claim of consistent superiority.
minor comments (5)
  1. [Appendix C and Assumption 1] The paper acknowledges that standard LoRA initialization has B_0 = 0, which violates the full-rank Assumption 1 at t = 0, and suggests either full-rank initialization or a warm-up. However, it is not stated which of these is used in the LLM experiments in Appendix D.6; please clarify this in the experimental setup.
  2. [Table 2 caption] The caption says 'The best results are depicted in solid lines,' but the table contains no lines; presumably the intended word is 'bold.' Please fix the caption.
  3. [Appendix D.5] The model is referred to as 'GPT3-turbo' in one place and 'ChatGPT-3.5-turbo' in Table 3; please use a consistent name.
  4. [Checklist, Limitations] The checklist answer for Limitations says limitations are acknowledged in Section 5 and Appendix A, but the outlined future directions do not mention the main theoretical limitation identified in my major comment, namely the small-η regime in Theorem 3. Please add an explicit limitation statement.
  5. [§4.2, text before Table 2] The text states RefLoRA and RefLoRA-S 'outperform all competitors on 5 out of 8 datasets,' but the table does not make it immediately transparent which 5 datasets this refers to, since some datasets use multiple metrics (e.g., QQP Acc/F1). Please specify the comparison rule explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the derivation is self-contained, with only a regime-mismatch overclaim that is not a circular reduction.

full rationale

The derivation chain is self-contained. RefLoRA's optimal S_t is obtained by minimizing the explicit loss upper bound in (7), which is relaxed to the closed-form objective (8); the proof of Theorem 3 solves the stationarity condition (21) directly and identifies S̃_t as the minimizer in the large-η branch and scaled S̃_t in the small-η branch. No parameter is fitted to target accuracies or losses, and the experimental hyperparameters are chosen by standard validation rather than by the theory. Citations to the authors' own prior work ([30], [31]) appear only in related-work context or as baseline descriptions and are not load-bearing premises. The one notable gap is that Algorithm 1 and Section 3.3 always adopt the balanced update S_t = S̃_t, whereas Theorem 3 certifies S̃_t as the minimizer of (8) only when η ≥ 1/(C̃_t L) or η < 0; in the small-η branch the exact minimizer is a scaled version of S̃_t. This is a scope/overclaim issue about which branch applies in practice, not a circular construction: the implemented rule is an explicit simplification, and the theory's inputs do not already contain the theorem's conclusion.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The theoretical method introduces no new free parameters or postulated entities. The derivation depends on two standard assumptions (full-rank factorization, Lipschitz smoothness) and two pragmatic choices that are not fully justified theoretically: neglecting O(Lη^3) and using the balanced S̃_t for all η. The only tuned numbers are standard learning rates and ranks from the experimental setup.

free parameters (2)
  • Per-dataset learning rate η = e.g., 4e-4 (MNLI, MRPC, QNLI), 1e-3 (SST-2, CoLA, QQP, STS-B), 8e-4 (RTE); 8e-5 to 3e-4 for LLaMA models (Tables 7 and…
    Selected by grid search on validation data following prior work. Standard hyperparameter tuning; does not enter the theoretical derivation.
  • LoRA rank r and scaling α = r=8 for GLUE, r=16/32 for LLaMA, r=4 for DreamBooth; α=8 or 2r
    Set following prior work (LoRA, DoRA, LLM-Adapters). Not fitted to the target results.
assumptions (4)
  • domain assumption Assumption 1: rank(A_t) = rank(B_t) = r for all t > 0.
    Used in Lemma 1 to characterize all equivalent factorizations and in Theorem 3 to ensure S̃_t is SPD. LoRA's default initialization sets B_0 = 0, so the assumption fails at t=0; the authors apply RefLoRA only for t>0 or suggest warm-up.
  • standard math Assumption 2: the loss ℓ has L-Lipschitz gradient w.r.t. W.
    Standard smoothness assumption used to derive the quadratic upper bound (6) and Proposition 2. The paper notes that smoothness w.r.t. A and B is not assumed.
  • ad hoc to paper The O(Lη^3) term in (7) is negligible in practice.
    Proposition 2 bounds the cross term by O(Lη^3) and drops it, citing typical η ~ 10^{-4} and ∥∇ℓ∥ ~ 10^{-1}. This is an empirical magnitude argument, not a theorem.
  • ad hoc to paper The balanced update S_t = S̃_t is adopted for all η.
    Section 3.3 chooses S̃_t instead of the full minimizer S*_t from Theorem 3 to avoid needing L. This changes the theoretical optimality guarantee when η is in the small-η branch of (10).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models." pith.science (2026). https://pith.science/paper/AAYZJP23

@misc{pith2026250518877,
  author       = {Pith},
  title        = {Pith review of: RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AAYZJP23}},
  note         = {Machine review of arXiv:2505.18877}
}
read the original abstract

Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due to inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adaptation (RefLoRA) method promotes a flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and commonsense reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of-the-art LoRA variants.

Figures

Figures reproduced from arXiv: 2505.18877 by the authors.

Figure 1
Figure 1. Visualization of loss ℓ(Wt + ∆W˜ t) and upper bound (7). LoRA corresponds to St = Ir, while our refactoring (ref.) optimizes St [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparison of LoRA, ScaledGD, and RefLoRA for matrix factorization [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Images generated from Stable Diffusion fine-tuned with different approaches. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Convergence and complexity comparison 88.5% and 98.7% throughput compared to LoRA, at the additional memory cost of 132MB and < 1MB. In contrast, the throughput of LoRA-Pro and LoRA-RITE are 60.2% and 72.6% of LoRA, requiring 134MB and 140MB extra memory. This is consi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 39 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    A convergence analysis of gradient descent for deep linear neural networks

    Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. InProc. Int. Conf. on Machine Learning (ICML), 2018

    work page 2018

  3. [3]

    Nonlinear programming.Journal of the Operational Research Society, 48(3):334–334, 1997

    Dimitri P Bertsekas. Nonlinear programming.Journal of the Operational Research Society, 48(3):334–334, 1997

    1997

  4. [4]

    Piqa: Reasoning about physical common- sense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. InProc. AAAI Conf. Artif. Intel., pages 7432–7439, 2020

    work page 2020

  5. [5]

    Cambridge University Press, 2023

    Nicolas Boumal.An introduction to optimization on smooth manifolds. Cambridge University Press, 2023

    2023

  6. [6]

    Cambridge university press, 2004

    Stephen P Boyd and Lieven Vandenberghe.Convex optimization. Cambridge university press, 2004

    2004

  7. [7]

    SemEval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation

    Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. InProc. Int. Workshop Semant. Eval., pages 1–14. ACL, 2017

    work page 2017

  8. [8]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

Show all 71 references
  1. [9]

    On the measure of intelligence.arXiv:1911.01547, 2019

    François Chollet. On the measure of intelligence.arXiv:1911.01547, 2019

    1911 arXiv

  2. [10]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computatio...

    2019

  3. [11]

    Jan Maire, Leiden, 1637

    René Descartes.La Géométrie. Jan Maire, Leiden, 1637

  4. [12]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. InProc. Neural Information Processing Systems (NeurIPS), volume 36, pages 10088– 10115, 2023

    2023

  5. [13]

    Automatically constructing a corpus of sentential paraphrases

    Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. InProc. Int. Workshop Paraphrasing, 2005

    2005

  6. [14]

    Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced

    Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. InProc. Neural Information Processing Systems (NeurIPS), volume 31, 2018

    2018

  7. [15]

    Parameter- efficient fine-tuning with discrete fourier transform

    Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform. InProc. Int. Conf. on Machine Learning (ICML), 2024

    2024

  8. [16]

    MIT press Cambridge, 2016

    Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.Deep learning, volume 1. MIT press Cambridge, 2016

    2016

  9. [17]

    The Llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Parameter-efficient transfer learning with diff pruning

    Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020

    2012 arXiv

  11. [19]

    FLORA: Low-rank adapters are secretly gradient compres- sors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. FLORA: Low-rank adapters are secretly gradient compres- sors. InProc. Int. Conf. on Machine Learning (ICML), 2024. 11

    2024

  12. [20]

    LoRA+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Proc. Int. Conf. on Machine Learning (ICML), volume 235, pages 17783–17806, 21–27 Jul 2024

    2024

  13. [21]

    DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. InProc. Int. Conf. on Learning Representations (ICLR), 2023

    2023

  14. [22]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. InProc. Int. Conf. on Machine Learning (ICML), volume 97, pages 2790–2799, 09–15 Jun 2019

    2019

  15. [23]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InProc. Int. Conf. on Learning Representations (ICLR), 2022

    2022

  16. [24]

    LLM-Adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. LLM-Adapters: An adapter family for parameter-efficient fine-tuning of large language models. InProc. Conf. on Empirical Methods in Natural Language Processing (...

    2023

  17. [25]

    FedPara: Low-rank hadamard product for communication-efficient federated learning

    Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. FedPara: Low-rank hadamard product for communication-efficient federated learning. InProc. Int. Conf. on Learning Representations (ICLR), 2022

    2022

  18. [26]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProc. Int. Conf. on Learning Representations (ICLR), 2015

    2015

  19. [27]

    Quantum-peft: Ultra parameter-efficient fine-tuning.arXiv preprint arXiv:2503.05431, 2025

    Toshiaki Koike-Akino, Francesco Tonin, Yongtao Wu, Frank Zhengqing Wu, Leyla Naz Candogan, and V olkan Cevher. Quantum-peft: Ultra parameter-efficient fine-tuning.arXiv preprint arXiv:2503.05431, 2025

    2025 arXiv

  20. [28]

    VeRA: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. InProc. Int. Conf. on Learning Representations (ICLR), 2024

    2024

  21. [29]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059, November 2021

    2021

  22. [30]

    Implicit regularization of sharpness-aware minimization for scale-invariant problems

    Bingcong Li, Liang Zhang, and Niao He. Implicit regularization of sharpness-aware minimization for scale-invariant problems. InProc. Neural Information Processing Systems (NeurIPS), volume 37, pages 44444–44478, 2024

    2024

  23. [31]

    On the crucial role of initialization for matrix factorization

    Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the crucial role of initialization for matrix factorization. InProc. Int. Conf. on Learning Representations (ICLR), 2025

    2025

  24. [32]

    Geometric means.Linear algebra and its applications, 385:305–334, 2004

    Chi-Kwong Li and Roy Mathias. Geometric means.Linear algebra and its applications, 385:305–334, 2004

    2004

  25. [33]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. InProc. Conf. Assoc. Comput. Linguist. Meet. (ACL), pages 4582–4597, August 2021

    2021

  26. [34]

    LoftQ: LoRA-fine-tuning-aware quantization for large language models

    Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoftQ: LoRA-fine-tuning-aware quantization for large language models. InProc. Int. Conf. on Learning Representations (ICLR), 2024

    2024

  27. [35]

    ReLoRA: High-rank training through low-rank updates

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReLoRA: High-rank training through low-rank updates. InProc. Int. Conf. on Learning Representations (ICLR), 2024

    2024

  28. [36]

    Exploring versatile generative language model via parameter-efficient transfer learning

    Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning. InProc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 441–459, November 2020

    2020

  29. [37]

    Svft: Parameter-efficient fine-tuning with singular vectors.arXiv:2405.19597, 2024

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-efficient fine-tuning with singular vectors.arXiv:2405.19597, 2024

    2024 arXiv

  30. [38]

    Parameter-efficient orthogonal finetuning via butterfly factorization.arXiv preprint arXiv:2311.06243, 2023

    Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization.arXiv preprint arXiv:2311.06243, 2023. 12

    2023 arXiv

  31. [39]

    Cola: Compute-efficient pre-training of llms via low-rank activation.arXiv preprint arXiv:2502.10940, 2025

    Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Zi Yang, Paul Hovland, Bogdan Nicolae, Franck Cappello, and Zheng Zhang. Cola: Compute-efficient pre-training of llms via low-rank activation.arXiv preprint arXiv:2502.10940, 2025

    2025

  32. [40]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InProc. Int. Conf. on Learning Representations (ICLR), 2019

    2019

  33. [41]

    Pissa: Principal singular values and singular vectors adap- tation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adap- tation of large language models. InProc. Neural Information Processing Systems (NeurIPS), volume 37, pages 121038–121072, 2024

    2024

  34. [42]

    Can a suit of armor conduct electricity? A new dataset for open book question answering.arXiv:1809.02789, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering.arXiv:1809.02789, 2018

    2018 arXiv

  35. [43]

    Learning on loras: Gl-equivariant processing of low-rank weight spaces for large finetuned models.arXiv:2410.04207, 2024

    Theo Putterman, Derek Lim, Yoav Gelberg, Stefanie Jegelka, and Haggai Maron. Learning on loras: Gl-equivariant processing of low-rank weight spaces for large finetuned models.arXiv:2410.04207, 2024

    2024 arXiv

  36. [44]

    Know what you don’t know: Unanswerable questions for SQuAD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. InProc. Conf. Assoc. Comput. Linguist. Meet. (ACL), pages 784–789, 2018

    2018

  37. [45]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProc. Conf. Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    2022

  38. [46]

    AdapterDrop: On the efficiency of adapters in transformers

    Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. InProc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 7930–7946, November 2021

    2021

  39. [47]

    McGraw-Hill, New York, 3rd edition, 1976

    Walter Rudin.Principles of Mathematical Analysis. McGraw-Hill, New York, 3rd edition, 1976

    1976

  40. [48]

    Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. InProc. Conf. Computer Vision and Pattern Recognition (CVPR), pages 22500–22510, June 2023

    2023

  41. [49]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  42. [50]

    Socialiqa: Commonsense reasoning about social interactions.arXiv:1904.09728, 2019

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions.arXiv:1904.09728, 2019

    1904 arXiv

  43. [51]

    Ge- oloRA: Geometric integration for parameter efficient fine-tuning

    Steffen Schotthöfer, Emanuele Zangrando, Gianluca Ceruti, Francesco Tudisco, and Jonas Kusch. Ge- oloRA: Geometric integration for parameter efficient fine-tuning. InProc. Int. Conf. on Learning Repre- sentations (ICLR), 2025

    2025

  44. [52]

    Cambridge university press, 2014

    Shai Shalev-Shwartz and Shai Ben-David.Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  45. [53]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 16...

    2013

  46. [54]

    Training neural networks with fixed sparse masks

    Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. InProc. Neural Information Processing Systems (NeurIPS), volume 34, pages 24193–24205, 2021

    2021

  47. [55]

    Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022

    2022 arXiv

  48. [56]

    Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent.J

    Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent.J. Mach. Learn. Res., 22(150):1–63, 2021

    2021

  49. [57]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  50. [58]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 13

    2023 arXiv

  51. [59]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. InProc. Int. Conf. on Learning Representations (ICLR), 2019

    2019

  52. [60]

    Lora-ga: Low-rank adaptation with gradient approximation

    Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. In Proc. Neural Information Processing Systems (NeurIPS), volume 37, pages 54905–54931, 2024

    2024

  53. [61]

    LoRA-pro: Are low-rank adapters properly optimized? InProc

    Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-pro: Are low-rank adapters properly optimized? InProc. Int. Conf. on Learning Representations (ICLR), 2025

    2025

  54. [62]

    Neural network acceptability judgments.Trans

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments.Trans. Assoc. Comput. Linguist., 7:625–641, 2019

    2019

  55. [63]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. InProc. Conf. North Am. Chapter Assoc. Comput. Linguist., pages 1112–1122, 2018

    2018

  56. [64]

    Reft: Representation finetuning for language models

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. InProc. Neural Information Processing Systems (NeurIPS), volume 37, pages 63908–63962, 2024

    2024

  57. [65]

    DoRA: Weight-decomposed low-rank adaptation

    Shih yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. InProc. Int. Conf. on Machine Learning (ICML), 2024

    2024

  58. [66]

    Navigating text-to-image customization: From LyCORIS fine-tuning to model evaluation

    SHIH-YING YEH, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From LyCORIS fine-tuning to model evaluation. InProc. Int. Conf. on Learning Representations (ICLR), 2024

    2024

  59. [67]

    LoRA done RITE: Robust invariant transformation equilibration for loRA optimization

    Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. LoRA done RITE: Robust invariant transformation equilibration for loRA optimization. InProc. Int. Conf. on Learning Representations (ICLR), 2025

    2025

  60. [68]

    Hellaswag: Can a machine really finish your sentence?arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv:1905.07830, 2019

    1905 arXiv

  61. [69]

    Riemannian preconditioned LoRA for fine-tuning foundation models

    Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned LoRA for fine-tuning foundation models. InProc. Int. Conf. on Machine Learning (ICML), 2024

    2024

  62. [70]

    Limitations

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. InProc. Int. Conf. on Learning Representations (ICLR), 2023. 14 A Additional related work LoRA variants.LoRA has b...

    2023

  63. [71]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.