Pith. sign in

REVIEW 4 major objections 4 minor 49 references

LoRMA: Low-Rank Multiplicative Adaptation for LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that multiplying a pretrained weight matrix by a learned low-rank matrix can replace additive updates in fine-tuning, matching LoRA's accuracy with faster convergence.

desk verdict A genuinely new rank-inflation trick (LoRMApi) wrapped in an overclaimed multiplicative-update framing; LoRMA+ is essentially LoRA in disguise, and the faster-training claim does not survive a wall-clock check. read the letter →

arxiv 2506.07621 v1 pith:M73G7TVK submitted 2025-06-09 cs.CL cs.AIcs.LG

Harsh Bihany , Shubham Patel , Ashutosh Modi This is my paper
classification cs.CLcs.AIcs.LG
keywords parameter-efficientfine-tuninglow-rankadaptationLoRAmultiplicativeupdaterankinflationlargelanguagemodelsfasterconvergencemathreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LoRMA, a fine-tuning method that updates a frozen weight matrix $W_0$ by multiplying it on the left by a learned low-rank matrix, $W = \frac{\alpha}{r}(BA)W_0$, rather than adding a low-rank update as LoRA does. It proves that such multiplicative updates can in principle realize any desired transformation when $W_0$ has full column rank, and it introduces two rank-inflation mechanisms, $I^+$ and $I^\pi$, to keep the update from being crushed to rank $r$. Across GLUE, NLG, and math-reasoning tasks on RoBERTa, GPT-2, Gemma-2B, and LLaMA-3-8B, LoRMA+ and LoRMA$^\pi$ match or exceed LoRA accuracy while lowering the area under the training-loss curve, meaning they converge faster. If the claim holds, multiplicative adaptation is a drop-in alternative to additive LoRA that trains faster, merges into the weights for inference, and extends to existing LoRA variants such as Q-LoRA, AutoLoRA, DyLoRA, and DoRA.

What carries the argument

The load-bearing object is the multiplicative reparameterization $W = I(BA)\,W_0$, where $B \in \mathbb{R}^{d\times r}$ and $A \in \mathbb{R}^{r\times d}$ are low-rank factors and $I$ is a rank-inflation operator: $I^+(BA) = \frac{\alpha}{r}BA + I_d$ for the additive variant, and $I^\pi(BA)$ for the permutation variant that cyclically rotates each row of $BA$ by its row index. The existence theorem constructs $M_A = M W_0^+$, using the left inverse of a full-column-rank matrix, to show that premultiplication can reach any target; the inflation operators are what keep the transformation full-rank despite the inequality $\mathrm{rank}(BA) \le r$. The machinery also includes a re-ordering of matrix multiplications so that LoRMA's training-time complexity stays $O(dkb)$, the same order as LoRA.

What would settle it

For a RoBERTa layer, compute the numerical rank and smallest singular value of attention.self.query, then fine-tune the same task with LoRA and LoRMA on a version of the model whose query matrices are artificially replaced by rank-deficient matrices; if LoRMA's accuracy falls relative to LoRA as the rank deficiency grows, the invertibility assumption is load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that the additive low-rank update rule $W = W_0 + \frac{\alpha}{r}BA$ can be replaced by a multiplicative rule $W = I(BA)\,W_0$, where $I$ is one of two rank-inflating operations, without sacrificing accuracy and with faster convergence. Theorem 1 shows that whenever the frozen matrix $W_0 \in \mathbb{R}^{n\times m}$ has full column rank, every target matrix $M$ can be written as $M = M_A W_0$ by taking $M_A = M W_0^+$, so a premultiplicative transformation is in principle as expressive as addition. The two instantiations are LoRMA$^+$, which sets $I(BA) = \frac{\alpha}{r}BA + I_d$ and starts from $W = W_0$, and LoRMA$^\pi$, which applies a cyclic row-permutation to $BA$ and is initialized so that $I^\pi(BA) = I_d$. Empirically both variants match or beat LoRA on GLUE, E2E, and math benchmarks, and the $\pi$ variant produces near-full-rank updates (rank 1021 at $r = 8$), which is where the claim of a richer update space concretely lives.

Load-bearing premise

The multiplicative update can only be guaranteed to reach any target if the frozen weight matrices being adapted are full rank or very nearly so.

Editorial extensions

If this is right

  • Any task currently fine-tuned with LoRA can in principle be adapted with LoRMA, because premultiplication can realize any target update when the frozen matrix is full column rank.
  • LoRMA+ and LoRMA$^\pi$ should reach the same or better GLUE, NLG, and math accuracy in fewer training steps, since the reported training-loss AUC is 10.8\% to 52.0\% lower than LoRA on the tasks tested.
  • The final form $W_{\text{fine-tuned}} = I(BA)\,W_0$ can be merged into the frozen weights, so the method adds no inference-time latency.
  • Rank inflation is essential, not cosmetic: without it, LoRMA's performance collapses (for example, MRPC drops from 92.9 to 81.2).
  • Reverting a LoRMA-tuned model to the original weights requires $I(BA)$ to be invertible; when it is not, a copy of the original model parameters must be kept.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The existence theorem guarantees expressivity for a full multiplicand matrix, not for the rank-$r$ product $BA$ itself, so the proof motivates the multiplicative form but does not by itself explain why a rank-$r$ multiplier suffices; the experiments carry that part of the argument.
  • LoRMA+ produces updates constrained to rank $r$, exactly like LoRA, so its gains are better attributed to initialization and optimization geometry than to a fundamentally richer update space; the near-full-rank advantage belongs to LoRMA$^\pi$.
  • A testable extension is to apply multiplicative adaptation to models with intentionally low-rank or quantized frozen weights: if the invertibility premise is what makes LoRMA work, its advantage should shrink as those matrices become rank-deficient.
  • LoRMA$^\pi$ could be combined with rank-allocation schemes or singular-vector parameterizations to get an expressive adapter with a tight parameter budget; the paper notes the approach is orthogonal to LoRA variants but does not test these combinations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LoRMA, a parameter-efficient fine-tuning method that replaces LoRA's additive low-rank update with a multiplicative transformation of the frozen weight matrix, W = I(BA) W0, where I is either an identity-addition operator (LoRMA+) or a row-permutation operator (LoRMAπ). The authors prove an existence theorem for multiplicative transformations under an invertibility assumption, introduce two rank-inflation strategies to overcome the rank bottleneck of low-rank products, and evaluate the methods on RoBERTa, GPT-2, Gemma-2B, and LLaMA-3-8B on GLUE, E2E, WebNLG, DART, GSM-8K, and MATH. The paper claims competitive or better accuracy than LoRA, faster convergence, and a richer parameter-exploration space.

Significance. If the central claims are correct, the work would make a useful contribution to PEFT by offering an alternative multiplicative update family with a genuinely higher-rank update for the permutation variant. The paper is commendable for its extensive hyperparameter reporting, code release, rank-monitoring ablations, and comparisons across multiple model families and task types. The existence theorem is a standard linear-algebra fact and is clearly proved. However, the primary advertised advantage, a 'richer space of matrix multiplicative transformations,' is not supported for the LoRMA+ variant because its update has rank at most r (Table 15), identical to LoRA. The genuinely higher-rank behavior is specific to LoRMAπ, which shows slightly lower average accuracy than LoRA in the reported GLUE results. The faster-convergence claim is also weakened by the absence of wall-clock or FLOP-normalized measurements, especially given the approximately 2× per-step cost shown in Table 11.

major comments (4)
  1. [§3.2.2, Eq. (8) and Table 15] The central claim of a 'richer space of matrix multiplicative transformations' is not substantiated for LoRMA+. In Eq. (8), the forward pass is h = (α/r · BA + I_d) W0 x, so the trainable update is ΔW = (α/r · BA) W0. Since W0 is frozen and BA has rank at most r, ΔW has rank at most r, and Table 15 reports rank 8 for r = 8, exactly the same constraint as LoRA. For invertible W0 the update space spanned by LoRMA+ is the same as that of LoRA up to a fixed invertible linear transformation, i.e., a reparameterization rather than an expressive superset. The richer-space claim applies only to LoRMAπ. The paper should either restrict the claim to LoRMAπ or provide concrete evidence that LoRMA+ explores a different or larger effective space than LoRA.
  2. [Remark after Corollary 1.2] The existence result (Theorem 1 and Corollary 1.2) and the equivalence argument for LoRMA+ rely on the adapted weight matrices being invertible or close to full rank. The Remark states that the authors verified the matrices are 'full rank or close to full rank, typically within 99% of the maximum possible rank,' but no ranks, condition numbers, or thresholds are reported anywhere in the manuscript or appendix. If any of the adapted matrices (e.g., attention.self.query in RoBERTa or c_attn in GPT-2) is significantly rank-deficient, Corollary 1.2 does not apply, and LoRMA+ may span a strictly smaller update space than LoRA. The verification data must be provided.
  3. [§5.1 and Table 11 (Appendix A)] The claim that faster step-count convergence implies 'lower training time' (Introduction, §5) is unsupported. Fig. 6 and Table 5 report speedups measured in training steps, but Table 11 shows that LoRMA+ costs approximately 2dkb + 2drb + db per step versus LoRA's dkb + krb + drb + db, i.e., roughly 2× for square matrices with r ≪ d. A 1.5× reduction in steps at a 2× per-step FLOP cost does not yield wall-clock savings; no wall-clock measurements or FLOP-normalized convergence curves are provided. The Limitations section flags only the Iπ time overhead, not the I+ constant-factor overhead, so the computational-cost discussion is internally inconsistent. The authors should either report wall-clock times and FLOP-normalized curves or remove the 'lower training time' claim.
  4. [§5.1 and Table 5] The faster-convergence evidence is limited to two GLUE tasks (SST-2 and CoLA) using one metric (AUC of the training-loss curve). This is too narrow to support the general statement that LoRMA has 'much faster convergence' across all reported experiments (GLUE, NLG, math). The authors should provide per-task convergence measures for the full benchmark suite, or explicitly confine the convergence claim to the measured tasks.
minor comments (4)
  1. [Appendix C, Table 13] On DART and WebNLG, LoRMA+ underperforms LoRA on BLEU (43.64 vs 45.35 and 49.98 vs 52.27). The main text describes the results as 'competitive' and 'at par with LoRA'; a direct discussion of these metrics would improve accuracy of the claims.
  2. [Table 17 (Appendix D.4)] The hyperparameter table lists the LLaMA-3-8B learning rate as '5E4', which is presumably a typo for '5E-4' as given in the text of Appendix D.4. Please correct.
  3. [Figure 6] The speedup factors ('2.9x Speedup (Step 1460)' and '1.5x Speedup (Step 2760)') are not defined. State the loss threshold or criterion used to determine the step at which each method reaches a given loss value.
  4. [§3.2.1] The cyclic row-shift permutation Iπ is described verbally ('the i-th row is rotated by i'), but the direction of rotation (left or right) and the handling of non-square matrices are not formally specified. A precise matrix definition would remove ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is a proved linear-algebra fact, and all central claims are empirical evaluations rather than predictions derived from fitted inputs.

full rationale

LoRMA's derivation chain is self-contained. Theorem 1 (Section 3.1) is a standard linear-algebra existence result with a constructive proof (MA = M M0^+); it is not derived from LoRMA's update rule, and it is used only as motivation for considering multiplicative updates, not as a prediction of performance. The method definitions (Equations 6-9) are proposals, not consequences of the theorem. Rank-inflation claims are empirically verified in Section 5.2 and Appendices E.2-E.3, including the paper's own Table 15 acknowledging that LoRMA+ has rank-8 updates identical in expressivity to LoRA; the richer-parameter-space benefit is attributed to LoRMAπ, where the update is near-full-rank. The faster-convergence claim is measured post hoc via AUC of training-loss curves (Table 5, Figure 6), so it is an empirical observation rather than a circular prediction. No fitted parameter is renamed as a prediction, and no load-bearing argument reduces to a self-citation. Concerns about wall-clock speed and the unverified invertibility assumption are correctness and rigor issues, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method's central claim rests on two hand-chosen hyperparameters (rank r and scale alpha), a domain assumption about full-rank pretrained matrices, and an unproved claim that cyclic-shift permutation generically inflates rank. No new physical or categorical entities are postulated.

free parameters (2)
  • Scaling factor alpha = 4 or 8 per RoBERTa task (Table 14); 32 for GPT-2 (I+) and 8 (Ipi); not explicitly reported for Gemma/Llama
    Chosen by hand per task to control update magnitude; not derived from theory.
  • Rank r = 8 (RoBERTa base/large), 4 (GPT-2), 4 (Gemma-2B), 8 (LLaMA-3-8B)
    Set to match parameter budgets of baselines; varying r is analyzed in Section 5.5.
assumptions (3)
  • standard math Rank properties: R(M1 * M2) <= min(R(M1), R(M2)) and R(M1 + M2) >= |R(M1) - R(M2)|
    Used in Section 3.1 to motivate the rank bottleneck and inflation; standard linear algebra (Strang, 2009), no additional verification needed.
  • domain assumption The adapted pretrained weight matrices (e.g., attention query, c_attn) are non-singular or close to full rank (within 99% of maximum rank)
    Stated in the Remark after Corollary 1.2; required for the existence proof and for LoRMA+ to cover the same update space as LoRA; verification data not shown in the paper.
  • ad hoc to paper The cyclic row-shift permutation Ipi on a rank-r product BA yields a full-rank matrix in general
    Demonstrated on a 3x3 example (Fig. 3) and empirically on 1024x1024 matrices (Appendix E.2), but no general proof is given; the claim that Ipi(BA) is almost full rank throughout training is used to justify LoRMApi's expressiveness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRMA: Low-Rank Multiplicative Adaptation for LLMs." pith.science (2026). https://pith.science/paper/M73G7TVK

@misc{pith2026250607621,
  author       = {Pith},
  title        = {Pith review of: LoRMA: Low-Rank Multiplicative Adaptation for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M73G7TVK}},
  note         = {Machine review of arXiv:2506.07621}
}
read the original abstract

Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.

Figures

Figures reproduced from arXiv: 2506.07621 by the authors.

Figure 1
Figure 1. Transformation of a vector W by two meth￾ods: one is via rotation and scaling, the other is via the addition of a vector v. to as PEFT (Parameter Efficient Fine Tuning)) has been proposed to address large computational costs associated with fine-tuning. Various PEFT techniques previously have been de￾vised (Han et al., 2024); however, they often in￾troduce trade-offs such as lack of parallelism, in￾creased inference… view at source ↗
Figure 2
Figure 2. Comparing LoRA (a) and LoRMA (b). @ denotes matrix multiplication. I+ and Iπ represent additive and permutation based rank inflation respectively (§3). In case of LoRMA, initialization of A and B depends on the type of inflation (§3). entire space of d × k matrices (Aghajanyan et al., 2021; Hu et al., 2022). The current LoRA-based approaches (Yang et al., 2024) have employed ad￾ditive transformations, where the low-… view at source ↗
Figure 3
Figure 3. Permutation-Based Inflation Iπ operation. Re￾arrange matrix entries to inflate the rank. matrices to capture the multiplicative transforma￾tion. 3.2.1 Permutation-Based Inflation (Iπ) Permutation-based rank inflation utilizes the idea of strategic re-arrangement of elements of the ma￾trices to increase the rank of a matrix. The rows of the matrix are rotated cyclically in incremental steps. The i th row is rotated b… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Impact on the resultant matrix on updating a [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparing performance across ranks for the GLUE tasks RTE, STSB, CoLA, MRPC for RoBERTa [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Train loss curves for CoLA: RoBERTabase for various techniques. Method MRPC STS-B RTE QQP LoRMA 81.2 15.6 52.7 83.0 LoRMA+ 92.9 92.2 86.6 91.3 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Variation of rank of resultant multiplicative [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Training and Evaluation Accuracy variation for LoRA, LoRMA [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 11 canonical work pages

  1. [1]

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. https://doi.org/10.18653/v1/2021.acl-long.568 Intrinsic dimensionality explains the effectiveness of language model fine-tuning . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

    doi:10.18653/v1/2021.acl-long.568 2021

  2. [2]

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. https://doi.org/10.18653/v1/2022.acl-short.1 B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1--9, Dublin, Ireland. Association...

    doi:10.18653/v1/2022.acl-short.1 2022

  3. [3]

    Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. https://doi.org/10.18653/v1/s17-2001 Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation . In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics

    doi:10.18653/v1/s17-2001 2017

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. http://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems

    arXiv 2021

  5. [5]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

    2023

  6. [6]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. 2005. https://aclanthology.org/I05-5002/ Automatically constructing a corpus of sentential paraphrases . In Proceedings of the Third International Workshop on Paraphrasing ( IWP 2005)

    work page 2005

  7. [7]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.64 A survey on in-context learning . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107--1128, Miami, Florid...

    doi:10.18653/v1/2024.emnlp-main.64 2024

  8. [8]

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

Show all 49 references
  1. [9]

    Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. https://doi.org/10.18653/v1/W17-3518 The W eb NLG challenge: Generating text from RDF data . In Proceedings of the 10th International Conference on Natural Language Generation, pages 124--1...

    2017 doi

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [11]

    Demi Guo, Alexander Rush, and Yoon Kim. 2021. https://doi.org/10.18653/v1/2021.acl-long.378 Parameter-efficient transfer learning with diff pruning . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Con...

    2021 doi

  4. [12]

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. https://openreview.net/forum?id=lIsCS8b6zj Parameter-efficient fine-tuning for large models: A comprehensive survey . Transactions on Machine Learning Research

    2024

  5. [13]

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. https://openreview.net/forum?id=NEv8YqBROO Lo RA +: Efficient low rank adaptation of large models . In Forty-first International Conference on Machine Learning

    2024

  6. [14]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. http://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset

    2021 arXiv

  7. [15]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. https://proceedings.mlr.press/v97/houlsby19a.html Parameter-efficient transfer learning for NLP . In Proceedings of the 36th In...

    2019

  8. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  9. [17]

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2024. https://openreview.net/forum?id=NjNfLdxr3A Ve RA : Vector-based random matrix adaptation . In The Twelfth International Conference on Learning Representations

    2024

  10. [18]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix-tuning: Optimizing continuous prompts for generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021 doi

  11. [19]

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2024. https://openreview.net/forum?id=DLJznSp6X3 Relo RA : High-rank training through low-rank updates . In The Twelfth International Conference on Learning Representations

    2024

  12. [20]

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/48c368f105e8145b945227b73255635a-Paper-Confere...

    2024

  13. [21]

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353

    2024 arXiv

  14. [22]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach

    2019 arXiv

  15. [23]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  16. [24]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft

    2022

  17. [25]

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. 2025. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605

    2025

  18. [26]

    Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern ...

    2021 doi

  19. [27]

    Jekaterina Novikova, Ond r ej Du s ek, and Verena Rieser. 2017. https://doi.org/10.18653/v1/W17-5525 The E 2 E dataset: New challenges for end-to-end generation . In Proceedings of the 18th Annual SIG dial Meeting on Discourse and Dialogue , pages 201--206, Saarbr \"u cken, Ge...

    2017 doi

  20. [28]

    Aleksandar Petrov, Philip Torr, and Adel Bibi. 2024. https://openreview.net/forum?id=JewzobRhay When do prompting and prefix-tuning work? a theory of capabilities and limitations . In The Twelfth International Conference on Learning Representations

    2024

  21. [29]

    Adam Poliak. 2020. https://doi.org/10.18653/v1/2020.eval4nlp-1.10 A survey on recognizing textual entailment as an NLP evaluation . In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 92--109, Online. Association for Computational Linguistics

    2020 doi

  22. [30]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  23. [31]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don`t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 78...

    2018 doi

  24. [32]

    Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. 2019. http://arxiv.org/abs/1907.01041 Natural language understanding with the quora question pairs dataset

    2019 arXiv

  25. [33]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170/ Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Em...

    2013

  26. [34]

    Gilbert Strang. 2009. Introduction to Linear Algebra, fourth edition. Wellesley-Cambridge Press, Wellesley, MA

    2009

  27. [35]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  28. [36]

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/123fd8a56501194823c8e0dca00733df-Paper-Conference.pdf Hydralora: An asymmetric lora architecture for efficient fine-tuning . In Advances in Neural I...

    2024

  29. [37]

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023. https://doi.org/10.18653/v1/2023.eacl-main.239 D y L o RA : Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation . In Proceedings of the 17th Conference of the...

    2023 doi

  30. [38]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  31. [39]

    Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu. 2024. https://doi.org/10.18653/v1/2024.acl-long.156 PR o L o RA : Partial rotation empowers more parameter-efficient L o RA . In Proceedings of the 62nd Annual Meeting of the Associatio...

    2024 doi

  32. [40]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. https://doi.org/10.1162/tacl_a_00290 Neural network acceptability judgments . Transactions of the Association for Computational Linguistics, 7:625--641

    2019 doi

  33. [41]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  34. [42]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  35. [43]

    Menglin Yang, Jialin Chen, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Qianru Zhang, Min Zhou, Irwin King, et al. 2024. Low-rank adaptation for foundation models: A comprehensive review. arXiv preprint arXiv:2501.00365

    2024

  36. [44]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. https://openreview.net/forum?id=N8N0hgNDRt Metamath: Bootstrap your own mathematical questions for large language models . In The Twelfth Inte...

    2024

  37. [45]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. https://openreview.net/forum?id=lq62uWRJjiY Adaptive budget allocation for parameter-efficient fine-tuning . In The Eleventh International Conference on Learning Representations

    2023

  38. [46]

    Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. https://doi.org/10.18653/v1/2024.naacl-long.282 A uto L o RA : Automatically tuning matrix ranks in low-rank adaptation based on meta learning . In Proceedings of the 2024 Conference of the North American ...

    2024 doi

  39. [47]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  40. [48]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.