Pith.

Machine Learning · arXiv:2611.17191 v1 · submitted 12 Nov 2026 · 33 references

Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off

Tomohiro Sawada, Kartik Goyal

The paper says

“Low-Rank Adaptation (LoRA) often underperforms full fine-tuning, especially in heterogeneous and continual-learning settings in which substantial forgetting occurs. In general, it suffers from an undesirable learning-forgetting trade-off across diverse fine-tuning settings.”

more of the abstract

We identify a potential contributor: LoRA adapters, especially high-rank ones, frequently fail to realize the available capacity afforded by the configured rank. Specifically, we find that standard LoRA updates have effective rank far below their configured rank, limiting how broadly the update energy is distributed. Therefore, we propose training mechanisms that optimize the original objective while maximizing adapter effective rank. Our main method adds a simple, stable, differentiable, and SVD-free Rényi-2 effective-rank regularizer to the language-modeling loss. We also connect capacity maximization to norm-bounded updates. In mixed-domain rank sweeps on three models evaluated with multiple random seeds, a penalty weight of λ = 0.01 lowers mean Wiki perplexity at every tested rank of at least 16 relative to unpenalized LoRA. Mean training perplexity improves or increases by at most 0.02 in all but one model-rank comparison. Single-task and scheduling experiments further show that the balance between learning and retention depends on the configuration. We also observe that several LoRA variants in the literature also suffer from the identified capacity issue, and our approach provides a complementary axis of improvement to LoRA-based parameter-efficient fine-tuning. These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier. Our code is available at https://github.com/tomohiro-sawada/lora-capacity-utilization

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A solid, well-executed paper that introduces a simple, principled regularizer for LoRA and backs it with extensive experiments; the main weakness is that the causal mechanism (effective rank vs. norm control) is not fully isolated, but the authors acknowledge this and the empirical finding stands. This line is wrong

6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 33 references say.

6 minor objections

The paper says, at Section 4.1

Pith says: minor objection 1 of 6

The definition of 'Train PPL' should be clarified earlier in the main text. It is described as 'training-mixture perplexity' but is actually an exponential moving average of the training loss, not held-out validation perplexity. This distinction is important for interpreting the results and should be stated in the main text, not just in Appendix F.

The paper says, at Abstract and Section 4.4

Pith says: minor objection 2 of 6

The claim 'Mean training perplexity improves or increases by at most 0.02 in all but one model–rank comparison' is accurate, but the exception (Qwen2.5-3B-Instruct at r=16, where Train PPL increases by 0.313) should be explicitly mentioned for completeness.

show 4 more

The paper says, at Figure 2

Pith says: minor objection 3 of 6

The dense FT point is a single run. Consider using different markers for single-run vs multi-run means to improve clarity, as the current legend does not distinguish them visually.

The paper says, at Appendix F.1

Pith says: minor objection 4 of 6

The dense baseline hyperparameter selection uses training loss rather than held-out validation. While this does not affect the main LoRA-to-LoRA comparisons, it may not yield the best possible dense baseline. This should be noted as a caveat when interpreting dense comparisons.

The paper says, at Section 5, Table 4

Pith says: minor objection 5 of 6

The weighted unaligned mass for penalized LoRA is slightly higher than for standard LoRA (0.972-0.973 vs 0.960), yet the penalized adapters forget less. The paper notes this but does not explain the apparent contradiction. A brief discussion of why this might be the case would strengthen the analysis.

The paper says, at Section 4

Pith says: minor objection 6 of 6

The paper uses 'StarCoder PPL' and 'Wiki PPL' as forgetting probes from OLMo-Mix-1124. It would be helpful to explicitly state that these are not the actual pretraining corpora for the models used (Qwen, Llama) but are used as general pretraining distribution proxies.

What would settle it

Re-run the mixed-domain experiment on Qwen2.5-0.5B at r=896 with λ=0.01, but replace the Rényi-2 penalty with a penalty that directly maximizes the Frobenius norm ∥ΔW∥_F^2 while keeping the operator norm fixed (as in NB-LoRA but without the Stiefel–Cayley parameterization). If the resulting effective rank and retention metrics match those of the Rényi-2 penalty, then the benefit is attributable to norm control rather than spectral uniformity. If they diverge, the spectral uniformity objective is the active ingredient.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence moderate
  2. Skeptic agrees 48 s · 43,839 tokens read, 652 written
  3. Editor wrote the verdict above 10 s · 44,636 tokens read, 672 written
  4. Referee minor revision: 6 minor 40 s · 48,633 tokens read, 875 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Hey,

You should know about this paper: it introduces a Rényi-2 effective-rank penalty for LoRA training and shows, across a lot of experiments, that it reduces forgetting on pretraining proxies while keeping target-task fit competitive. The core idea is simple—add an SVD-free term that encourages the adapter to use its allocated rank more evenly—and the paper does a thorough job of validating it.

What's actually new: the observation that standard LoRA underutilizes its configured rank (effective rank far below nominal rank) is not new per se, but using effective rank as a trainable objective, specifically via the Rényi-2 surrogate, is. The paper provides a clean derivation, connects it to norm-bounded updates, and ships code. The experiments are extensive: multiple models (Qwen2.5, Llama-3.2, OLMo-2, Pythia), multiple ranks, multiple seeds, mixed-domain and single-task settings, and comparisons to several LoRA variants (PiSSA, LoRA-XS, VeRA, SVF, TinyLoRA). The results are consistent: at λ=0.01, Wiki perplexity drops at every rank ≥16 across three models, with minimal training perplexity cost. That's a real empirical finding.

The soft spots are real but not fatal. The main one is that the paper cannot separate effective-rank maximization from norm control—the norm-bounded route also raises effective rank, and the authors say so explicitly (Section 4.1). So the causal claim is ambiguous, but the paper doesn't overclaim; it says the method improves the frontier, not that it proves a mechanism. The dense baseline hyperparameter selection uses training loss rather than held-out validation, which is a minor methodological concern. The PiSSA exclusion (one run with apparent data replay) is handled transparently. The scheduling experiments are on a short budget and the mixed-domain results use early-start penalty, so the Train PPL cost at λ=0.1 might be reducible—the authors note this. The limitations section is honest.

Who this is for: anyone working on LoRA-based fine-tuning, especially in continual or multi-domain settings. The method is easy to implement and the paper gives clear guidance on hyperparameters. It deserves a serious referee; the experiments are reproducible and the claims are supported. I'd cite it if I were working on PEFT for LLMs.

Recommendation: accept with minor revisions. The main ask would be to add a small experiment or discussion that more directly tests the causal role of effective rank (e.g., a synthetic setting where rank is controlled independently of norm), but this is not a dealbreaker.

The strongest objection

The paper's central claim—that adding a Rényi-2 effective-rank penalty to the language-modeling loss reduces forgetting on pretraining proxies while maintaining competitive target-task fit—is supported by extensive, well-documented experiments across multiple models, ranks, tasks, and seeds. The authors transparently acknowledge the key ambiguity: the norm-bounded route also raises effective rank, and the experiments do not fully separate effective-rank maximization from norm control (Section 4.1, sentence b124). However, this is a limitation of the causal interpretation, not a flaw in the empirical finding. The paper does not claim to have proven a causal mechanism; it claims to have demonstrated a method that improves the learning–forgetting frontier, and that claim is backed by the data. The reader's weakest_assumption correctly identifies this ambiguity, but the paper's own discussion already covers it. No additional load-bearing concern beyond what the authors themselves flag is apparent. The method is simple, the code is released, the experiments are reproducible, and the results are consistent across settings. The paper's limitations section (b235) is honest and complete. Therefore, no significant objection is identified.

The reader's weakest_assumption is that the observed improvements might be caused by side effects of the penalty (norm regularization, altered gradient scales) rather than by the increase in effective rank. The paper itself acknowledges this in Section 4.1 (sentence b124): 'these experiments do not separate effective-rank maximization from norm control'. This is a genuine ambiguity, but it is a limitation of the causal interpretation, not a threat to the empirical claim. The paper's central claim is that the method improves the learning–forgetting frontier, not that it does so specifically via effective-rank maximization. The empirical evidence for the method's effectiveness is strong: multiple models, ranks, seeds, and tasks, with consistent results. The authors also provide a norm-bounded route (NB-LoRA) that achieves similar results, which further supports the robustness of the approach. The concrete test I propose would help resolve the causal ambiguity, but the paper's main claim does not depend on that resolution. Therefore, the verdict should remain ACCEPT (UNCHANGED). I agree with the reader's identification of the weakest assumption.

The referee's report

If the result holds, it provides a simple, practical, and theoretically motivated method to improve the learning-forgetting frontier in LoRA fine-tuning. The method is well-motivated by the observation of capacity underutilization, which is empirically demonstrated. The paper includes extensive experiments with multiple seeds, uncertainty estimates, and across multiple models and tasks, lending credibility to the findings. The code is released, enhancing reproducibility. The paper is transparent about limitations, including the ambiguity between effective-rank maximization and norm control. The work is significant for the parameter-efficient fine-tuning community as it offers a new axis for improvement orthogonal to existing methods.

Pith's reading of the paper (a machine's summary, not the authors')

Large language models are often fine-tuned with a technique called LoRA, which learns a small low-rank adjustment to the model's weights. The idea is that a low-rank update is enough to specialize the model to a new task while being cheap to train. However, this paper finds that even when you configure a high rank, the actual update often uses far fewer effective directions than allowed, meaning the model does not fully exploit its 'capacity budget'. The paper introduces a simple training penalty, based on the Rényi-2 entropy of the singular values of the update, that encourages the update to use its directions more evenly. This penalty is computed efficiently without a full singular-value decomposition, so it adds only a small overhead.

The authors test this penalty on a mix of math, translation, and instruction-following tasks, across several models and ranks. They report that with a small penalty weight, the model forgets less on pre-training-like text (measured by Wiki and StarCoder perplexity) while learning the target domain just as well or slightly better. They also show that the method works when applied to other LoRA variants like PiSSA and LoRA-XS, and that the balance between learning and retention depends on when the penalty is turned on.

In short, the paper turns a previously diagnostic measure, effective rank, into a training objective that can help mitigate the learning-forgetting trade-off in parameter-efficient fine-tuning.

Core claim. We propose effective-rank maximization via the Rényi-2 penalty, a parameterization-agnostic, SVD-free term added to the language-modeling loss, and showed across mixed-domain SFT, task-specific SFT, rank sweeps, and scheduling ablations that it can reduce forgetting on pretraining proxies while maintaining competitive target-task fit. Specifically, a penalty weight of λ=0.01 lowers mean Wiki perplexity at every tested rank of at least 16 relative to unpenalized LoRA.

The premise it rests on. The central premise that the observed improvements in the learning-forgetting trade-off are caused by the increase in effective rank, rather than by other side effects of the penalty such as norm regularization or altered gradient scales. The paper itself acknowledges (Section 4.1) that the norm-bounded route also raises effective rank and that 'these experiments do not separate effective-rank maximization from norm control', leaving the causal role of effective rank ambiguous.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Low-Rank Adaptation (LoRA) often underperforms full fine-tuning, especially in heterogeneous and continual-learning settings in which substantial forgetting occurs. In general, it suffers from an undesirable learning-forgetting trade-off across diverse fine-tuning settings. We identify a potential contributor: LoRA adapters, especially high-rank ones, frequently fail to realize the available capacity afforded by the configured rank. Specifically, we find that standard LoRA updates have effective rank far below their configured rank, limiting how broadly the update energy is distributed. Therefore, we propose training mechanisms that optimize the original objective while maximizing adapter effective rank. Our main method adds a simple, stable, differentiable, and SVD-free Rényi-2 effective-rank regularizer to the language-modeling loss. We also connect capacity maximization to norm-bounded updates. In mixed-domain rank sweeps on three models evaluated with multiple random seeds, a penalty weight of λ = 0.01 lowers mean Wiki perplexity at every tested rank of at least 16 relative to unpenalized LoRA. Mean training perplexity improves or increases by at most 0.02 in all but one model-rank comparison. Single-task and scheduling experiments further show that the balance between learning and retention depends on the configuration. We also observe that several LoRA variants in the literature also suffer from the identified capacity issue, and our approach provides a complementary axis of improvement to LoRA-based parameter-efficient fine-tuning. These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier. Our code is available at https://github.com/tomohiro-sawada/lora-capacity-utilization

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-14

The authors' main claim, as the classifier picked it out 1.00

“These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier.”

How the main claim is stated: hedged possibility 0.99

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.44 · human participants 0.02 · controlled comparison 0.93 · randomized assignment 0.22 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Low-Rank Adaptation (LoRA) often underperforms full fine-tuning, especially in heterogeneous and continual-learning settings in which substantial forgetting occurs. background 1.00 0.08 0.03 asserted 0.99
In general, it suffers from an undesirable learning-forgetting trade-off across diverse fine-tuning settings. background 1.00 0.10 0.04 asserted 1.00
We identify a potential contributor: LoRA adapters, especially high-rank ones, frequently fail to realize the available capacity afforded by the configured rank. result 0.51 0.72 0.11 hedged possibility 0.90
Specifically, we find that standard LoRA updates have effective rank far below their configured rank, limiting how broadly the update energy is distributed. result 0.97 0.94 0.07 asserted 1.00
Therefore, we propose training mechanisms that optimize the original objective while maximizing adapter effective rank. contribution 0.90 0.59 0.19 asserted 0.99
Our main method adds a simple, stable, differentiable, and SVD-free R\'enyi-2 effective-rank regularizer to the language-modeling loss. method 0.95 0.94 0.10 asserted 1.00
We also connect capacity maximization to norm-bounded updates. contribution 0.57 0.78 0.04 asserted 0.98
In mixed-domain rank sweeps on three models evaluated with multiple random seeds, a penalty weight of $\lambda=0.01$ lowers mean Wiki perplexity at every tested rank of at least $16$ relative to unpenalized LoRA. result 1.00 0.97 0.12 asserted 1.00
Mean training perplexity improves or increases by at most $0.02$ in all but one model-rank comparison. result 0.99 0.95 0.04 asserted 1.00
Single-task and scheduling experiments further show that the balance between learning and retention depends on the configuration. result 0.98 0.91 0.06 asserted 0.98
We also observe that several LoRA variants in the literature also suffer from the identified capacity issue, and our approach provides a complementary axis of improvement to LoRA-based parameter-efficient fine-tuning. interpretation 0.41 0.69 0.09 asserted 0.97
These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier. interpretation 1.00 0.69 0.81 hedged possibility 0.99
Our code is available at https://github.com/tomohiro-sawada/lora-capacity-utilization contribution 0.69 0.44 0.02 asserted 0.89

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-14: 1 request, 9,562 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. Effective-rank utilization of standard LoRA adapters across LoRA ranks, decomposed by depth. Each
Figure 2 of the paper
Figure 2. Dense FT denotes dense down-projection fine-tuning. Mixed-domain continual-learning baselines on
Figure 3 of the paper
Figure 3. LoRA rank sweep on Qwen2.5-0.5B on the mixed-domain continual-learning proxy setup (aggregated

Figures are the authors', reproduced from the paper.

References (33)
  1. [1] P.-A. Absil, Robert Mahony, and Rodolphe Sepulchre. 2009. https://doi.org/10.1515/9781400830244 Optimization Algorithms on Matrix Manifolds . Princeton University Press, Princeton on Pithdoi:10.1515/9781400830244
  2. [2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. https://arxiv.org/abs/2012.13255 Intrinsic dimensionality explains the effectiveness of language model fine-tuning . Preprint, arXiv:2012.13255 on PithPith's reviewarXiv
  3. [3] Lo \"i c Barrault, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias M \"u ller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. https://doi.org/10.1... on Pithdoi:10.18653/v1/w19-5301
  4. [4] Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. 2025. https://arxiv.org/abs/2405.17604 LoRA-XS : Low-rank adaptation with extremely small number of parameters . Preprint, arXiv:2405.17604 on PithPith's reviewarXiv
  5. [5] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. 2024. https://openreview.net/forum?id=aloEru2qCG LoRA learns less an... on Pith
  6. [6] Massimo Bini, Leander Girrbach, and Zeynep Akata. 2025. https://arxiv.org/abs/2503.18225 DeLoRA : Decoupling angles and strength in low-rank adaptation . Preprint, arXiv:2503.18225 on PithPith's reviewarXiv
  7. [7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 on PithPith's reviewarXiv
  8. [8] Thomas M. Cover and Joy A. Thomas. 2006. Elements of information theory, 2nd edition. Wiley-Interscience on Pith
  9. [9] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. 2020. https://arxiv.org/abs/1912.05671 Linear mode connectivity and the lottery ticket hypothesis . Preprint, arXiv:1912.05671 on PithPith's reviewarXiv
  10. [10] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 LoRA : Low-rank adaptation of large language models . In International Conference on Learning Representations on Pith
  11. [11] Damjan Kalajdzievski. 2023. https://arxiv.org/abs/2312.03732 A rank stabilization scaling factor for fine-tuning with LoRA . Preprint, arXiv:2312.03732 on PithPith's reviewarXiv
  12. [12] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. https://doi.org/10.1073/pnas.1... on Pithdoi:10.1073/pnas.1611835114
  13. [13] Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. 2024. https://arxiv.org/abs/2310.11454 VeRA : Vector-based random matrix adaptation . Preprint, arXiv:2310.11454 on PithPith's reviewarXiv
  14. [14] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2023. https://arxiv.org/abs/2307.05695 ReLoRA : High-rank training through low-rank updates . Preprint, arXiv:2307.05695 on PithPith's reviewarXiv
  15. [15] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024 a . https://proceedings.mlr.press/v235/liu24bn.html DoRA : Weight-decomposed low-rank adaptation . In Proceedings of the 41st International Conference on ... on Pith
  16. [16] Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. 2024 b . https://doi.org/10.18653/v1/2024.naacl-long.35 AL o RA : Allocating low-rank adaptation for fine-tuning large language models . In Proceedings of the 2024 Conference of the North American Chapter of the As... on Pithdoi:10.18653/v1/2024.naacl-long.35
  17. [17] Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. 2018. https://openreview.net/forum?id=Byt3oJ-0W Learning latent permutations with Gumbel--Sinkhorn networks . In International Conference on Learning Representations on Pith
  18. [18] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2025. https://arxiv.org/abs/2404.02948 PiSSA : Principal singular values and singular vectors adaptation of large language models . Preprint, arXiv:2404.02948 on PithPith's reviewarXiv
  19. [19] John X. Morris, Niloofar Mireshghallah, Mark Ibrahim, and Saeed Mahloujifar. 2026. https://arxiv.org/abs/2602.04118 Learning to reason in 13 parameters . Preprint, arXiv:2602.04118 on PitharXiv
  20. [20] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, an... on PithPith's reviewarXiv
show 13 more
  1. [21] Sebastian Raschka. 2023. https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms Practical tips for finetuning LLMs using LoRA (low-rank adaptation) . Ahead of AI / Raschka AI Research Lab on Pith
  2. [22] Alfr \'e d R \'e nyi. 1961. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 547--561. University of California Press, Berkeley, Calif on Pith
  3. [23] Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606--610. IEEE on Pith
  4. [24] John Schulman and Thinking Machines Lab . 2025. https://doi.org/10.64434/tml.20250929 LoRA without regret . Thinking Machines Lab: Connectionism. September 29, 2025 on Pithdoi:10.64434/tml.20250929
  5. [25] Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. 2024. https://arxiv.org/abs/2410.21228 LoRA vs full fine-tuning: An illusion of equivalence . Preprint, arXiv:2410.21228 on PitharXiv
  6. [26] Shivalika Singh, Freddie Vargus, Daniel D’souza, B \"o rje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, et al. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. In Proceedings of t... on Pith
  7. [27] Richard Sinkhorn. 1964. A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, 35(2):876--879 on Pith
  8. [28] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. 2022. https://arxiv.org/abs/2206.06122 Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning . Preprint, arXiv:2206.06122 on PithPith's reviewarXiv
  9. [29] Asher Trockman and J. Zico Kolter. 2021. https://arxiv.org/abs/2104.07167 Orthogonalizing convolutional layers with the Cayley transform . arXiv preprint arXiv:2104.07167 on PithPith's reviewarXiv
  10. [30] Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. 2024. https://arxiv.org/abs/2406.09044 MiLoRA : Harnessing minor singular components for parameter-efficient LLM finetuning . Preprint, arXiv:2406.09044 on PithPith's reviewarXiv
  11. [31] Ruigang Wang, Krishnamurthy Dvijotham, and Ian R. Manchester. 2025. https://arxiv.org/abs/2501.19050 Norm-bounded low-rank adaptation . Preprint, arXiv:2501.19050 on PitharXiv
  12. [32] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025. https://arxiv.org/abs/... on PithPith's reviewarXiv
  13. [33] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. https://arxiv.org/abs/2303.10512 AdaLoRA : Adaptive budget allocation for parameter-efficient fine-tuning . Preprint, arXiv:2303.10512 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment