Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Updating only the half of a model's weight matrices with the largest condition numbers matches full LoRA accuracy while cutting fine-tuning time by 16%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:32 UTC pith:HNGBQB3Q

load-bearing objection κ-LoRA is a cheap and plausible selection heuristic, but the causal story—that high-κ matrices drive LoRA gains—is unproven because the paper never compares against random or low-κ selection at the same budget. the 4 major comments →

arxiv 2607.22489 v1 pith:HNGBQB3Q submitted 2026-07-24 cs.LG cs.AI

k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

Jianghui Wang , Silong Yong , Francesco Orabona , Marco Canini , Katia P. Sycara , Yaqi Xie This is my paper
classification cs.LG cs.AI MSC 15A1865F3568T07
keywords condition numberLoRAlow-rank adaptationsingular value decompositionparameter-efficient fine-tuningspectral analysismodel selectionfine-tuning efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Not all LoRA matrices are equally worth tuning: the condition number of each frozen weight matrix predicts how much a bounded low-rank update can alter its behavior, with larger condition numbers indicating directions that are easier to reshape. The paper's method, κ-LoRA, exploits this by ranking all eligible projection matrices within architectural groups and attaching LoRA adapters only to the top half by condition number. In experiments across 7B language models and DeBERTa, this halves the trainable parameters, cuts fine-tuning time by 16–22%, and matches or slightly surpasses standard LoRA on most tasks. The authors also report that the condition numbers of the adapted matrices decrease during training, which they interpret as evidence that the benefit comes from spectrally rebalancing these ill-conditioned transformations.

Core claim

The paper's central claim is that adaptation sensitivity in pre-trained transformers is highly uneven across weight matrices and is largely captured by a single spectral statistic, the condition number κ(W)=σmax(W)/σmin(W). For a matrix W, a low-rank update of Frobenius norm ε can change the action of W along any singular direction by at most ε/σi; the maximum relative change, attained on the smallest singular direction, is ρ(W;ε)=ε/σmin(W)=εκ(W)/σmax(W). Since σmax(W) is observed to vary relatively little within an architectural group (attention versus MLP projections), ranking matrices by κ(W) is approximately the same as ranking by ρ(W;ε), i.e., by how much a bounded low-rank update can c

What carries the argument

The condition number κ(W)=σmax(W)/σmin(W) and the derived identity ρ(W;ε)=εκ(W)/σmax(W), which bounds the maximum relative change in a matrix's action along any singular direction achievable by a low-rank update of Frobenius norm ε. The condition number is invariant to uniform scaling of W, making it a scale-free way to compare matrices whose overall magnitudes differ. κ-LoRA uses it as a one-shot, training-free ranking criterion within architectural groups; under the paper's 'mild assumption' that σmax varies within a comparatively small range inside a group, ranking by κ is approximately equivalent to ranking by ρ, and therefore selects the matrices where bounded low-rank updates have the

Load-bearing premise

The load-bearing premise is that within each architectural group the largest singular values σmax(W) vary within a comparatively small range, so that ρ(W;ε)=εκ(W)/σmax(W) is approximately proportional to κ(W); if σmax varies widely within a group, ranking by κ need not maximize the total achievable change, and the theoretical rationale for the selection collapses.

What would settle it

Compute, for a representative large model (e.g., LLaMA-2-7B), the within-group distribution of σmax(W) across all attention and MLP projections. If the range of σmax within a group is comparable to or larger than the range of κ(W), then the proportionality ρ≈(ε/σmax)κ fails and ranking by κ may not maximize Φ(S). A second, behavioral falsifier: fine-tune the same backbone with LoRA adapters attached to the bottom half of matrices by condition number (i.e., the lowest κ), and compare accuracy against both full LoRA and κ-LoRA. If low-κ matrices produce comparable accuracy to high-κ selection, t

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Selecting the top 50% of projection matrices by condition number halves the number of trainable LoRA parameters and reduces adapter checkpoint size by roughly 45–50%, with fine-tuning time dropping 15.6–17.0% on 7B NLG backbones and 22.2% on GLUE with DeBERTa.
  • κ-LoRA matches or exceeds full LoRA on 11 of 15 (model, task) pairs in the NLG experiments and on 7 of 12 GLUE metrics, with the largest improvements on MATH, MT-Bench, and MBPP.
  • The selection step is data-free and training-free: it requires only one SVD of each frozen weight matrix, so the overhead is negligible and the ranking can be computed once and reused for any downstream task.
  • The observation that 83–87% of selected matrices become better conditioned during fine-tuning suggests that LoRA's primary effect on high-κ matrices is to rebalance their spectra toward isotropy, which could serve as a principle for designing future spectral-based adaptation methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The theoretical ranking argument rests on the unverified 'mild assumption' that σmax(W) is roughly constant within an architectural group. A direct check would be to measure the within-group variation of σmax in a large model; if its range is comparable to the range of κ, ranking by κ may not maximize total achievable change, and ranking by ε/σmin would be the more faithful criterion.
  • The paper argues that high-κ matrices are 'worth tuning' because they admit larger relative changes, but it does not directly test whether that sensitivity translates into task performance. A clean ablation would compare κ-LoRA against random selection of the same fraction of matrices, or against selection by σmin alone, holding all other training settings fixed; a small accuracy gap would weaken
  • The observed decrease in condition numbers during training is consistent with spectral rebalancing, but any gradient-based update that shrinks σmax or grows σmin would produce the same effect. A control experiment that artificially reorders matrices by condition number (or trains only low-κ matrices) would help isolate whether the selection signal, rather than generic training dynamics, is respons
  • Because the selection is a preprocessing step, it should compose with other efficiency techniques such as quantization or adaptive rank allocation; the paper notes this combination as future work, and if the savings are multiplicative, κ-LoRA could become a standard front-end for parameter-efficient fine-tuning pipelines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes κ-LoRA, a method that selects the top half of the weight matrices in each architectural group by the condition number κ(W)=σ_max/σ_min of the frozen pretrained weights, and attaches LoRA adapters only to those matrices. The theoretical motivation is Eq. (3): the maximum relative change along a singular direction under a bounded low-rank update is ρ(W;ε)=ε/σ_min(W)=ε κ(W)/σ_max(W). Under an unverified 'mild assumption' that σ_max is roughly constant within each architectural group, selecting high-κ matrices maximizes the total achievable relative change Φ(S). Experiments on NLG (LLaMA-2-7B, Mistral-7B, Gemma-7B) and NLU (DeBERTa-v3-base) report wall-clock time savings around 16% (up to 22% on GLUE) with accuracy roughly matching full LoRA, and the paper additionally observes that the condition numbers of selected matrices decrease during training.

Significance. If the causal claim is established, κ-LoRA is an attractive method: the selection rule is data-free, training-free, one-shot, and orthogonal to quantization and other LoRA extensions, and the reported speedups are consistent across several backbones and task families. The derivation of ρ(W;ε) is correct, and the rank-1 achievability argument is valid. However, the paper's central claim — that high-κ matrices 'drive most of the performance gains' — is not empirically supported without same-budget control experiments, and the σ_max constancy assumption underpinning the theory is never tested. The observed decrease in condition numbers is correlational and does not by itself establish a causal role for κ. The manuscript's own Limitations section concedes only 'greater leverage' rather than optimality, a weaker statement than the abstract's causal language.

major comments (4)
  1. [Section 4, Tables 1-2, Figure 2] No same-budget control experiment is reported. κ-LoRA is compared only to full-LoRA; the ablation sweep (Figure 2) varies the budget α but not the selection criterion. The results are fully consistent with the null hypothesis that any 50%-parameter subset — random, bottom-κ, or an existing selection method such as LoRA-Drop or AFLoRA — yields similar speedup and accuracy parity. To support the causal claim that high-κ matrices specifically drive adaptation gains, the authors must add matched-budget baselines: random selection, bottom-κ selection, and at least one existing selection method. Without these, the paper only demonstrates that half of the LoRA modules can be dropped, which is already known.
  2. [Section 3.2, Eq. (3)] The theoretical ranking by κ relies on the 'mild assumption' that σ_max(W) varies within a comparatively smaller range inside each architectural group. This assumption is never verified. Figure 1 shows only condition-number distributions, not σ_max distributions. If σ_max varies substantially within a group, ranking by κ does not maximize Φ(S), and the theoretical justification collapses. The authors should report σ_max ranges/statistics per group, or directly verify that ranking by ρ(W;ε) and by κ agree on the selected sets.
  3. [Section 3.2 and Limitations] The proxy ρ(W;ε) is the maximum relative change along a single singular direction under a bounded low-rank update. The paper asserts that this measures how much a matrix contributes to downstream adaptation, but no evidence links ρ to task performance. The Limitations section concedes that the analysis establishes 'greater leverage rather than full optimality of top-κ selection,' yet the abstract and introduction make the stronger claim that high-κ matrices 'drive most of the performance gains.' This discrepancy should be resolved, and the proxy validity requires an explicit experiment (e.g., comparing adaptation quality of matrices selected by high ρ vs. high κ vs. random at the same budget).
  4. [Abstract and Appendix A] The abstract claims 'reducing memory cost by 4.5%,' but Appendix A (Table 4) measures on-disk adapter checkpoint sizes, not training-time memory usage (activations, gradients, optimizer states). Training memory is never measured. The claim should be reworded to 'storage' or 'checkpoint size,' or supplemented with actual memory profiling during training.
minor comments (4)
  1. [Abstract] 'We show for the first time that not all LoRA matrices are equally worth tuning' is overstated given prior methods such as AdaLoRA, LoRA-Drop, and AFLoRA already demonstrate uneven adapter importance. Suggest 'for the first time using a condition-number selection criterion.'
  2. [Table 2] GLUE results are reported without standard deviations. Given the known high variance of the RTE split (which the paper itself notes), at least a few seeds should be reported or the absence justified.
  3. [Figure 2] The normalized scores are difficult to read. A numeric table accompanying the figure, or clearer axis labels, would aid interpretation.
  4. [Appendix A, Table 4] The column labeled '∆ratio' is described as the saving as a fraction of the base-model storage cost; clarify explicitly that this is disk storage, not memory, to avoid confusion with the abstract's 'memory cost' claim.

Circularity Check

0 steps flagged

No significant circularity: the κ-ranking rule follows from a derived bound, and the 50% selection ratio is a reported ablation choice rather than a predicted quantity.

full rationale

The paper's derivation chain is self-contained. Equation (3), ρ(W;ε)=ε κ(W)/σ_max(W), follows algebraically from the definitions ρ(W;ε)=ε/σ_min(W) and κ(W)=σ_max(W)/σ_min(W); it is not fitted or assumed. The selection rule (Algorithm 1) ranks matrices by κ computed once from pretrained weights, so the selection signal is not constructed from the downstream accuracies it is later compared with. The 50% ratio is chosen from the ablation sweep in Figure 2 and reported as such, not derived and then called a prediction, so it is a tuned hyperparameter rather than a circular input. No load-bearing self-citation appears: references such as PiSSA [25] are used for experimental protocol, and none of the paper's authors supplies the central premise through a self-citation. The unverified 'mild assumption' that σ_max is roughly constant within architectural groups and the absence of a same-budget random/bottom-κ control are substantive empirical-support concerns, but they are not definitional circularity: the paper explicitly states in Limitations that the analysis 'establishes greater leverage rather than full optimality of top-κ selection.' The central claim therefore does not reduce by construction to its inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The method introduces no new entities. It relies on two domain assumptions (σ_max constancy and ρ as a leverage proxy) that are stated but not empirically validated, plus one standard-math fact. The only fitted parameter is the selection ratio α.

free parameters (1)
  • selection ratio α = 0.5
    The top-50% budget is chosen from an ablation sweep (Figure 2) as a Pareto knee; it is not derived from theory and is the key hyperparameter of the method.
axioms (3)
  • domain assumption σ_max(W) varies within a comparatively smaller range inside each architectural group, so ρ is approximately proportional to κ.
    Stated in Section 3.2 as a 'mild assumption' but never empirically verified; load-bearing for the ranking by condition number to maximize Φ(S).
  • domain assumption ρ(W;ε), the maximum relative change along a singular direction under a bounded low-rank update, is a valid proxy for adaptation leverage.
    Used to justify that high-κ matrices are the most valuable to adapt; asserted, not derived or tested against alternative measures.
  • standard math Rank-1 updates of the form ΔW = ε u v_i^T can be realized by LoRA parameterization and achieve the bound.
    True linear algebra: such an update has Frobenius norm ε and achieves relative change ε/σ_i; LoRA can express rank-1 updates, so the bound is attainable within the parameterization.

pith-pipeline@v1.3.0-alltime-deepseek · 15577 in / 11184 out tokens · 99969 ms · 2026-08-01T04:32:40.060629+00:00 · methodology

0 comments
read the original abstract

Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices. However, LoRA remains computationally costly because it updates all matrices uniformly, regardless of their actual contribution to adaptation. This cost is especially prohibitive for large-scale models with billions of parameters and for resource-constrained settings such as edge deployment and on-device fine-tuning. We show for the first time that not all LoRA matrices are equally worth tuning: matrices with smaller condition numbers (the ratio of largest to smallest singular value) are already well-balanced across directions and contribute only marginally to adaptation, whereas matrices with larger condition numbers contain underdeveloped directions that span richer subspaces and drive most of the performance gains. This observation itself is a key contribution of our work, and it motivates a more selective approach to fine-tuning. Building on this insight, we propose \k{appa}-LoRA, a method that optimizes LoRA by focusing updates on the matrices with the largest condition numbers, which capture the most informative directions of change. By restricting LoRA updates to the top 50% of weight matrices ranked by condition number, \k{appa}-LoRA halves the trainable parameter count and correspondingly reduces compute and memory cost. Extensive experiments across multiple benchmarks show that this design cuts fine-tuning time by 16.2% on average while matching the accuracy of standard LoRA and reducing memory cost by 4.5%. Further analysis reveals that the condition numbers of the selected matrices consistently decrease over training, suggesting that \k{appa}-LoRA's effectiveness stems from targeted spectral rebalancing rather than parameter selection alone.

Figures

Figures reproduced from arXiv: 2607.22489 by Francesco Orabona, Jianghui Wang, Katia P. Sycara, Marco Canini, Silong Yong, Yaqi Xie.

Figure 1
Figure 1. Figure 1: Condition numbers of different projection matrices in transformer architectures exhibit [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: κ-LoRA sweep, normalized to full￾LoRA at LLaMA2-7B. When α becomes too small, performance degradation appears most noticeably on reasoning-intensive tasks such as MATH, suggesting that overly aggressive sparsification removes matrices that still contribute meaningfully to complex reasoning behaviors. In contrast, increasing α beyond 50% yields diminishing performance gains while steadily increasing fine-tu… view at source ↗
Figure 3
Figure 3. Figure 3: LoRA training systematically reduces the condition number of high- [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 19 linked inside Pith

  1. [1]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  4. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL, 2019

    2019

  7. [7]

    Sparse low-rank adaptation of pre-trained language models

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 4133–4145, 2023

    2023

  8. [8]

    Sharp minima can generalize for deep nets

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. InInternational Conference on Machine Learning, pages 1019–1028. PMLR, 2017

    2017

  9. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  10. [10]

    Hifi: High-information attention heads hold for parameter-efficient model adaptation

    Anchun Gui and Han Xiao. Hifi: High-information attention heads hold for parameter-efficient model adaptation. InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (V olume 1: Long Papers), pages 8521–8537, 2023

    2023

  11. [11]

    Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing, 2021

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing, 2021

    2021

  12. [12]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019. 11

    2019

  14. [14]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b.arXiv preprint arXiv:23...

    Pith/arXiv arXiv 2023

  16. [16]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  17. [17]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Losparse: Structured compression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. InInternational Conference on Machine Learning, pages 20336–20350. PMLR, 2023

    2023

  20. [20]

    Svft: Parameter- efficient fine-tuning with singular vectors.Advances in Neural Information Processing Systems, 37:41425–41446, 2024

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham K Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter- efficient fine-tuning with singular vectors.Advances in Neural Information Processing Systems, 37:41425–41446, 2024

    2024

  21. [21]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Aflora: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models

    Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, and Peter Beerel. Aflora: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Papers), pages 161–167, 2024

    2024

  23. [23]

    SGDR: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017

    2017

  24. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  25. [25]

    PiSSA: Principal singular values and singular vectors adaptation of large language models.arXiv preprint arXiv:2404.02948, 2024

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models.arXiv preprint arXiv:2404.02948, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Lisa: Lay- erwise importance sampling for memory-efficient large language model fine-tuning.Advances in Neural Information Processing Systems, 37:57018–57049, 2024

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Lay- erwise importance sampling for memory-efficient large language model fine-tuning.Advances in Neural Information Processing Systems, 37:57018–57049, 2024

    2024

  27. [27]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  28. [28]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Anto...

    Pith/arXiv arXiv 2024

  29. [29]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    Pith/arXiv arXiv 2023

  30. [30]

    SIAM, 2022

    Lloyd N Trefethen and David Bau.Numerical linear algebra. SIAM, 2022

    2022

  31. [31]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017

    2017

  32. [32]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018

    2018

  33. [33]

    Milora: Harnessing minor singular components for parameter-efficient llm finetuning

    Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pages 4823–4836, 2025

    2025

  34. [34]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  35. [35]

    Qa-lora: Quantization-aware low-rank adaptation of large language models.arXiv preprint arXiv:2309.14717, 2023

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models.arXiv preprint arXiv:2309.14717, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    MetaMath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Bag of tricks for training data extraction from language models

    Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. Bag of tricks for training data extraction from language models. InInternational Conference on Machine Learning, pages 40306–40320. PMLR, 2023

    2023

  38. [38]

    AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512, 2023

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512, 2023. 13

    Pith/arXiv arXiv 2023

  39. [39]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena.Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [40]

    OpenCodeInterpreter: Integrating code generation with execution and refinement

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodeInterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Lora- drop: Efficient lora parameter pruning based on output evaluation

    Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. Lora- drop: Efficient lora parameter pruning based on output evaluation. InProceedings of the 31st International Conference on Computational Linguistics, pages 5530–5543, 2025

    2025

  42. [42]

    Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices.arXiv preprint arXiv:2309.02411, 2023

    Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices.arXiv preprint arXiv:2309.02411, 2023. 14 Appendix A Memory and compute requirements We report our experiment memory usage at Table 4. Base (MB) is the canonical Hugging Face checkpoint (*.safeten...

    Pith/arXiv arXiv 2023