Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Weight Spectra Induced Efficient Model Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning amplifies the top singular values of weight matrices, and rescaling those directions with a learnable mask outperforms LoRA on three benchmarks.

desk verdict The SVD-to-mask equivalence is false, and the paper's mechanism claim hinges on it; the empirical variant is real but the spectral story needs replacing. read the letter →

arxiv 2505.23099 v1 pith:5B3RTUHG submitted 2025-05-29 cs.LG

Chongjie Si , Xuankun Yang , Muqing Liu , Yadao Wang , Xiaokang Yang , Wenbo Su , Bo Zheng , Wei Shen This is my paper
classification cs.LG MSC 68T0715A18
keywords singularvaluedecompositionparameter-efficientfine-tuningLoRAspectralanalysislow-rankadaptationfoundationmodelsSpec
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when a large pre-trained model is fine-tuned on a downstream task, the weight matrices change in a highly structured way: the top singular values are amplified and the top singular vectors rotate toward task-specific directions, while the rest of the spectrum and the lower singular vectors stay almost unchanged. On the basis of this observation, the paper proposes SpecLoRA, a parameter-efficient fine-tuning method that multiplies the pre-trained weight matrix by a learnable mask that scales a small top-left block, and adds a low-rank LoRA update. It reports consistent gains over strong baselines: 89.48 average on GLUE with DeBERTaV3-base, 85.5 on Commonsense170K with LLaMA3-8B, and 76.7 on VTAB-1K with ViT-B/16, all at a parameter budget on par with LoRA. If the spectral claim is right, it explains why low-rank adaptation works and points to a cheaper, more targeted way to adapt foundation models.

What carries the argument

The central object is the singular value decomposition $W = U\Sigma V^\top$ of each pre-trained weight matrix, used both as an analytical lens and as the motivation for a spectral modulation mask. The paper's key move is to replace explicit SVD reconstruction with the Hadamard product $W \to (\Gamma \odot W) + AB$, where $\Gamma$ is a learnable $n \times m$ matrix that scales the first $k$ rows of the first $k$ columns (a corner of the matrix) and leaves the rest unchanged. This formulation is claimed to modulate the dominant singular directions while preserving the global structure, and the low-rank term $AB$ is added to capture residual task-specific directions. The mask's learnable scaling vector $d \in \mathbb{R}^k$ is the only new parameter beyond LoRA.

What would settle it

Take a generic pre-trained weight matrix $W$, compute its SVD, apply the SpecLoRA mask with a fixed nonzero $d$, and compare the top singular vectors of $\Gamma \odot W$ with those of the explicitly SVD-rescaled matrix $\begin{bmatrix} D U_{1:k} & U_{k+1:n}\end{bmatrix}\Sigma V^\top$; if they diverge for a typical $W$, the assertion that the efficient implementation reproduces spectral rescaling is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that fully fine-tuning a pre-trained weight matrix $W$ changes it almost entirely through its top singular components: the singular value spectrum of the fine-tuned matrix overlaps with the pre-trained one except for an amplification of the largest values, and the corresponding left singular vectors become nearly orthogonal to the original while the remaining singular vectors stay closely aligned. This is interpreted as evidence that task-specific knowledge is injected into a low-dimensional subspace. Based on this, SpecLoRA modifies the forward pass to $W \to (\Gamma \odot W) + AB$, where $\Gamma$ is a learnable scaling mask on a small corner of $W$ and $AB$ is the usual LoRA update; the mask is meant to rescale the top-$k$ singular directions without a full SVD. The paper claims this mechanism outperforms LoRA, DoRA, and other PEFT baselines on natural language understanding, commonsense reasoning, and vision tasks.

Load-bearing premise

The load-bearing premise is that the efficient mask $\Gamma \odot W$ is a faithful way to rescale the top-$k$ singular directions of $W$, even though scaling a fixed corner of a generic matrix does not generally alter its singular vectors in that way.

Editorial extensions

If this is right

  • If fine-tuning only amplifies the top singular values, then a parameter-efficient method that rescales those values should recover most of the benefit of full fine-tuning; the reported benchmark results are consistent with that.
  • SpecLoRA improves over LoRA at the same parameter budget on GLUE, commonsense reasoning, and VTAB-1K, suggesting that spectral guidance adds signal beyond the low-rank update alone.
  • The method is compatible with existing PEFT pipelines because it only adds a mask multiply before the LoRA update, so it can be combined with adapters or prompts.
  • The paper reports larger relative gains on low-resource tasks such as RTE and CoLA, indicating that spectral rescaling is especially helpful when training data is scarce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the spectral analysis is done on only a handful of selected layers and matrices; a natural extension is to test whether the amplification-and-reorientation pattern holds across all layers and across different model families, which would determine how universal the claim is.
  • Because the mask only scales a fixed corner of $W$, the method may in effect be rescaling coordinate-wise magnitudes rather than true singular directions; if so, the empirical gains could stem from a different mechanism, such as per-feature input scaling, and the spectral interpretation would need to be revised.
  • The near-orthogonality of top singular vectors between pre-trained and fine-tuned weights could be inflated by SVD instability when the top singular values are close together; computing canonical correlations between the principal subspaces, rather than pairwise vector cosines, would provide a more robust test of the reorientation claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper analyzes singular value/vector changes during full fine-tuning of LLaMA3-8B on Commonsense170K, claiming that fine-tuning mainly amplifies the top singular values while leaving the rest largely intact and reorients the dominant singular vectors toward task-specific directions. Based on this, it proposes SpecLoRA, a PEFT method defined as W -> (Gamma ⊙ W) + AB with a block mask Gamma, and claims this is an efficient implementation of an SVD-based rescaling of the top-k left singular directions. The paper reports consistent average improvements over strong baselines on GLUE, commonsense reasoning, and VTAB-1K benchmarks.

Significance. If the claimed mechanism were correct, the paper would provide a useful bridge between spectral analysis of fine-tuning and PEFT design, and the proposed mask is simple, adds negligible parameters, and shows consistent average improvements over strong baselines (e.g., +1.35 on GLUE over LoRA, +0.5 on commonsense reasoning over DoRA, +1.0 on VTAB-1K over SSF). However, the paper provides no code, no quantitative spectral summaries, no error bars, and its central implementation claim is algebraically false. The empirical gains therefore do not validate the spectral hypothesis, and the contribution reduces to an empirical heuristic whose advantage over existing methods is modest and possibly not statistically significant. The paper does not ship machine-checked proofs, reproducible code, parameter-free derivations, or falsifiable predictions.

major comments (3)
  1. [Sec. 4.3, Eqs. (6)-(8)] The claimed equivalence between the SVD-based update in Eq. (6) and the Hadamard-mask implementation in Eq. (7) is false for a generic weight matrix. With Δ_impl = (Γ-1)⊙W, the update has nonzero entries only in the top-left k×k block, whereas the update in Eq. (6) equals (tilde U_{1:k} - U_{1:k}) Σ_{1:k} V_{1:k}^T, which is generally nonzero in all m columns of the first k rows because V_{1:k}^T is dense, and its entries depend on the global SVD of W. No choice of the learnable vector d makes these updates equal unless W has special structure such as being coordinate-aligned. Consequently, the implemented method does not rescale the top singular directions, and the empirical gains in Tables 1-3 cannot be attributed to the advertised spectral mechanism. The paper provides neither a proof of the equivalence nor a numerical check that the mask approximately reproduces the SVD update.
  2. [Sec. 3, Figs. 1-2] The spectral analysis is purely qualitative and is not sufficient to support the paper's central empirical claim. The results are based on a single model (LLaMA3-8B) and a single dataset (Commonsense170K), with figures showing 'randomly selected layers' and no aggregate statistics, confidence intervals, or quantification of how much the top singular values are amplified. The cosine-similarity analysis in Fig. 2 also ignores the sign ambiguity and potential degeneracy of singular vectors, which can make 'corresponding singular vectors' ill-defined. Without rigorous quantification, the claimed regularities remain anecdotal, and the motivation for SpecLoRA is correspondingly weak.
  3. [Sec. 5.4.3, Table 4] The 'Bottom' variant is not defined precisely. Since the mask Γ in Eq. (8) operates on the top-left k×k block of W in the implemented method, it is unclear what 'applying the same mechanism to the bottom singular directions' means, or whether the variant actually targets the bottom singular directions at all. The ablation therefore does not establish that modifying top singular directions is better than modifying bottom ones, and it also inherits the ambiguity from the first major comment.
minor comments (5)
  1. [Eq. (8)] The notation Γ = [[d d ... d] with k copies; 1 blocks] is ambiguous: it is not clear whether d is a row or column vector, or how the k×k block is tiled into an n×m matrix. Please define Γ explicitly, e.g., Γ_{ij} = d_i for i ≤ k and j ≤ k, and 1 otherwise.
  2. [Related Work] PISSA [35] and MiLoRA [51] appear in Table 2 but are not discussed in Section 2; since they are directly relevant SVD-based PEFT methods, a brief discussion would help position the contribution.
  3. [Table 3] Table 3 is difficult to read because the numeric columns are not aligned with the task names; please reformat it so that each method row is clearly separated and the average is easy to verify.
  4. [Sec. 1] In Section 1, 'with the form as' should be 'with the form'; also, the sentence beginning 'This formulation inherently aligns...' is vague and should be made precise.
  5. [Experiments] The paper does not report whether the results in Tables 1-3 are single runs or averages over multiple seeds; please state this explicitly and, if possible, provide standard deviations or error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the spectral observations motivate, but do not define, the SpecLoRA update; the unproven equivalence between Eq. (6) and Eq. (7) is a correctness gap, not a circular reduction.

full rationale

The paper's derivation chain is not circular. Section 3 reports descriptive SVD observations: fine-tuning amplifies top singular values (Fig. 1) and reorients top singular vectors (Fig. 2). Section 4 then designs SpecLoRA to rescale the top singular directions (Eqs. 4-6), with the learnable vector d and hyperparameter k trained on downstream task accuracy rather than read off from the Section 3 statistics. The empirical validation (Tables 1-3) is against external benchmarks (GLUE, Commonsense170K, VTAB-1k), so the reported improvements are not logically entailed by the observation that originally motivated the design. This is a standard design-inspiration loop, not a definitional reduction. The self-citations ([46]-[48]) appear in related-work and terminology contexts and are not load-bearing; no uniqueness theorem is imported, and no fitted parameter is renamed as a prediction. The paper's main weakness is a correctness gap: Eq. (7) with Gamma from Eq. (8) is claimed, without proof, to be an efficient implementation of Eq. (6), but for a generic weight matrix, scaling the top-left k-by-k block does not rescale the top-k singular directions. That is an internal-consistency and attribution issue, not circularity, so under the hard rules it does not raise the circularity score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central mechanism rests on the equivalence of the SVD reconstruction and the mask implementation, which is assumed rather than proven, and on generalizing a single spectral analysis to other models. The only hand-tuned hyperparameter central to the method is k.

free parameters (2)
  • k (number of scaled rows and columns in mask) = 200 for NLU and commonsense, 32 for VTAB
    Chosen per benchmark rather than derived; Fig. 3 shows performance depends on it, so the method's success relies on this hand-set hyperparameter.
  • LoRA rank r = 16 on commonsense, 2 on GLUE, unspecified for VTAB
    Standard hyperparameter inherited from LoRA; included for completeness since the method's parameter count depends on it.
assumptions (3)
  • domain assumption SVD comparison by sorted index is meaningful: singular vectors of pretrained and fine-tuned weights can be matched at the same index without sign or permutation ambiguity.
    Section 3.2 computes cosine similarity between same-index singular vectors; degenerate or crossing singular values would make this comparison unreliable.
  • ad hoc to paper The Hadamard mask in Eq. (7) is equivalent to the SVD row-scaling in Eq. (6).
    Section 4.3 asserts this without proof; for a general weight matrix the operations differ, so the method's spectral interpretation rests on this assumption.
  • domain assumption Findings from LLaMA3-8B fine-tuned on Commonsense170K generalize to other models, tasks, and layers.
    Section 3 draws global conclusions from one setup; Section 5 then applies the method to DeBERTaV3 and ViT without re-validating the spectral finding on those models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Weight Spectra Induced Efficient Model Adaptation." pith.science (2026). https://pith.science/paper/5B3RTUHG

@misc{pith2026250523099,
  author       = {Pith},
  title        = {Pith review of: Weight Spectra Induced Efficient Model Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5B3RTUHG}},
  note         = {Machine review of arXiv:2505.23099}
}
read the original abstract

Large-scale foundation models have demonstrated remarkable versatility across a wide range of downstream tasks. However, fully fine-tuning these models incurs prohibitive computational costs, motivating the development of Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA, which introduces low-rank updates to pre-trained weights. Despite their empirical success, the underlying mechanisms by which PEFT modifies model parameters remain underexplored. In this work, we present a systematic investigation into the structural changes of weight matrices during fully fine-tuning. Through singular value decomposition (SVD), we reveal that fine-tuning predominantly amplifies the top singular values while leaving the remainder largely intact, suggesting that task-specific knowledge is injected into a low-dimensional subspace. Furthermore, we find that the dominant singular vectors are reoriented in task-specific directions, whereas the non-dominant subspace remains stable. Building on these insights, we propose a novel method that leverages learnable rescaling of top singular directions, enabling precise modulation of the most influential components without disrupting the global structure. Our approach achieves consistent improvements over strong baselines across multiple tasks, highlighting the efficacy of structurally informed fine-tuning.

Figures

Figures reproduced from arXiv: 2505.23099 by the authors.

Figure 1
Figure 1. Singular value distributions of selected weight matrices before and after fine-tuning. We [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Cosine similarity between corresponding singular vectors of pre-trained and fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation study on the number of trainable parameters (i.e., rank setting) of SpecLoRA. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study on the number of trainable parameters (i.e., rank setting) of SpecLoRA. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws

    stat.ML 2025-09 conditional novelty 6.0 of 10

    For a high-dimensional Gaussian sequence model, empirical risk minimization in single-head tied attention has exactly computable test error, interpolation and recovery thresholds, and a singular-value spectrum that be...

Reference graph

Works this paper leans on

59 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020

    arXiv 2012

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  3. [3]

    The second pascal recognising textual entailment challenge

    Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 1. Citeseer, 2006

    2006

  4. [4]

    The fifth pascal recognizing textual entailment challenge

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009

    work page 2009

  5. [5]

    Lora learns less and forgets less

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673, 2024

    arXiv 2024

  6. [6]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [8]

    Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation

    Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017

    arXiv 2017

Show all 59 references
  1. [9]

    Adaptformer: Adapting vision transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664–16678, 2022

    2022

  2. [10]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  4. [12]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy M Cohen, Simran Kaur, Yuanzhi Li, Kolter J.Zico, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. International Conference on Learning Representations (ICLR), 2021

    2021

  5. [13]

    The pascal recognising textual entailment challenge

    Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer, 2005

    2005

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  7. [15]

    Automatically constructing a corpus of sentential paraphrases

    Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005

    2005

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  9. [17]

    Trilora: Integrating svd for advanced style personalization in text-to-image generation

    Chengcheng Feng, Mu He, Qiuyu Tian, Haojie Yin, Xiaofang Zhao, Hongwei Tang, and Xingqiang Wei. Trilora: Integrating svd for advanced style personalization in text-to-image generation. arXiv preprint arXiv:2405.11236, 2024. 10

    2024 arXiv

  10. [18]

    The third pascal recognizing textual entailment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9, 2007

    2007

  11. [19]

    To- wards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  12. [20]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  13. [21]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021

    2021 arXiv

  14. [22]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  15. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  16. [24]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023

    2023 arXiv

  17. [25]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022

    2022

  18. [26]

    Convolutional bypasses are better vision transformer adapters

    Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint arXiv:2207.07039, 2022

    2022 arXiv

  19. [27]

    Fact: Factor-tuning for lightweight adaptation on vision transformer

    Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 1060–1068, 2023

    2023

  20. [28]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023

    2023

  21. [29]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  22. [30]

    Scaling & shifting your features: A new baseline for efficient model tuning

    Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109–123, 2022

    2022

  23. [31]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  24. [32]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  25. [33]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 11

    1907 arXiv

  26. [34]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024

    2024

  27. [35]

    Pissa: Principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948, 2024

    2024 arXiv

  28. [36]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  29. [37]

    Adapterfusion: Non-destructive task composition for transfer learning

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020

    2005 arXiv

  30. [38]

    Pre-trained models for natural language processing: A survey

    Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897, 2020

    2020

  31. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  32. [40]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  33. [41]

    Residual prompt tuning: Improving prompt tuning with residual reparameterization

    Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization. arXiv preprint arXiv:2305.03937, 2023

    2023 arXiv

  34. [42]

    Effective rank: A measure of effective dimensionality.European Signal Processing Conference, 2007

    Olivier Roy and Martin Vetterli. Effective rank: A measure of effective dimensionality.European Signal Processing Conference, 2007

    2007

  35. [43]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  36. [44]

    Socialiqa: Com- monsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  37. [45]

    Dept: Decomposed prompt tuning for parameter-efficient fine-tuning

    Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. arXiv preprint arXiv:2309.05173, 2023

    2023 arXiv

  38. [46]

    Unleashing the power of task-specific directions in parameter efficient fine-tuning

    Chongjie Si, Zhiyi Shi, Shifan Zhang, Xiaokang Yang, Hanspeter Pfister, and Wei Shen. Unleashing the power of task-specific directions in parameter efficient fine-tuning. arXiv preprint arXiv:2409.01035, 2024

    2024

  39. [47]

    Flora: Low-rank core space for n-dimension

    Chongjie Si, Xuehui Wang, Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xi- aokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405.14739, 2024

    2024 arXiv

  40. [48]

    See further for parameter efficient fine-tuning by standing on the shoulders of decomposition

    Chongjie Si, Xiaokang Yang, and Wei Shen. See further for parameter efficient fine-tuning by standing on the shoulders of decomposition. arXiv preprint arXiv:2407.05417, 2024

    2024 arXiv

  41. [49]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language proce...

    2013

  42. [50]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  43. [51]

    Milora: Harnessing minor singular components for parameter-efficient llm finetuning

    Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. arXiv preprint arXiv:2406.09044, 2024. 12

    2024 arXiv

  44. [52]

    Neural network acceptability judgments

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641, 2019

    2019

  45. [53]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017

    2017 arXiv

  46. [54]

    Mixture-of-subspaces in low-rank adaptation

    Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. Mixture-of-subspaces in low-rank adaptation. arXiv preprint arXiv:2406.11909, 2024

    2024 arXiv

  47. [55]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021

    2021

  48. [56]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  49. [57]

    The visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019

    2019

  50. [58]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2022

    2022

  51. [59]

    Neural prompt search

    Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022. 13 Appendix We here present some experimental details. Table 5: Hyper-parameter configurations for commonsense reasoning task. Hyper-parameter LoRA AdaLoRA DoRA SpecLoRA Ra...

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.