Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that EDoRA, by freezing SVD-initialized low-rank matrices and training only a small square matrix between them, can match or beat LoRA and DoRA on GLUE with up to 30x fewer trainable parameters.

desk verdict Plausible, clearly specified PEFT combination; the parameter-count win is real, but the GLUE superiority claim needs baseline transparency and significance testing. read the letter →

arxiv 2501.12067 v1 pith:I3MA2CSQ submitted 2025-01-21 cs.LG cs.AIcs.CL

Hamid Nasiri , Peter Garraghan This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords EDoRAparameter-efficientfine-tuninglow-rankadaptationsingularvaluedecompositionweightmagnitude-directionGLUEbenchmarkRoBERTa
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EDoRA is a parameter-efficient fine-tuning method that aims to reproduce the learning behavior of full fine-tuning while using far fewer trainable parameters than LoRA or DoRA. The method splits each pretrained weight matrix into a trainable magnitude vector and a directional component, freezes two low-rank matrices built from the top singular vectors of that direction, and trains only a small square matrix placed between them. On the GLUE benchmark with RoBERTa-base, the paper reports that EDoRA at rank 32 uses 86.0K trainable parameters and reaches 84.48% average performance, compared with the best LoRA result of 83.66% using 2.0M parameters and the best DoRA result of 83.78% using 2.1M parameters. The claimed implication is that memory-constrained and small-data deployments can get full fine-tuning-like adaptation without the parameter overhead.

What carries the argument

The load-bearing object is the update rule $W' = m\frac{W_0 + B R A}{\lVert W_0 + B R A\rVert_c}$, where $W_0$ is the frozen pretrained weight, $m$ is a trainable magnitude vector, $\lVert \cdot \rVert_c$ denotes the vector-wise column norm, and $A = V_r^T$, $B = U_r\Sigma_r$ are frozen low-rank matrices built from the top $r$ singular vectors of the directional matrix $D = W_0/\lVert W_0\rVert_c$. The small trainable matrix $R \in \mathbb{R}^{r\times r}$, initialized with a near-zero Gaussian, carries the directional adaptation inside a subspace aligned with the most important singular directions of the pretrained weights. This restricts the trainable parameter count per weight matrix to $n + r^2$, versus $2nr$ for LoRA and $n + 2nr$ for DoRA, which is the source of the reported efficiency gain.

What would settle it

Re-run the six GLUE tasks with identical hyperparameter search budgets and all five seeds for every method, then compare EDoRA's average against the best baseline; if the gap falls within the run-to-run spread reported in Table 2, the claimed advantage is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that magnitude-direction decomposition, SVD-based initialization, and a frozen low-rank frame with a trainable $r\times r$ core together recover the learning pattern of full fine-tuning at a fraction of the parameter cost. Concretely, EDoRA writes the adapted weight as $W' = m\frac{W_0 + B R A}{\lVert W_0 + B R A\rVert_c}$, where $A$ and $B$ are frozen matrices from the truncated SVD of the directional part and $R$ is the only trainable low-rank matrix, with $m$ a trainable per-column magnitude. On six GLUE tasks the paper reports that EDoRA improves over LoRA, DoRA, and LoRA-XS on RTE, STSB, CoLA, and SST2, stays competitive on MRPC and QNLI, and achieves the best average score with the smallest parameter count. An ablation attributes a consistent part of the gain to SVD initialization, which outperforms random initialization at every rank by an average of about 1.46 percentage points.

Load-bearing premise

The load-bearing premise is that the LoRA, DoRA, and LoRA-XS baselines were tuned as carefully as EDoRA, so the reported accuracy gap reflects the method rather than uneven search effort.

Editorial extensions

If this is right

  • Because the adapted weight merges into $W_0$ before inference, EDoRA adds no inference latency, matching LoRA and DoRA on that front.
  • Savings grow with model width: for GPT-3 at rank 16 the paper computes 31x fewer trainable parameters than LoRA and 32x fewer than DoRA, and estimates that 1 million personalized checkpoints would drop from 144TB to 4.59TB of memory.
  • On the small-data GLUE tasks RTE and CoLA, EDoRA posts its largest gains, consistent with the claim that its compact parameterization reduces overfitting.
  • SVD initialization is load-bearing within the method: the reported ablation gives it an average 1.46-point advantage over random initialization across all tested ranks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to check whether the frozen singular frame caps performance on tasks that need directions outside the top-$r$ subspace, since EDoRA never updates $A$ or $B$ during training.
  • The recipe suggests a broader family: any frozen orthogonal frame plus a small trainable core could be compared against the SVD frame as an inexpensive ablation, separating the value of the frame from the value of the decomposition.
  • The reported average gap of roughly 0.8 points sits within the standard deviations in Table 2, so an independent replication with matched search budgets and more seeds is the natural next check before treating the ranking as settled.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes EDoRA, a parameter-efficient fine-tuning (PEFT) method that builds on the weight decomposition of DoRA. EDoRA decomposes a pre-trained weight matrix into a trainable magnitude vector and a directional component, freezes two low-rank matrices A and B obtained from a truncated SVD of the directional matrix, and introduces a small trainable square matrix R between them. The number of trainable parameters is n + r^2 per n x n weight matrix (magnitude vector plus R), which is far smaller than LoRA's 2nr or DoRA's n + 2nr. Experiments on six GLUE tasks with RoBERTa-base compare EDoRA with LoRA, DoRA, and LoRA-XS, reporting that EDoRA achieves the highest average accuracy (84.48%) at rank 32 while using 86.0K trainable parameters, versus LoRA's best 83.66% at 2.0M parameters and DoRA's best 83.78% at 2.1M parameters. An ablation study shows that SVD initialization improves average accuracy by about 1.5 points over random initialization.

Significance. If the empirical results are confirmed, EDoRA is a meaningful improvement in parameter efficiency: it reduces trainable parameters by roughly one to two orders of magnitude relative to LoRA and DoRA while preserving accuracy on the GLUE benchmark. The parameter-efficiency analysis in Section 3.1 is straightforward and correct, and the SVD-initialization ablation in Table 3 is a clean, self-contained comparison that supports the design choice. The authors provide code, which facilitates reproducibility. The main limitation is that the cross-method comparison in Table 2 is not fully controlled: baseline hyperparameters are absent, no significance tests are reported, and the rank-selection protocol is unclear. These are addressable issues, and the central efficiency claim (parameter reduction) does not depend on the small accuracy differences; the claim of 'competitive or superior' accuracy would be strengthened by rigorous experimental reporting.

major comments (4)
  1. [§4.1, Table 2] The empirical comparison is not fully controlled because hyperparameters for the baselines are not reported. The text states that 'Hyperparameters were optimized through grid search, and the selected values are presented in Table 4,' but Table 4 lists only EDoRA's per-rank learning rates and classifier learning rates. Without the LoRA, DoRA, and LoRA-XS configurations and search budgets, it is impossible to rule out that the baselines were undertuned. On the small-data tasks (RTE, CoLA, MRPC), performance is highly sensitive to learning rate and epoch count, and EDoRA uses rates up to 5e-3 with 50 epochs on several tasks. Please report all baseline hyperparameters (or provide them via a public configuration file) and a description of the shared tuning protocol.
  2. [§4.2, Table 2] The claimed accuracy advantage is not accompanied by uncertainty quantification across the five seeds. The reported per-task standard deviations are large relative to several differences (e.g., EDoRA's RTE is 78.70±1.37 versus DoRA's 76.17±1.24, and EDoRA's MRPC 88.97±1.53 versus LoRA's 89.46±1.08), and the average advantage over DoRA is 0.70 points (84.48 vs 83.78). No significance tests or confidence intervals are reported, so it is unclear whether the cross-method differences are meaningful given seed noise. Report the mean and standard deviation of the per-method average over the five seeds, and perform paired tests (e.g., a paired t-test or Wilcoxon test) between EDoRA and each baseline, or provide effect sizes.
  3. [§3, Eq. (4)] The initialization of the magnitude vector m is not specified. In DoRA, m is initialized as the column-wise norm of W0, which is necessary for the initial model to equal the pretrained network. EDoRA states that 'Initializing A, B, and R as mentioned, ensures that the learning process starts with a model nearly identical to the pre-trained model,' but this is only true if m is also set appropriately. Without this detail, the method is not fully reproducible and the initialization claim is not substantiated. Please specify the initialization of m and confirm whether it is trainable.
  4. [§4.1–4.2] The paper does not state whether the reported GLUE numbers are development-set or test-set results, nor does it describe the model-selection protocol for choosing the rank. Table 2 presents results for ranks 4–32 for each method, and the text selects the best rank per method (e.g., LoRA rank 20, DoRA rank 20, EDoRA rank 32) without a held-out validation procedure. If ranks were selected based on the same numbers that are then compared, the comparison is subject to post-hoc selection effects. Please clarify the data split used and specify how the best rank was chosen (e.g., by dev accuracy).
minor comments (6)
  1. [Abstract] The sentence 'Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters' contains a subject-verb agreement error; 'reduces' should be 'reduce'.
  2. [§1, Contributions] The claim of 'over 45x fewer trainable parameters compared to LoRA and DoRA when applied to the GPT-3 model' is not directly supported by Table 1, which shows a 31.35x/32.33x reduction at r=16 and 59.08x/60.00x at r=32; please clarify the rank or specify that this is an average over the reported ranks.
  3. [§2.1] The sentence 'The author showed that SPOT substantially improves...' should be 'The authors showed...' because the reference is to Vu et al.
  4. [§2.3] The text 'Zhang et. al (Zhang et al., 2023) introduced AdaLoRA in 2023' has a formatting issue ('et. al' should be 'et al.') and the phrase 'in 2023' is redundant given the citation year.
  5. [§4.1] Please report the optimizer, batch size, sequence length, warmup steps, and exact model checkpoint (e.g., 'roberta-base' from HuggingFace), as these details affect reproducibility even when code is available.
  6. [§3.1, Eqs. (7)–(8)] The parameter-efficiency ratios are derived for a square weight matrix W ∈ R^{n×n}; for non-square weight matrices the ratios depend on the aspect ratio. Please state this generalization or clarify that the analysis is for a simplified square case.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EDoRA's proposed update is defined by its own equations, its parameter-efficiency ratios are arithmetic identities, and its GLUE results are measured rather than fitted.

full rationale

The derivation chain in Sections 3 and 3.1 is self-contained. Equation (4) defines EDoRA's update as W' = m (W0 + B R A) / ||W0 + B R A||_c, and Eqs. (5)-(6) specify the SVD-based initialization of A and B; these are architectural definitions, not circular reductions. The parameter-efficiency ratios in Eqs. (7)-(8) are arithmetic identities computed from the parameter-count definitions P_LoRA = 2nr, P_DoRA = n + 2nr, and P_EDoRA = n + r^2; they are not empirical predictions fitted to data. The central empirical claim in Table 2 is a measured comparison on GLUE using medians and standard deviations over five seeds, and no fitted parameter is renamed as a prediction. The 'learning pattern closer to full fine-tuning' narrative is borrowed from the external DoRA paper (Liu et al., 2024) as motivation for the decomposition; it is not load-bearing for the algebraic construction, and it is not a self-citation because no reference in the paper is authored by Nasiri or Garraghan. Concerns about baseline hyperparameter reporting, rank selection across the same benchmark, and the absence of significance testing are legitimate experimental-quality issues, but they do not make the derivation circular.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or model entities. It relies on standard SVD, a rank hyperparameter, and grid-searched training hyperparameters, plus two domain assumptions inherited from DoRA and LoRA-XS.

free parameters (5)
  • Rank r = 4 to 32 per experiment
    Chosen by hand; controls the size of the trainable matrix R and the number of singular vectors kept. Performance varies with r (Figure 3).
  • EDoRA learning rate = Per-task values in Table 4
    Grid-searched per task and rank; directly affects reported GLUE numbers.
  • Classifier learning rate = Per-task values in Table 4
    Grid-searched alongside EDoRA learning rate; affects reported results.
  • Epochs = 20 or 50 depending on task (Table 4)
    Chosen per task; part of the tuned configuration.
  • R initialization scale sigma = Unspecified small value
    Set to a small value; not quantified in the paper, affecting the starting update size.
assumptions (3)
  • domain assumption Top singular vectors of the column-normalized directional matrix form a good subspace for adaptation.
    Inherited from LoRA-XS and intrinsic dimensionality results (Aghajanyan et al. 2020); not proven for this parameterization.
  • domain assumption Decoupling magnitude from direction, as in DoRA, yields learning dynamics closer to full fine-tuning.
    Asserted in Sections 1 and 3 without measurement; the claimed behavioral benefit is not verified.
  • domain assumption GLUE benchmark results on RoBERTa-base generalize to other models and tasks.
    Only six GLUE tasks and one base model are tested; the abstract claims generality for LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition." pith.science (2026). https://pith.science/paper/I3MA2CSQ

@misc{pith2026250112067,
  author       = {Pith},
  title        = {Pith review of: EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I3MA2CSQ}},
  note         = {Machine review of arXiv:2501.12067}
}
read the original abstract

Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters. However, they often suffer from scalability issues and differences between their learning pattern and full fine-tuning. To overcome these limitations, we propose Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA): a novel PEFT method that decomposes pre-trained weights into magnitude and directional components. By freezing low-rank matrices, initializing them by singular value decomposition, and introducing a small trainable matrix between them, EDoRA achieves substantial reduction in trainable parameters while maintaining learning capacity. Experimental results on the GLUE benchmark demonstrate that EDoRA achieves competitive or superior performance compared to state-of-the-art methods, such as LoRA and DoRA, with up to 30x fewer trainable parameters. This makes EDoRA a highly efficient solution for adapting LLMs to diverse tasks under memory-constrained settings. Code is available at https://github.com/Hamid-Nasiri/EDoRA .

Figures

Figures reproduced from arXiv: 2501.12067 by the authors.

Figure 1
Figure 1. An overview of EDoRA The fine-tuned weight W′ can be updated incrementally by a low-rank decomposition: W′ = W0 + ∆W = W0 + BA (1) where W0 ∈ R m×n represents the pre-trained weight 3 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Relationship between the number of trainable parameters and average performance. 4 8 12 16 20 25 32 Rank 78 79 80 81 82 83 84 85 86 Average Performance (%) EDoRA LoRA DoRA LoRA-XS [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Impact of rank on average performance across different methods. and CoLA tasks, as reflected in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The impact of rank on average performance across differ￾ent methods. The number of trainable parameters is shown above the bar plots for reference. At each rank, EDoRA is compared with LoRA, DoRA and LoRA-XS. For clarity, EDoRA’s average performance is displayed alongs…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOST: Low-rank and Sparse Pre-training for Large Language Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    LOST pre-trains LLaMA models from 60M to 7B using an SVD-initialized low-rank plus channel-wise sparse decomposition and reports C4 perplexity at or below full-rank models with substantially smaller memory.

  2. Taming LLMs by Scaling Learning Rates with Gradient Grouping

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An optimizer wrapper that clusters per-layer momentum and scales learning rates by cluster-wise median deviations improves perplexity and accuracy across LLM and MLLM training, and lets LoRA pretraining approach full-...

  3. AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers

    cs.SE 2025-04 conditional novelty 5.0 of 10

    An LLM multi-agent framework for paper-to-code generation succeeded on all eight benchmarked ML papers and reached 49.2% on PaperBench Code-Dev, ahead of prior systems.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Aghajanyan, A., Zettlemoyer, L., and Gupta, S. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020

    arXiv 2012

  3. [3]

    Lora-xs: Low-rank adaptation with extremely small number of parameters

    Ba azy, K., Banaei, M., Aberer, K., and Tabor, J. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604, 2024

    arXiv 2024

  4. [4]

    B., Mann, B., Ryder, N., Subbiah, M., and Kaplan, J

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., and Kaplan, J. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  5. [5]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  6. [6]

    A survey on multimodal large language models for autonomous driving

    Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 958--979, 2024

    work page 2024

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  8. [8]

    P., Clark, J

    Edalati, A., Tahaei, M., Kobyzev, I., Nia, V. P., Clark, J. J., and Rezagholizadeh, M. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650, 2022

    arXiv 2022

Show all 32 references
  1. [9]

    Curlora: Stable llm continual fine-tuning and catastrophic forgetting mitigation

    Fawi, M. Curlora: Stable llm continual fine-tuning and catastrophic forgetting mitigation. arXiv preprint arXiv:2408.14572, 2024

    2024 arXiv

  2. [10]

    Sara: Singular-value based adaptive low-rank adaption

    Gu, J., Chen, S., Wang, Z., Zhang, Y., and Gong, P. Sara: Singular-value based adaptive low-rank adaption. arXiv preprint arXiv:2408.03290, 2024

    2024 arXiv

  3. [11]

    Parameter-efficient transfer learning for nlp

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp.\ 2790--2799. PMLR, 2019

    2019

  4. [12]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  5. [13]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Jin, P., Takanobu, R., Zhang, W., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13700--13710, 2024

    2024

  6. [14]

    A rank stabilization scaling factor for fine-tuning with lora

    Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732, 2023

    2023 arXiv

  7. [15]

    J., Blankevoort, T., and Asano, Y

    Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  8. [16]

    Albert: A lite bert for self-supervised learning of language representations

    Lan, Z. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019

    1909 arXiv

  9. [18]

    A medical multimodal large language model for future pandemics

    Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L., Liu, Z., Zheng, Y., Sun, X., et al. A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6 0 (1): 0 226, 2023

    2023

  10. [19]

    F., Cheng, K.-T., and Chen, M.-H

    Liu, S.-y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024

    2024

  11. [20]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  12. [21]

    Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., and Roth, D. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56 0 (2): 0 1--40, 2023

    2023

  13. [22]

    Adapterhub: A framework for adapting transformers

    Pfeiffer, J., R \"u ckl \'e , A., Poth, C., Kamath, A., Vuli \'c , I., Ruder, S., Cho, K., and Gurevych, I. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779, 2020

    2007 arXiv

  14. [23]

    Bidora: Bi-level optimization-based weight-decomposed low-rank adaptation

    Qin, P., Zhang, R., and Xie, P. Bidora: Bi-level optimization-based weight-decomposed low-rank adaptation. arXiv preprint arXiv:2410.09758, 2024

    2024 arXiv

  15. [24]

    and Kingma, D

    Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016

    2016

  16. [25]

    Sung, Y.-L., Nair, V., and Raffel, C. A. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34: 0 24193--24205, 2021

    2021

  17. [26]

    J., Ting, D

    Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W. Large language models in medicine. Nature medicine, 29 0 (8): 0 1930--1940, 2023

    1930

  18. [27]

    SP o T : Better frozen model adaptation through soft prompt transfer

    Vu, T., Lester, B., Constant, N., Al-Rfou, R., and Cer, D. SP o T : Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 5039--5059, Dublin, Ireland...

    2022 doi

  19. [28]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: a multi-task benchmark and analysis platform for natural language understanding. corr abs/1804.07461 (2018). arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  20. [29]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  21. [30]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  22. [31]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  23. [32]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  24. [33]

    SP o T : Better Frozen Model Adaptation through Soft Prompt Transfer

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2018 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.