Pith. sign in

REVIEW 5 major objections 6 minor 48 references

TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TriAdaptLoRA claims consistent gains over prior adaptive-rank PEFT methods by growing ranks where a cheap Frobenius-norm change score indicates importance, reporting 89.34 average GLUE and 85.82 EM on SQuAD 2.0.

desk verdict The triangular-split rank-growth idea is a real variant, but the headline GLUE number is a per-task best-of-two blend, and the paper needs code and per-seed data before the central claim holds. read the letter →

arxiv 2501.08008 v1 pith:QGIS7CQP submitted 2025-01-14 cs.CL cs.AI

Yao Liang , Yuwei Wang , Yi Zeng This is my paper · ORCID
classification cs.CLcs.AI
keywords parameter-efficientfine-tuninglow-rankadaptationadaptiverankgrowthdynamicthresholdtriangulardecompositionFrobeniusnormimportanceGLUEbenchmarkSQuAD2.0
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that parameter-efficient fine-tuning can be improved by deciding, during training, which low-rank incremental matrices deserve a larger rank. It introduces TriAdaptLoRA, which writes each incremental matrix as $\Delta W = B(L+U)A$ with $L$ lower and $U$ upper triangular, so rank can grow by adding rows to $L$ and columns to $U$. An importance score based on the change in the rank-normalized Frobenius norm of $D=L+U$ picks which matrices receive rank increments, and a dynamic threshold decides how many matrices are updated at each step. On GLUE with DeBERTaV3-base the method reports an average of 89.34, above AdaLoRA (88.99) and IncreLoRA (88.90), and it also reports gains on SQuAD 2.0. The paper concludes that adaptive rank growth with this cheap importance proxy is both more accurate and cheaper than prior adaptive-rank methods.

What carries the argument

The load-bearing object is the dense triangular transformation matrix $D=L+U$ inside each low-rank increment, with $\Delta W = B D A$. $L$ and $U$ are lower and upper triangular matrices that are augmented in opposite directions (rows to $L$, columns to $U$), letting the rank grow while reusing existing parameters. The importance metric in Equation 11 turns the change in the rank-normalized Frobenius norm of $D$ into a scalar score; the adaptive threshold in Equations 13 and 14 converts a remaining rank budget into the number $k(t)$ of matrices that get expanded. An orthogonality penalty on $A$ and $B$ (Equation 10) and a scaling factor $\alpha/(r+\epsilon)$ stabilize training as the rank grows.

What would settle it

Replace the importance score in Equation 11 with random noise drawn on the same scale while keeping all other components fixed, and evaluate on MNLI: the central claim predicts a clear drop below the reported 90.64 accuracy, whereas a null result would show the importance ranking is not load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that the triangular transformation matrix $D=L+U$ is a sufficient and efficient carrier of rank adaptation. The paper argues that tracking the normalized Frobenius norm of $D$ over training, $S_m^{(t)} = \|L_m^{(t)}+U_m^{(t)}\|_F / r_m^{(t)} - \|L_m^{(t-1)}+U_m^{(t-1)}\|_F / r_m^{(t-1)}$, identifies which incremental matrices benefit most from extra rank, and that a budget-driven threshold (linear or nonlinear) should control how many matrices grow at each step. The reported experiments support the claim: TriAdaptLoRA averages 89.34 on GLUE versus 88.99 for AdaLoRA and 88.90 for IncreLoRA, and reaches 85.82 EM / 88.90 F1 on SQuAD 2.0 versus 85.56 / 88.66 for IncreLoRA. The authors further claim reduced computational overhead, because importance evaluation costs $O(r^2 M T)$ instead of $O((rn+dr)MT)$, and no need for task-specific tuning of the growth threshold.

Load-bearing premise

The load-bearing premise is that the change in the rank-normalized Frobenius norm of the triangular matrix $D$ correctly identifies which incremental matrices should receive more rank; if this proxy does not track true task-relevant importance, the adaptive allocation advantage disappears.

Editorial extensions

If this is right

  • Adaptive rank growth under a dynamic threshold removes the need to hand-tune the fixed $k$ used by IncreLoRA, while keeping the total rank budget bounded by $R_0$.
  • The $O(r^2 M T)$ importance evaluation makes rank adjustment cheap enough to run periodically on large models, cutting the per-step gradient-based scoring used by AdaLoRA and IncreLoRA.
  • The triangular split enables bidirectional rank expansion, so parameters are added both as new rows of $L$ and new columns of $U$, which the authors argue yields more uniform and continuous final rank distributions across layers.
  • On the reported benchmarks the method is claimed to consistently outperform the compared PEFT baselines, including full fine-tuning on SQuAD 2.0 under the nonlinear threshold mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The triangular split roughly doubles the number of trainable entries per rank unit compared with a plain $BA$ update, so a comparison at equal nominal rank is not parameter-matched; re-evaluating at equal parameter counts would clarify whether the gain comes from the allocation policy or from more parameters.
  • Because the importance score uses only the Frobenius norm of $D$, it is architecture-agnostic and could be transferred to low-rank adapters in other domains, such as vision or multimodal models, without modification.
  • The dynamic threshold can be read as a form of budget scheduling; comparing it against simple cosine or exponential schedules of $k(t)$ would separate the effect of the importance ranking from the effect of the schedule shape.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes TriAdaptLoRA, a parameter-efficient fine-tuning method that represents the incremental update as ∆W = B(L+U)A, where L and U are lower/upper triangular matrices and D = L+U is a dense transformation in the low-rank subspace. The rank of each incremental matrix is grown during training according to an importance score (Equation 11) and a dynamic budget governed by either a linear or nonlinear threshold (Equations 12–14). The authors evaluate TriAdaptLoRA on eight GLUE tasks with DeBERTaV3-base and on SQuAD 2.0, comparing against full fine-tuning, BitFit, adapters, LoRA, AdaLoRA, and IncreLoRA, and they include ablations on the orthogonality constraint, Frobenius-norm normalization, warm-up steps, rank-update interval, and reference rank. The headline claim is that TriAdaptLoRA consistently outperforms existing PEFT methods while reducing computational overhead.

Significance. The triangular split with adaptive rank growth is a plausible and potentially useful design, and the paper gives a fairly complete algorithmic description of the proposed procedure. The experimental protocol is partly thorough: three-seed averages are reported for GLUE, and the paper includes ablations and sensitivity analyses on RTE. However, the reported evidence does not currently support the central claim of consistent superiority. The best GLUE average comes from a row that matches neither of the two defined configurations, both defined modes lose to AdaLoRA on five of eight tasks, the SQuAD 2.0 results have no variance, the computational savings are theoretical rather than measured, and no code is provided. If the authors report the two defined configurations honestly and reword the claims, the method may still be of interest as a simple heuristic alternative with fewer AdaLoRA-style hyperparameters, but the paper as written overstates its results.

major comments (5)
  1. [Table I, §IV-B3] The row labeled 'TriAdaptLoRA' with average 89.34 is not a configuration defined in Sections III-D or IV-B3, and no single run can reproduce it. Its entries coincide with the Non-Linear row on MNLI (90.64), SST-2 (95.68), CoLA (71.6), and STS-B (91.79), and with the Linear row on QQP (92.09), QNLI (94.37), RTE (87.84), and MRPC (90.77). This is a per-task best-of-two selection, not a method result. Moreover, both defined configurations underperform AdaLoRA on five of eight tasks (Linear on MNLI, SST-2, QQP, QNLI, and STS-B; Non-Linear on MNLI, SST-2, QQP, QNLI, and RTE). The abstract's claim that TriAdaptLoRA 'consistently outperforms existing PEFT methods' and 'achieves superior performance' is therefore not supported by Table I. Please report each configuration separately, remove the blended row, and reword the claims accordingly.
  2. [Tables I, II, IV] Several entries report exact zero standard deviations from three-seed averages, for example AdaLoRA MNLI 90.66±0, TriAdaptLoRA QQP/QNLI 92.09±0/94.37±0, and TriAdaptLoRA (Non-Linear) CoLA 71.6±0. Exact zero standard deviations across tasks are implausible and likely reflect rounding or reporting artifacts, and they cannot support the abstract's 'enhanced stability' claim. Table IV reports SQuAD 2.0 results with no standard deviations, number of seeds, or error bars, even though Section IV-B2 promises three-seed averages for GLUE. Please provide full variance and seed information for every reported result, or state the number of runs per table.
  3. [§III-C, §V] The computational-overhead claim is theoretical only. Section III-C derives arithmetic complexity O(r^2 MT) versus O((rn+dr)MT), and Section V concludes that TriAdaptLoRA 'reduces energy consumption,' but no wall-clock time, FLOPs, peak memory, or energy measurements are reported anywhere in the paper. The abstract's 'reduced computational overhead' is therefore not empirically validated. Please add measured runtime/memory/energy comparisons or soften the claim.
  4. [§III-D, Algorithm 1] The written rank-growth algorithm does not implement the intended warm-up delay. In Algorithm 1, rank growth is performed at every step t with R(t−1)>0, and Equation 13 uses α(t)=(t−t0)/(T−t0). For t<t0, α(t) is negative, so k(t) becomes negative for the linear mode and the lower bound k(t)=max(k(t),1) forces one rank increment per step before warm-up. The nonlinear mode similarly yields k(t)=1 for t<t0. Thus the 'warm-up steps' t0 do not actually delay rank growth as described, which contradicts the sensitivity analysis in Section IV-D1. Please add an explicit t≥t0 condition (or the intended equivalent) and clarify how warm-up was implemented in the reported experiments.
  5. [§III-E, Eq. (11)] The theoretical justification of the importance metric is a loose motivation rather than a proof. Section III-E shows that D=L+U appears in the gradient expressions, but it does not establish that the normalized Frobenius-norm change in Equation 11 tracks task-relevant parameter importance; any differentiable statistic of D would appear in a chain-rule expansion. Since the adaptive rank-allocation advantage of TriAdaptLoRA rests entirely on this heuristic, either the derivation should be made rigorous or the paper should provide direct empirical validation, for example by comparing the importance ranks with held-out performance changes when individual matrices are expanded.
minor comments (6)
  1. [§III-D, Table I] The 'Fixed-k' variant is used in Table I and Section IV-B4 but is never defined in the method section; please provide its definition or remove it.
  2. [§IV-C] SQuAD 2.0 is described as a natural language generation task, but it is an extractive question-answering benchmark; please adjust the task taxonomy.
  3. [Appendix A, Table V] Table V contains formatting/typographical issues, including 'def ault' in the LoRA Dropout row and an unclear grouping despite the text saying 'the first fourteen are shared hyperparameters'; please reformat the table for clarity.
  4. [Appendix B, Table VII] The MRPC dev-set size is listed as 408k, which is inconsistent with the actual MRPC dev set of 408 sentence pairs; this appears to be a typo.
  5. [Reproducibility] No code or data release is mentioned; providing a link to an implementation would materially improve reproducibility, especially for the exact rank-growth schedules and the initialization of new rows/columns during expansion.
  6. [§III-B, §III-D] The symbol α is used both for the scaling factor in Equation 5 and for the time-varying threshold function α(t) in Equations 13–14; please use distinct notations to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the importance metric, rank-growth rule, and orthogonality constraint are internal heuristics benchmarked against external baselines, not results forced by their own definitions.

full rationale

TriAdaptLoRA is an empirically evaluated PEFT method; its claims are benchmark results, not derivations from its own assumptions. The importance score S(t)_m (Eq. 11) is defined as a change in rank-normalized Frobenius norm of D = L + U and is used inside training to choose which incremental matrices gain rank; it is a heuristic allocation rule, not a fitted parameter later renamed as a prediction. The adaptive threshold rules k(t) (Eqs. 13-14) and the top-k selection via S_theta are algorithm definitions, not self-validating predictions. The orthogonality regularizer (Eq. 10) is explicitly borrowed from AdaLoRA [21], and the rank-growth mechanism from IncreLoRA [22]; both are external prior work, and neither is invoked as a uniqueness theorem. The Section III-E gradient analysis is informal motivation rather than a derivation that presupposes its conclusion. The reported GLUE and SQuAD numbers are measured against external baselines. One reporting concern, not a circularity: Table I's row labeled 'TriAdaptLoRA' is a per-task blend of the Linear and Non-Linear rows (e.g., MNLI/SST-2/STS-B match Non-Linear, while QQP/QNLI/RTE/MRPC match Linear), so the headline 'consistently outperforms' rests on an undefined aggregate configuration; this is an empirical-support/aggregation issue, not a circular derivation. Similarly, selecting the F-norm/rank normalization after seeing Table III is test-set peeking, not circularity. Because no prediction or first-principles result reduces to its own input, the circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method depends on several tuned hyperparameters and on heuristic modeling assumptions. The importance metric, the Gaussian initialization of newly added B columns, and the fairness of the baseline comparison are not independently verified.

free parameters (6)
  • reference rank rref = 8 (all tasks)
    Sets the total rank budget R0 = rref * M; chosen per task in hyperparameter search (Table V).
  • alpha (scaling factor) = 16 or 32 depending on task
    Controls the magnitude of the update; tuned per task (Table V).
  • rank increment Delta r = 1 or 4 depending on task
    The number of new rows/columns added at each rank growth step; per-task hyperparameter (Table V).
  • warm-up steps t0 = 600 to 8000 depending on task
    Determines the start of rank growth in the linear/nonlinear threshold schedule; tuned per task (Table V).
  • rank update interval = 100 to 1000 steps depending on task
    Period between rank growth operations; tuned per task (Table V).
  • orthogonality coefficient = 0.1 or 0.3 depending on task
    Regularization weight for R(A,B) from AdaLoRA; tuned per task (Table V).
assumptions (4)
  • domain assumption The change in normalized Frobenius norm of D = L + U is a valid importance signal for rank allocation.
    Section III-C, Eq. 11; stated as a 'positing' without proof. Section III-E gives a heuristic gradient argument.
  • ad hoc to paper Growing the rank by appending rows/columns to A, B, L, U and initializing new entries with Gaussian noise preserves training stability, especially the Gaussian initialization of new B columns.
    Section III-D says new rows/columns are Gaussian-initialized, despite B initially zero; no analysis of the resulting forward-pass jump.
  • domain assumption The low-rank class {B(L+U)A} with r growing covers the practical adaptation subspace of LoRA/AdaLoRA/IncreLoRA for the tested tasks.
    Section III-B; mathematically true for fixed r (any rank-r matrix can be written this way), but the dynamic growth protocol changes the optimization landscape.
  • domain assumption The comparison protocol matches that of Zhang et al. (2023) [22] (IncreLoRA) for all methods.
    Section IV-B2; unclear if baselines were rerun under identical conditions, affecting the fairness of the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning." pith.science (2026). https://pith.science/paper/QGIS7CQP

@misc{pith2026250108008,
  author       = {Pith},
  title        = {Pith review of: TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGIS7CQP}},
  note         = {Machine review of arXiv:2501.08008}
}
read the original abstract

The fine-tuning of Large Language Models (LLMs) is pivotal for achieving optimal performance across diverse downstream tasks. However, while full fine-tuning delivers superior results, it entails significant computational and resource costs. Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, address these challenges by reducing the number of trainable parameters, but they often struggle with rank adjustment efficiency and task-specific adaptability. We propose Triangular Adaptive Low-Rank Adaptation (TriAdaptLoRA), a novel PEFT framework inspired by neuroscience principles, which dynamically optimizes the allocation of trainable parameters. TriAdaptLoRA introduces three key innovations: 1) a triangular split of transformation matrices into lower and upper triangular components to maximize parameter utilization, 2) a parameter importance metric based on normalized Frobenius norms for efficient adaptation, and 3) an adaptive rank-growth strategy governed by dynamic thresholds, allowing flexible parameter allocation across training steps. Experiments conducted on a variety of natural language understanding and generation tasks demonstrate that TriAdaptLoRA consistently outperforms existing PEFT methods. It achieves superior performance, enhanced stability, and reduced computational overhead, particularly under linear threshold-driven rank growth. These results highlight its efficacy as a scalable and resource-efficient solution for fine-tuning LLMs.

Figures

Figures reproduced from arXiv: 2501.08008 by the authors.

Figure 1
Figure 1. Schematic diagram of TriAdaptLoRA’s overall structure. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Heatmaps illustrate the final rank distributions. These distributions are shown for each layer and weight matrix of the TriAdaptLoRA, AdaLoRA, and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Experimental results on the RTE task with different initial warm-up [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Experimental results on the RTE task with different rank update inter [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Experimental results on the RTE task with different reference rank [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 20 canonical work pages

  1. [1]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,”Advances in Neural Information Processing Systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  2. [2]

    Parameter-efficient fine-tuning of large- scale pre-trained language models,

    N. Ding, Y . Qin, G. Yang, F. Wei, Z. Yang, Y . Su, S. Hu, Y . Chen, C.-M. Chan, W. Chen et al. , “Parameter-efficient fine-tuning of large- scale pre-trained language models,” Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023

    2023

  3. [3]

    Large language models are built-in autoregressive search engines,

    N. Ziems, W. Yu, Z. Zhang, and M. Jiang, “Large language models are built-in autoregressive search engines,” in Findings of the Association for Computational Linguistics: ACL 2023 , A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 2666–2678. [Online]. Available: https://aclanthology...

    work page 2023

  4. [4]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  5. [5]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  6. [6]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  7. [7]

    A general language assistant as a laboratory for alignment,

    A. Askell, Y . Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma et al., “A general language assistant as a laboratory for alignment,” arXiv preprint arXiv:2112.00861 , 2021

    arXiv 2021

  8. [8]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” arXiv preprint arXiv:2109.01652 , 2021

    arXiv 2021

Show all 48 references
  1. [9]

    Metaicl: Learning to learn in context,

    S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi, “Metaicl: Learning to learn in context,” arXiv preprint arXiv:2110.15943 , 2021

    2021 arXiv

  2. [10]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022

    1950

  3. [11]

    Pre-trained models for natural language processing: A survey,

    X. Qiu, T. Sun, Y . Xu, Y . Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” Science China Technological Sciences, vol. 63, no. 10, pp. 1872–1897, 2020

    2020

  4. [12]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020

    2020

  5. [13]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  6. [14]

    Dora: Weight-decomposed low-rank adapta- tion,

    S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adapta- tion,” arXiv preprint arXiv:2402.09353 , 2024

    2024 arXiv

  7. [15]

    VeRA: Vector- based Random Matrix Adaptation,

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano, “VeRA: Vector- based Random Matrix Adaptation,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=NjNfLdxr3A

    2024

  8. [16]

    AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models,

    Z. Liu, S. Kundu, A. Li, J. Wan, L. Jiang, and P. Beerel, “AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),...

    2024

  9. [17]

    PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA,

    S. Wang, B. Xue, J. Ye, J. Jiang, L. Chen, L. Kong, and C. Wu, “PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L.-W. Ku, A. Martins, and V ....

    2024

  10. [18]

    Longlora: Efficient fine-tuning of long-context large language models,

    Y . Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “Longlora: Efficient fine-tuning of long-context large language models,” arXiv preprint arXiv:2309.12307, 2023

    2023 arXiv

  11. [19]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  12. [20]

    LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks,

    H. Wang, B. Ping, S. Wang, X. Han, Y . Chen, Z. Liu, and M. Sun, “LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L.-W. Ku, A. Ma...

    2024

  13. [21]

    Adaptive budget allocation for parameter-efficient fine-tuning,

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” arXiv preprint arXiv:2303.10512 , 2023

    2023 arXiv

  14. [22]

    Increlora: Incremental parameter allocation method for parameter- efficient fine-tuning,

    F. F. Zhang, L. Li, J.-C. Chen, Z. Jiang, B. Wang, and Y . Qian, “Increlora: Incremental parameter allocation method for parameter- efficient fine-tuning,” ArXiv, vol. abs/2308.12043, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:261076438

    2023 arXiv

  15. [23]

    Increasing model capacity for free: A simple strategy for parameter efficient fine-tuning,

    H. SONG, H. Zhao, S. Majumder, and T. Lin, “Increasing model capacity for free: A simple strategy for parameter efficient fine-tuning,” in The Twelfth International Conference on Learning Representations ,

  16. [24]

    D. O. Hebb, The organization of behavior: A neuropsychological theory. Psychology press, 2005

    2005

  17. [25]

    Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,” arXiv preprint arXiv:1804.07461 , 2018

    2018 arXiv

  18. [26]

    Know what you don’t know: Unanswerable questions for SQuAD,

    P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), I. Gurevych and Y . Miyao, Eds. Melbourne, Australia: Associat...

    2018

  19. [27]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  20. [28]

    DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation,

    M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics ....

    2023

  21. [29]

    Reinforcement learning improves behaviour from evaluative feedback,

    M. L. Littman, “Reinforcement learning improves behaviour from evaluative feedback,” Nature, vol. 521, no. 7553, pp. 445–451, 2015, publisher: Nature Publishing Group UK London

    2015

  22. [30]

    Control of synaptic plasticity in deep cortical networks,

    P. R. Roelfsema and A. Holtmaat, “Control of synaptic plasticity in deep cortical networks,” Nature Reviews Neuroscience , vol. 19, no. 3, pp. 166–180, Mar. 2018. [Online]. Available: https://www.nature.com/ articles/nrn.2018.6

    2018

  23. [31]

    Reinforcement learning: An introduction,

    R. S. Sutton, “Reinforcement learning: An introduction,” A Bradford Book, 2018

    2018

  24. [32]

    DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,

    P. He, J. Gao, and W. Chen, “DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/ forum?id=sE7-XhLxHA

    2023

  25. [33]

    Learning multiple visual domains with residual adapters,

    S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual domains with residual adapters,” Advances in neural information pro- cessing systems, vol. 30, 2017

    2017

  26. [34]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International Conference on Machine Learning . PMLR, 2019, pp. 2790–2799. 13

    2019

  27. [35]

    Towards a unified view of parameter-efficient transfer learning,

    J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=0RDcd5Axok

    2022

  28. [36]

    Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, S. Ravfogel, and Y . Goldberg, “Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” arXiv preprint arXiv:2106.10199 , 2021

    2021

  29. [37]

    Neural network accept- ability judgments,

    A. Warstadt, A. Singh, and S. R. Bowman, “Neural network accept- ability judgments,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 625–641, 2019

    2019

  30. [38]

    Comparison of the predicted and observed secondary structure of t4 phage lysozyme,

    B. W. Matthews, “Comparison of the predicted and observed secondary structure of t4 phage lysozyme,” Biochimica et Biophysica Acta (BBA)- Protein Structure, vol. 405, no. 2, pp. 442–451, 1975

    1975

  31. [39]

    A broad-coverage challenge corpus for sentence understanding through inference,

    A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” arXiv preprint arXiv:1704.05426, 2017

    2017 arXiv

  32. [40]

    Automatically constructing a corpus of sen- tential paraphrases,

    B. Dolan and C. Brockett, “Automatically constructing a corpus of sen- tential paraphrases,” in Third International Workshop on Paraphrasing (IWP2005), 2005

    2005

  33. [41]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  34. [42]

    The pascal recognising textual entailment challenge,

    I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in Machine learning challenges workshop. Springer, 2005, pp. 177–190

    2005

  35. [43]

    The second pascal recognising textual entailment challenge,

    R. B. Haim, I. Dagan, B. Dolan, L. Ferro, D. Giampiccolo, B. Magnini, and I. Szpektor, “The second pascal recognising textual entailment challenge,” in Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , vol. 7, 2006, pp. 785–794

    2006

  36. [44]

    The third pascal recognizing textual entailment challenge,

    D. Giampiccolo, B. Magnini, I. Dagan, and W. B. Dolan, “The third pascal recognizing textual entailment challenge,” in Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing , 2007, pp. 1–9

    2007

  37. [45]

    The fifth pascal recognizing textual entailment challenge

    L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo, “The fifth pascal recognizing textual entailment challenge.” TAC, vol. 7, p. 8, 2009

    2009

  38. [46]

    Recursive deep models for semantic compositionality over a sentiment treebank,

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y . Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proceedings of the 2013 conference on empirical methods in natural language processing , 2013, pp. 1631–1642

    2013

  39. [47]

    Semeval- 2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation,

    D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “Semeval- 2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation,” arXiv preprint arXiv:1708.00055 , 2017

    2017 arXiv

  40. [2024]

    Available: https://openreview.net/forum?id=H3IUunLy8s

    [Online]. Available: https://openreview.net/forum?id=H3IUunLy8s

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.