Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

The paper introduces L1RA, a LoRA variant that prunes redundant adapter ranks and reallocates them within a fixed budget, and claims it reaches lower perplexity than LoRA and AdaLoRA at nearly the same training cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:47 UTC pith:GKKFWAAV

load-bearing objection L1RA is a legitimate but modest extension of LoRA rank pruning—runtime reallocation of freed ranks—and its main empirical claim is confounded by the parameter-count increase that reallocation to FFN layers causes. the 3 major comments →

arxiv 2509.04884 v1 pith:GKKFWAAV submitted 2025-09-05 cs.CL cs.PF

L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning

Raul Singh , Nicolo Brunello , Vincenzo Scotti , Mark James Carman This is my paper
classification cs.CL cs.PF
keywords Low-Rank Adaptation (LoRA)dynamic rank allocationL1 regularizationparameter-efficient fine-tuningLLM fine-tuningmemory budget estimationperplexityfeed-forward layers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

L1RA is a fine-tuning method that keeps the total LoRA rank budget fixed while letting individual adapters change size during training. Every adapter gets a small gate vector c, and the L1 penalty on c drives some gates to zero; when that happens, the rank is pruned and immediately given to another adapter—specifically the one whose gate values are farthest from zero. The paper claims that, for assistant fine-tuning on models like Llama 3.1 8B and Mistral 7B v0.3, this reuse of capacity yields lower perplexity than vanilla LoRA and AdaLoRA while keeping training time within about 1% of vanilla LoRA and peak memory within a few percent. If that holds, a user only has to choose a total rank or memory budget, and the method decides where capacity is actually needed. The same experiments also expose a consistent diagnostic pattern: rank flows toward the feed-forward projections and the attention output projection, mostly in the upper layers.

Core claim

The paper claims that L1RA—LoRA with an extra per-rank gate vector c and an L1 penalty λ∥c∥1 on each adapter—can find a better use of a fixed rank budget than fixing all adapter ranks equal. During training, a gate component shrinking to zero triggers pruning of that rank, and the freed rank is immediately reallocated to the adapter with the largest minimum gate value, interpreted as the one most in need. In assistant fine-tuning experiments on Llama 3.1 8B and Mistral 7B v0.3, L1RA achieves the lowest perplexity among LoRA, AdaLoRA, and L1RA, with training time within about 1% of vanilla LoRA and peak GPU memory within roughly 3% (14.23 GB vs 13.84 GB on Llama; 13.94 GB vs 13.58 GB on Mistr

What carries the argument

The central object is the L1RA adapter, a LoRA adapter with an extra gate vector c between the A and B matrices, so the adapter output is x·(A·diag(c)·B). The L1 penalty λ∥c∥1 drives entries of c to zero; when an entry hits zero, the corresponding column of A and row of B are pruned and that rank becomes available. A reallocation loop then gives each spare rank to the unpruned adapter with the largest minimum gate value, keeping the sum of ranks fixed while letting the distribution shift. This prune-and-reallocate cycle is what carries the argument: it is the mechanism that turns a fixed rank budget into a dynamic, need-based allocation without starting from inflated ranks. A companion memor

Load-bearing premise

The claim rests on the assumption that the performance gain comes from dynamic rank reallocation rather than from L1RA's larger final trainable parameter count, since no LoRA baseline matches L1RA's final per-matrix ranks.

What would settle it

Train vanilla LoRA with the same final adapter architecture found by L1RA (same per-layer, per-matrix ranks, same parameter count) and the same total budget; if that static LoRA matches L1RA's perplexity, the reallocation mechanism adds nothing. Alternatively, run L1RA with the reallocation loop disabled: if perplexity is unchanged, reusing ranks is not where the benefit lies.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If L1RA is right, a practitioner does not need to guess per-layer ranks; setting a single total rank budget and letting L1RA reallocate is enough to match or beat uniform LoRA, at least on assistant fine-tuning.
  • The measured training-time overhead of less than 1% over vanilla LoRA means the dynamic pruning and reallocation machinery is cheap enough not to negate LoRA's efficiency benefits.
  • Starting from the budget rank rather than from an inflated rank, as AdaLoRA/SORA-style methods require, means L1RA can fit in the same memory envelope as a vanilla LoRA run, which matters on consumer GPUs where the margin is a few GB.
  • The rank distribution results suggest a static prior: bias LoRA rank toward Wup, Wgate, Wdown, and upper layers; L1RA's diagnostics identify which modules actually need adaptation and can guide manual allocation when the dynamic method is not available.
  • Paired with MEMORY-GELATO's peak-memory estimates, the budget can be set before training so that the run fits the GPU, with L1RA absorbing the rank allocation decisions inside that budget.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The final adapter parameter counts are not matched: L1RA ends with 45.16M/50.06M trainable parameters versus 41.94M for vanilla LoRA, because freed ranks land in larger feed-forward matrices. The paper does not run a LoRA baseline with those final per-layer ranks, so part of the perplexity gain may be a parameter-count effect rather than an effect of dynamic reallocation.
  • A direct ablation would disable reallocation—prune ranks but discard the freed budget—to measure how much of the gain actually comes from reusing capacity; the paper does not include this comparison.
  • The sorting heuristic (give a spare rank to the adapter with the largest minimum gate value) is plausible but unvalidated against alternatives; comparing it to gradient-based sensitivity or Fisher-weighted allocation would show whether it is near the optimal rank assignment.
  • If the FFN-heavy, output-heavy rank pattern generalizes across tasks, a static FFN-heavy rank allocation with matched parameter count may capture most of L1RA's gain without any dynamic machinery; this can be tested on the same OpenOrca setup.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces L1RA, a LoRA variant that uses an L1-regularized per-adapter gate vector c to prune low-importance ranks during fine-tuning and reallocate the freed ranks to other adapters, subject to a fixed total rank budget. The authors also present MEMORY-GELATO, a GPU memory estimator used to set the starting rank. The main empirical claim is that L1RA achieves same or better perplexity than vanilla LoRA and AdaLoRA on OpenOrca instruction fine-tuning of two 4-bit quantized 7–8B models, with comparable training time and memory. The paper further reports post-training rank distributions, observing that FFN and output-projection matrices receive more rank than attention matrices, especially in later layers. The rank-budget framing is central: L1RA is claimed to improve performance by moving capacity within a fixed rank budget rather than by increasing it.

Significance. If the central claim were established, L1RA would be a practically useful method for resource-constrained fine-tuning: it offers a simple mechanism for dynamic rank reallocation, with open-source code and a validated memory estimator. The rank-distribution analysis also provides interpretability evidence consistent with prior findings on FFN layers. However, the empirical support is currently too thin and internally confounded to support the abstract's claims. The rank budget is conserved, but the parameter budget is not, because freed ranks are placed into higher-dimensional FFN matrices; the paper's own Section 8 explicitly acknowledges this. The performance gain over vanilla LoRA could therefore reflect the larger final adapter parameter count rather than the dynamic reallocation policy. The single-run perplexity comparisons also lack error bars, and the memory measurements are admitted in Section 8 to be unreliable due to suspected trainer-side offloading. The strengths—code release, clear algorithmic description, and the MEMORY-GELATO validation—are real, but the core efficiency claim needs substantially stronger evidence.

major comments (3)
  1. [Tables 2–3; Section 8] The central claim of 'same or better performances' within a fixed rank budget is confounded by final parameter count. L1RA ends with 45.16M (Llama) and 50.06M (Mistral) trainable adapter parameters versus 41.94M for vanilla LoRA—increases of 7.68% and 19.36% (Table 3). Section 8 explains this is because reallocated ranks go to FFN matrices with roughly 4× higher inner dimensionality. Thus the rank budget is conserved, but the parameter budget is not. The observed PPL improvements (-2.11% and -0.68%, Table 3) could be due to the extra capacity in high-dimensional layers rather than to the dynamic reallocation mechanism. The paper lacks a vanilla LoRA baseline with matched final parameter count, a static FFN-heavy LoRA allocation, or an ablation that replaces L1RA's reallocation with pruning-only or random reallocation. Such an ablation is necessary to isolate the contribution of the algor
  2. [Section 7, Table 2; Limitations] The empirical comparison rests on a single run for each configuration. No seeds, error bars, or significance tests are reported. The L1RA-vs-LoRA PPL differences are 2.11% (Llama) and 0.68% (Mistral) on one dataset (OpenOrca), and the Mistral difference is particularly small. The paper's own Limitations section concedes that only two models and one task were evaluated. This is not sufficient to support the abstract's 'series of comprehensive experiments' or the claim that L1RA 'achieves same or better performances.' Multiple seeds and ideally downstream task metrics are needed; at minimum, the variance across runs should be reported.
  3. [Section 8; Table 2 footnote 2] The claimed 'comparable or even reduced computational overhead' is not supported by the reported memory data. Section 8 states that the measured memory consumption 'does not comply with our expectations' and suspects internal optimization or offloading by the HuggingFace trainer; it further reports that a small handmade training loop produced higher memory consumption 'more in line with the number of parameters.' This admission means the memory comparisons in Table 2 are unreliable. In addition, the AdaLoRA PPL values are flagged as 'slightly altered' because PPL was computed from a loss that included the regularisation term, and the separate corrected computations are not reported. The comparison against AdaLoRA should either be based on clean PPL numbers or removed until those numbers are provided.
minor comments (5)
  1. [Algorithm 1; Section 4] After reallocation, Algorithm 1 normalizes the c vector by its sum (c ← c / Σc). This operation is not motivated and may interact with the L1 regularisation strength λ. Please clarify why it is needed and how it affects training dynamics.
  2. [Equation (3)] Equation (3) is written as '∆W = ...', but the text describes a regularisation term L_SVD. This is likely a typo; the equation should define L_SVD.
  3. [Section 7] Minor typos: 'we can the average evolution' and 'wee how' appear in the results discussion. Also, 'finsert' and 'freallocate' are written inconsistently across the text and pseudocode.
  4. [Section 2; Section 6] SORA's abbreviation is not expanded in the text, and VERA and DYLORA are described but not included in the experimental comparison. A sentence explaining why only AdaLoRA and vanilla LoRA are compared would improve clarity.
  5. [Figure 9] The rank heatmaps in Figure 9 are dense and hard to read, especially for 32 layers × 7 matrices. Consider providing summary statistics or a cleaner visualization format.

Circularity Check

0 steps flagged

No significant circularity: L1RA is an empirical algorithm evaluated against external baselines; the rank-budget/parameter-count confound is a validity issue, not a derivation-level circularity.

full rationale

L1RA is presented as an algorithm, not as a derived theorem, and no load-bearing step reduces to its own inputs by construction. The core mechanism (Algorithm 1, Eq. 4) defines how ranks are pruned via L1 regularization on a gating vector c and reallocated to unpruned adapters; the final rank distribution is an output of training, not an input. The claimed 'insights' from Figures 6–9 are descriptive observations about where ranks landed, and the paper explicitly corroborates them with external work (Geva et al., 2021; Biderman et al., 2023), not with self-citations. The MEMORY-GELATO tool is validated against independently measured peak memory usage (Table 1), and its estimates are not used to generate the perplexity results. The main weakness—that L1RA attains lower PPL while ending with more trainable adapter parameters than vanilla LoRA (45.16M/50.06M vs 41.94M in Table 2)—is an experimental confound, not a circular derivation: the paper does not fit a parameter and then rename that fit as a prediction. Section 8 openly acknowledges the parameter-count increase and even notes that measured memory consumption does not match parameter counts, and the Limitations section discloses the narrow evaluation scope. These caveats weaken the empirical strength of the central claim but do not make the claim true by definition or by self-citation. No uniqueness theorem, ansatz, or load-bearing self-citation is used to force the results. Accordingly, the paper is self-contained against external benchmarks and receives a circularity score of 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

L1RA introduces a new gate vector and a reallocation heuristic, and it depends on several unproven optimization and evaluation assumptions. The free parameters are standard hyperparameters chosen by hand. The main load-bearing axiom is that L1-induced pruning plus min-c-based reallocation is better than fixed rank allocation, which is not isolated from the parameter-count confound.

free parameters (5)
  • lambda (L1 regularization coefficient) = 1e-3
    Chosen by hand for both models; controls the pruning pressure on the c vector and is central to the method.
  • eta_c (learning rate for c vectors) = 1e-2
    Chosen separately from the main learning rate; controls how quickly ranks are pruned or reallocated.
  • rank_update_period = 5% of training steps
    Determines how often pruning and reallocation are applied; not ablated.
  • initial rank r = 16
    The rank budget for L1RA, LoRA, and AdaLoRA baselines; set by the experimenters.
  • alpha (LoRA scaling) = 16
    Standard LoRA scaling hyperparameter, set to the same value across methods.
axioms (4)
  • domain assumption L1 regularization on the gate vector causes useful ranks to persist and redundant ranks to shrink to zero.
    The pruning mechanism relies on this optimization behavior; no proof or ablation is given, and it depends on optimizer dynamics.
  • ad hoc to paper Reallocating each spare rank to the unpruned adapter with the largest minimum c value (Equation 5) improves final performance.
    This ordering heuristic is motivated by an SVD analogy but is not derived or ablated against other allocation strategies.
  • domain assumption Perplexity on the OpenOrca dataset is a sufficient measure of assistant fine-tuning quality.
    The paper evaluates only perplexity, not downstream task accuracy or generation quality, yet concludes 'same or better performances'.
  • ad hoc to paper Comparing methods that start at the same initial rank but end with different parameter counts is a fair comparison.
    L1RA ends with more trainable parameters than LoRA because ranks are moved to larger matrices, confounding the comparison.
invented entities (1)
  • per-adapter rank gate vector c no independent evidence
    purpose: Eases L1 sparsity and enables fast pruning and reallocation of individual ranks.
    The gate vector is a new model component introduced by the paper. There is no external validation of the gate mechanism beyond the two reported fine-tuning experiments, and the rank-distribution insights come from this mechanism itself.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 33977 in / 9559 out tokens · 92713 ms · 2026-08-05T05:47:38.789427+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning." pith.science (2026). https://pith.science/paper/GKKFWAAV

@misc{pith2026250904884,
  author       = {Pith},
  title        = {Pith review of: L1RA: Dynamic Rank Assignment in LoRA Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GKKFWAAV}},
  note         = {Machine review of arXiv:2509.04884}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The ability of Large Language Models (LLMs) to solve complex tasks has made them crucial in the development of AI-based applications. However, the high computational requirements to fine-tune these LLMs on downstream tasks pose significant challenges, particularly when resources are limited. In response to this challenge, we introduce L1RA, a novel technique aimed at dynamically distributing the rank of low-rank adapters during fine-tuning using LoRA. Given a rank budget (i.e., total sum of adapters rank), L1RA leverages L1 regularisation to prune redundant ranks and redistribute them across adapters, thereby optimising resource utilisation. Through a series of comprehensive experiments, we empirically demonstrate that L1RA maintains comparable or even reduced computational overhead compared to other LoRA variants, including the vanilla approach, while achieving same or better performances. Moreover, the post-training analysis of rank distribution unveiled insights into the specific model components requiring the most adaptation to align with the task objective: the feed-forward layers and the attention output projection. These results highlight the efficacy of L1RA in not only enhancing the efficiency of LLM fine-tuning, but also in providing valuable diagnostic information for model refinement and customisation. In conclusion, L1RA stands as a promising technique for advancing the performance and interpretability of LLM adaptation, particularly in scenarios where computational resources are constrained.

Figures

Figures reproduced from arXiv: 2509.04884 by Mark James Carman, Nicolo Brunello, Raul Singh, Vincenzo Scotti.

Figure 1
Figure 1. Figure 1: LORA adapters: pre-trained weights are frozen while the two adapter matrices are updated during the fine-tuning. R r×dout into the network architecture (see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Motivating example: r ′ and r ′′ are such that r ′+r ′′ = 2r, so that the total amount of adapters memory is the same with and without optimal allocation. All the proposed solutions for dynamic rank assignment correctly work to reduce the rank used in the adapter matrices. However, these LORA vari￾ants are limited in the sense that they do not allow for spare (unused) ranks re-assignment and they rather wa… view at source ↗
Figure 3
Figure 3. Figure 3: L1RA adapters L1RA adapters, depicted in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: L1RA pruning and reallocation (lighter colours are for lower absolute values, white is 0). is in full use, with the various components of the c vector furthest from zero. The ordering of unpruned adapters is performed in Algorithm 1 by the finsert(·) function when saving them in ∆ϑu. After the rank re-allocation step, the training procedure reprises. This sorting step is inspired by SVD: assuming that in h… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of peak memory usage estimates from [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Matrix-wise distribution of layer-wise average [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Layer-wise evolution of matrix-wise average [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Layer-wise and Matrix-wise evolution of L1RA adapters rank during training. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning

    cs.CL 2026-05 unverdicted novelty 7.0

    MatryoshkaLoRA inserts a crafted diagonal matrix P into LoRA to learn accurate nested low-rank adapters that support dynamic rank selection with minimal performance drop.

Reference graph

Works this paper leans on

27 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ankur Bapna and Orhan Firat. 2019. https://doi.org/10.18653/V1/D19-1165 Simple, scalable adaptation for neural machine translation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , p...

    work page doi:10.18653/v1/d19-1165 2019

  2. [2]

    Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023. http://papers.nips.cc/paper/_files/paper/2023/hash/59404fb89d6194641c69ae99ecdf8f6d-Abstract-Conference.html In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing System...

    work page 2023

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  4. [4]

    Tri Dao. 2024. https://openreview.net/forum?id=mZn2Xyh9Ec Flashattention-2: Faster attention with better parallelism and work partitioning . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  5. [5]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html Qlora: Efficient finetuning of quantized llms . In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...

    work page 2023

  6. [6]

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.252 Sparse low-rank adaptation of pre-trained language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 4133--4145...

    doi:10.18653/v1/2023.emnlp-main.252 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  8. [8]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446 In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 5484--5495. Association for Computational Linguistics

    doi:10.18653/v1/2021.emnlp-main.446 2021

  9. [9]

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. https://doi.org/10.48550/ARXIV.2402.12354 Lora+: Efficient low rank adaptation of large models . CoRR, abs/2402.12354

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.12354 2024

  10. [10]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. http://proceedings.mlr.press/v97/houlsby19a.html Parameter-efficient transfer learning for NLP . In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach...

    2019

  11. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net

    2022

  12. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth \' e e Lacroix, and William El Sayed. 2023. https://doi.org/...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.06825 2023

  13. [13]

    Damjan Kalajdzievski. 2023. https://doi.org/10.48550/ARXIV.2312.03732 A rank stabilization scaling factor for fine-tuning with lora . CoRR, abs/2312.03732

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.03732 2023

  14. [14]

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. 2023. https://doi.org/10.48550/ARXIV.2310.11454 Vera: Vector-based random matrix adaptation . CoRR, abs/2310.11454

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.11454 2023

  15. [15]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/V1/2021.ACL-LONG.353 Prefix-tuning: Optimizing continuous prompts for generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Vir...

    doi:10.18653/v1/2021.acl-long.353 2021

  16. [16]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net

    work page 2019

  17. [17]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a627810151be4d13f907ac898ff7e948-Abstract-Conference.html Fine-tuning language models with just forward passes . In Advances in Neural Information Processing Systems 36: Annual Conference on Neural...

    work page 2023

  18. [18]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. https://doi.org/10.48550/ARXIV.2306.02707 Orca: Progressive learning from complex explanation traces of GPT-4 . CoRR, abs/2306.02707

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.02707 2023

  19. [19]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res., 21:140:1--140:67

    2020

  20. [20]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian - Jian Jiang, Han Wang, Matteo Manica, Sh...

    work page 2022

  21. [21]

    Vincenzo Scotti, Licia Sbattella, and Roberto Tedesco. 2024. https://doi.org/10.1145/3604281 A primer on seq2seq models for generative chatbots . ACM Comput. Surv. , 56(3):75:1--75:58

    work page doi:10.1145/3604281 2024

  22. [22]

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023. https://doi.org/10.18653/V1/2023.EACL-MAIN.239 Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrov...

    doi:10.18653/v1/2023.eacl-main.239 2023

  23. [23]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \' e mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. https://doi.org/10....

    doi:10.18653/v1/2020.emnlp-demos.6 2020

  24. [24]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. https://openreview.net/pdf?id=lq62uWRJjiY Adaptive budget allocation for parameter-efficient fine-tuning . In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

    work page 2023

  25. [25]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian - Yun Nie, and Ji - Rong Wen. 2023. https://doi.org/10.48550/ARXIV.2303.18223 A survey of large langua...

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.18223 2023

  26. [26]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  27. [27]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...