Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Fisher-information scoring rule can pick out the few transformer layers where low-rank experts are needed, and adapting only those layers matches or beats adapting every layer.

desk verdict Sensible layer-selection idea undermined by per-task oracle layer budgets and a mismatch between abstract claims and reported numbers. read the letter →

arxiv 2506.00495 v1 pith:GEG4RDLR submitted 2025-05-31 cs.LG cs.CLstat.ML

Xinyi Wang , Lirong Gao , Haobo Wang , Yiming Zhang , Junbo Zhao This is my paper
classification cs.LGcs.CLstat.ML
keywords parameter-efficientfine-tuninglow-rankadaptationmixture-of-expertsFisherinformationlayerselectionBayesianoptimizationsparselargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that for mixture-of-experts low-rank adaptation (MoE-LoRA), only a sparse subset of transformer layers needs trainable adapters. It introduces FLoE, a pipeline that scores each layer with Fisher information and Taylor importance, selects critical layers under a parameter budget, and uses Bayesian optimization to choose the LoRA rank. Fine-tuning the selected layers only is reported to match or exceed full-layer adaptation on single-domain and mixed-domain benchmarks while using far fewer trainable parameters. By adapting only 25% of layers, FLoE retains 93.1% of full fine-tuning accuracy on MMLU and improves mixed-domain performance by 7.0% over full-layer methods.

What carries the argument

The core object is a per-layer importance score built from the diagonal empirical Fisher information of the adapted loss with respect to binary masks on pretrained weights, traded against the Taylor importance of those weights under a global budget. A greedy algorithm iteratively masks heads and neurons with the smallest Taylor importance while maximizing cumulative Fisher information, a refinement stage swaps masked and unmasked entries to improve the trade-off, and a tuning stage reconstructs masked-layer activations to obtain continuous mask values. These scores produce the layer ranking that decides where MoE-LoRA adapters, each with a shared A matrix and multiple B experts, are deployed.

What would settle it

Run FLoE on a fixed target task but compute the layer ranking from a deliberately mismatched sample dataset, such as code data for a medical task, and compare against random layer selection at the same parameter budget: if the Fisher ranking does not consistently beat random selection, the transfer assumption fails. Additionally, measure the gradient norm at the end of the sample fine-tuning; if it is far from zero, the Taylor expansion underlying the importance scores is not valid in that regime.

Watch

Extended reading notes

Core claim

The central claim is that layer criticality for low-rank adaptation can be read from the geometry of a fine-tuned model: layers whose pretrained weights are insensitive to the adaptation loss (measured by Taylor importance) but whose adapter parameters are highly sensitive (measured by diagonal Fisher information) are the ones that need residual low-rank experts. The paper formalizes this as a constrained mask-search problem, solves it with a greedy search plus a local swap-based refinement, and then relaxes the binary masks to continuous values by minimizing residual-activation reconstruction error. The resulting sparse adapter placement is reported to outperform uniform full-layer LoRA-based methods, with the optimal number of layers varying by domain.

Load-bearing premise

The layer-importance ranking is computed on a model fine-tuned with full-layer adapters on a sample dataset, and the whole method assumes this ranking still identifies the correct critical layers on a different target dataset, while also assuming the fine-tuned model is near a local minimum so the first-order gradient term vanishes.

Editorial extensions

If this is right

  • Adapting only 8 of 32 layers of LLaMA2-7B with a 1A/4B MoE-LoRA adapter can match or exceed full-layer adaptation on Medical MMLU and mixed-domain BBH while using as little as 0.031% of model parameters.
  • The optimal number of adapted layers is task-dependent, so uniform placement of adapters across all layers is inefficient in both parameters and latency.
  • Bayesian optimization over rank, searching integer values in a bounded range in under 100 trials, can select a good LoRA rank without extensive grid search and retraining.
  • The layer selection transfers across model families: Gemma2-2B and Mistral-7B both reach their best reported performance with 8 selected layers, though the critically ranked layers lie in different depth regions.
  • Including the cost of layer selection, the total reported training runtime is still below full-layer adaptation on the tested datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Fisher-based ranking could be computed from a short, non-converged fine-tuning run instead of a fully converged model, which would make the method much cheaper and test whether the local-minimum assumption in the Taylor expansion is truly necessary.
  • Layer-importance rankings from a sample dataset could serve as a cheap domain-shift diagnostic: comparing rankings across source and target datasets may predict when adaptation needs more or fewer layers.
  • The explicit selection budget opens a natural extension to per-layer rank allocation, jointly choosing how many layers and how much rank each layer gets, which the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. FLoE proposes a two-stage sparse-adaptation framework for MoE-based LoRA. In the first stage, a full-layer MoE-LoRA model is fine-tuned on a sample dataset (Dolly-15K or a 1.25% FLANv2 subset), and Fisher-information and Taylor-importance scores are computed to rank transformer layers by their importance for adaptation. In the second stage, only the top-k layers are adapted on the target dataset, with a Bayesian-optimization procedure proposed to select LoRA ranks. The paper reports experiments on LLaMA2-7B, Gemma2-2B, and Mistral-7B across MMLU, medical, legal, code, math, and BBH benchmarks, claiming that sparse layer adaptation achieves accuracy comparable to or better than full-layer adaptation while using far fewer parameters.

Significance. The core idea is plausible and practically relevant: layer contributions to LoRA adaptation are heterogeneous, and identifying task-critical layers could reduce parameter and memory costs with little accuracy loss. The paper includes useful comparisons against random and weight-norm layer selection, a stability analysis over sample-dataset sizes, and experiments across several model families. However, the experimental protocol currently does not establish the central claim of sparse-adaptation advantage. The headline results are obtained by selecting the best layer count per task after seeing the per-layer performance curves, the abstract's '25% of layers' and '7.0% relative improvement' statements are not supported by the reported numbers, and the Bayesian rank allocator is not integrated into any main result. Because these issues affect the paper's central claims, the manuscript requires major revision.

major comments (5)
  1. [§4.2, Tables 1 and 3] The headline sparse-adaptation result is not evaluated under a fixed budget. Table 1's Ours row uses layer counts 8, 8, 20, 16, and 24 for MMLU, Medical, Law, HumanEval, and GSM8K, respectively, and these counts coincide with each task's best result in Table 3. The abstract's "adapting only 25% of layers" holds only for MMLU and Medical; Law uses 62.5% of layers, HumanEval uses 50%, and GSM8K uses 75%. Because the budget is chosen after seeing per-layer performance, the claim that sparse adaptation matches or exceeds full-layer adaptation is not demonstrated for any single budget. Table 2's fixed 8-layer results are a step in the right direction, but the main comparison and the abstract need to be rebuilt around a pre-specified budget policy, or should report results for all layer counts with an explicit and honest selection rule.
  2. [Abstract, Figure 5, Table 8] The abstract's "7.0% relative improvement over the best-performing full-layer methods" is not supported by Figure 5. With FLoE at 42.7 EM, the relative improvement over HydraLoRA (41.5) is about 2.9%, over LoRAMoE (40.3) about 6.0%, and over LoraHub (39.7) about 7.6%; no baseline in the figure yields exactly 7.0%. Additionally, Figure 5 labels FLoE at 0.051% trainable parameters, which corresponds to the 8-layer configuration in Table 8, but Table 8's 8-layer row reports 44.49 EM, not 42.7. If the figure instead uses the 12-layer configuration, the parameter fraction should be 0.077%, not 0.051%. This inconsistency prevents verification of the reported mixed-domain advantage.
  3. [§3.2, Eqs. (4) and (6)] The theoretical foundation of the importance score is mis-specified. Equation (4) defines the Taylor importance T_k as the sensitivity of the pre-training loss, but the scores are computed on a model that has already been fine-tuned on Dolly-15K or a FLANv2 subset; the loss used in the implementation is therefore a fine-tuning loss, not a pre-training loss. Furthermore, Eq. (6) drops the first-order term by assuming convergence to a local minimum, which is not justified for a model fine-tuned for one epoch with LoRA. The derivation needs an explicit statement of which loss is used and a justification for why the gradient term is negligible in this setting.
  4. [§3.4, Figure 4, Table 1] The Bayesian optimization rank allocator is not integrated into any reported result. Figure 4 concludes that r=4 is optimal for the Medical task, yet Table 1 and all main comparisons use r=8 throughout. The second claimed contribution is therefore unsupported by the experiments. The paper should either present results with BO-chosen ranks, compare those results against fixed-rank baselines, or clearly state that the main results use a fixed rank and that the BO component is a separate proposal rather than a validated component of FLoE.
  5. [Appendix D, Figure 8] The paper's own transfer analysis shows that dataset misalignment can change the number of layers needed to reach peak performance. Since the main protocol selects the per-task best layer count from the performance curve, the transfer test does not actually validate the pipeline; it confirms that the optimal budget shifts with the target dataset. A practical sparse-adaptation method must specify how the budget is set without oracle knowledge. The paper should provide and evaluate a principled budget-selection rule (for example, based on validation loss) and report results under that rule.
minor comments (4)
  1. [§3.1, text after Eq. (2)] The notation in the sparsity discussion is confusing: for a binary mask vector, \|m_k\|_0 and \|m_k\|_1 both count the number of nonzero entries, so the statement "Higher sparsity (\|m_k\|_1 \ll \|m_k\|_0)" is not meaningful as written. Please clarify which norm counts masked versus retained parameters.
  2. [Algorithm 1, line 9] The floor operation in the formula for the number of retained FFN neurons is not explicit; using a real-valued division inside min/max without a floor is ambiguous for a count of discrete neurons. Please write the formula with an explicit floor.
  3. [Table 4] The column layout of Table 4 is difficult to parse: the "Single LoRA Head" and "Multi LoRA Heads" groups are not clearly separated from the layer-count columns, and the Mistral-7B row appears to have an extra 32-layer column. Please reformat the table so that each configuration is unambiguous.
  4. [Throughout] There are several typos and formatting inconsistencies, including "deonte", "fine-tuing", "Contrained", "bewteen", and the inconsistent citation style in reference [42]. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: Fisher and Taylor scores are computed on a separately trained full-layer model, and target labels never enter the layer-importance computation; the per-task layer-budget selection is an evaluation-protocol concern, not a circular input-output relation.

full rationale

The derivation chain is self-contained. The importance scores in Eqs. 4-12 are computed on a full-layer MoE-LoRA model fine-tuned on a sample dataset (Dolly-15K or 1.25% of FLANv2), with gradients taken with respect to mask variables on that sample; the final sparsely adapted model is then retrained on the target dataset. Target test labels do not enter the Fisher/Taylor estimation, so the layer ranking is not defined in terms of the final evaluation. The BO rank allocator uses validation loss, which is standard hyperparameter selection rather than a circular fit. Appendix D even tests the transfer of rankings between sample and target datasets and reports that misalignment changes the optimal layer count, showing the authors treat the transfer as empirical rather than definitional. The only notable weakness is that Table 1's 'Ours' results use per-task layer counts (8/8/20/16/24) taken as the best entries from Table 3's sweep, so the abstract's '25% of layers' claim is not a fixed-budget result and the efficiency comparison is not made at a single budget. This is a reporting/selection-bias concern, not a case where a prediction is equivalent to its inputs by construction: the Fisher ranking itself is independent of that sweep. No load-bearing self-citations or imported uniqueness theorems are used.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

FLoE introduces no new physical entities. The free parameters are the layer count, rank, and unspecified budget C. The main axioms are standard approximations in pruning literature (Fisher/Hessian approximation, diagonal assumption) plus the transferability of importance scores across datasets.

free parameters (3)
  • Number of adapted layers k = 8 to 32, chosen per task (8 for MMLU, 8 for Medical, 20 for Law, 16 for HumanEval, 24 for GSM8K)
    The paper reports each task at its best layer count, meaning k is effectively selected based on the test results for that task.
  • LoRA rank r = 8 in all main tables; BO finds r=4 for the medical task but it is not used
    Rank is a hand-set hyperparameter in the reported experiments; the BO-determined rank is not integrated into the main comparisons.
  • Global budget C = Not specified
    The constrained optimization in Eq. 12 depends on a budget C, but the paper never states a value or how it is set, making the algorithm under-specified.
assumptions (4)
  • domain assumption The empirical Fisher Information Matrix approximates the Hessian of the mask loss (Eq. 8-9).
    This relies on local smoothness and convergence of the loss, which fine-tuned LLMs typically do not satisfy exactly. Stated in Section 3.2.
  • domain assumption The diagonal approximation of the FIM ignores cross-layer interactions (Section 3.2).
    The paper reduces complexity by dropping off-diagonal terms, which may miss correlated importance across layers.
  • ad hoc to paper Taylor importance T_k in Eq. 4 is computed on a 'pre-training loss', but the actual model used is fine-tuned on Dolly-15K, so the loss is a fine-tuning loss, not pre-training.
    No pre-training loss is accessible for LLaMA2; using a task-adaptation loss changes the meaning of T_k and is not disclosed.
  • domain assumption Layer importance rankings transfer from the sample dataset to the target dataset (Appendix D).
    The paper tests this empirically and finds that misalignment can affect the required number of layers, so this transfer is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts." pith.science (2026). https://pith.science/paper/GEG4RDLR

@misc{pith2026250600495,
  author       = {Pith},
  title        = {Pith review of: FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEG4RDLR}},
  note         = {Machine review of arXiv:2506.00495}
}
read the original abstract

Parameter-Efficient Fine-Tuning (PEFT) methods have emerged as a widely adopted strategy for adapting pre-trained Large Language Models (LLMs) to downstream tasks, significantly reducing memory and computational costs. However, most existing PEFT techniques uniformly deploy LoRA adapters across all layers, disregarding the intrinsic heterogeneity of layer contributions and task-specific rank requirements. This uniform paradigm leads to redundant parameter allocation and suboptimal adaptation efficiency. To address these limitations, we propose FLoE, a novel PEFT framework that introduces two key innovations: (i) a Fisher information-guided importance scoring mechanism to dynamically identify task-critical transformer layers for MoE-based low-rank adaptation, enabling sparse adapter deployment; and (ii) a Bayesian optimization-driven rank allocator that automatically determines optimal LoRA ranks on specific datasets without exhaustive grid search. Extensive experiments across diverse LLMs and benchmarks reveal that FLoE achieves impressive efficiency-accuracy trade-offs, making FLoE particularly advantageous in resource-constrained environments that necessitate rapid adaptation.

Figures

Figures reproduced from arXiv: 2506.00495 by the authors.

Figure 1
Figure 1. (a) Architecture of our MoE-based LoRA implementation. We adopt the asymmetric archi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Mechanism of FLoE layer selection. A densely masked layer intends to have higher [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Saliency maps illustrating the importance of layers on the single-domain (Lawyer-Instruct) and mixed-domain/task (FLANv2) datasets. The base model is LLaMA2-7B. Black dots indicates the top-5 critical layers for the given task [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Mixed domain results evaluated on the BBH benchmark (3-shot) using LLaMA2- 7B. FLoE achieves the highest Exact Match (EM) score with the lowest training parameter size [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Empirical results reveal that disabling this stage leads to a performance degradation as [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Rank stability comparison of layers with (left) and without (right) refinement stage. The plots show layer importance rankings (lower rank indicate higher importance) across three training data ratios (100%, 70%, 30%). The shaded regions depict min-max ranges of the ra…
Figure 8
Figure 8. Figure 8: Illustration of performance under different layer importance rankings. E Proof of Eq. 16 The tuning stage is designed to minimize the layer-wise reconstruction error between the masked and unmasked versions of the adapted model. The goal is to optimize mk such that the…
Figure 9
Figure 9. Figure 9: Layer importance heatmaps for Gemma2-2B (top) and Mistral-7B (bottom), highlighting critical adaptation layers (8-17 for Gemma2-2B; 27-31 for Mistral-7B). Saliency values reflect the contribution of each layer, with darker hues indicating higher importance. 16 [PITH_F…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

Reference graph

Works this paper leans on

52 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Optuna: A next- generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2623–2631, 2019

    work page 2019

  2. [2]

    Lawyer-instruct, 2024

    Alignment-Lab-AI. Lawyer-instruct, 2024

    work page 2024

  3. [3]

    LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

    Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Martin Katz, and Anders Søgaard. Lexfiles and legallama: Facilitating english multinational legal language model development. arXiv preprint arXiv:2305.07507, 2023

    work page Pith review arXiv 2023

  4. [4]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https: //github.com/sahil280114/codealpaca, 2023

    work page 2023

  5. [5]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  6. [6]

    Longlora: Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023

    arXiv 2023

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [8]

    Sparse low-rank adaptation of pre-trained language models

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696, 2023

    arXiv 2023

Show all 52 references
  1. [9]

    Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint arXiv:2312.09979, 2023

    2023 arXiv

  2. [10]

    Layer skip: Enabling early exit inference and self-speculative decoding

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710, 2024

    2024 arXiv

  3. [11]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning , pages 10323–10337. PMLR, 2023

    2023

  4. [12]

    MoLA: MoE LoRA with layer-wise expert allocation

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, and Vs Subrahmanian. MoLA: MoE LoRA with layer-wise expert allocation. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Lin...

    2025

  5. [13]

    Lora+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024

    2024 arXiv

  6. [14]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  8. [16]

    Lorahub: Efficient cross-task generalization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023

    2023 arXiv

  9. [17]

    Adaptive mixtures of local experts

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991

    1991

  10. [18]

    Identifying and mitigating vulnerabilities in llm-integrated applications

    Fengqing Jiang. Identifying and mitigating vulnerabilities in llm-integrated applications. Master’s thesis, University of Washington, 2024. 10

    2024

  11. [19]

    Mora: High-rank updating for parameter-efficient fine-tuning

    Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parameter-efficient fine-tuning. arXiv preprint arXiv:2405.12130, 2024

    2024 arXiv

  12. [20]

    Less is more: Selective layer finetuning with subtuning

    Gal Kaplun, Andrey Gurevich, Tal Swisa, Mazor David, Shai Shalev-Shwartz, and Eran Malach. Less is more: Selective layer finetuning with subtuning. arXiv preprint arXiv:2302.06354, 2023

    2023 arXiv

  13. [21]

    Vera: Vector-based random matrix adaptation

    Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  14. [22]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor,Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989

    1989

  15. [23]

    Surgical fine-tuning improves adaptation to distribution shifts

    Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466, 2022

    2022 arXiv

  16. [24]

    Conditional adapters: Parameter-efficient transfer learning with fast inference

    Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Parameter-efficient transfer learning with fast inference. Advances in Neural Information Processing Systems , 36:8152–8172, 2023

    2023

  17. [25]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  18. [26]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  19. [27]

    Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6), 2023

    2023

  20. [28]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950–1965, 2022

    1950

  21. [29]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  22. [30]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021

    2021 arXiv

  23. [31]

    Gpt understands, too

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 5:208–215, 2024

    2024

  24. [32]

    Alora: Allocating low-rank adaptation for fine-tuning large language models

    Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models. arXiv preprint arXiv:2403.16187, 2024

    2024 arXiv

  25. [33]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , pages 22631–22648. P...

    2023

  26. [34]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems , 36:21702–21720, 2023

    2023

  27. [35]

    Pissa: principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948, 2024

    2024 arXiv

  28. [36]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    C Mike, H Matt, M Ankit, X Jianwei, W Jun, S Sam, G Ali, W Patrick, Z Matei, and X Reynold. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    2023

  29. [37]

    Multi-head adapter routing for cross-task generalization

    Lucas Page-Caccia, Edoardo Maria Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization. Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    Lisa: Lay- erwise importance sampling for memory-efficient large language model fine-tuning

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Lay- erwise importance sampling for memory-efficient large language model fine-tuning. arXiv preprint arXiv:2403.17919, 2024. 11

    2024 arXiv

  31. [39]

    Tied-lora: Enhacing parameter efficiency of lora with weight tying

    Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter efficiency of lora with weight tying. arXiv preprint arXiv:2311.09578, 2023

    2023 arXiv

  32. [40]

    On the effect of dropping layers of pre-trained transformer models

    Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429, 2023

    2023

  33. [41]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  34. [42]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  35. [43]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  36. [44]

    Hydralora: An asymmetric lora architecture for efficient fine-tuning

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245, 2024

    2024 arXiv

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  38. [46]

    Dylora: Parameter efficient tun- ing of pre-trained models using dynamic search-free low-rank adaptation.arXiv preprint arXiv:2210.07558, 2022

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tun- ing of pre-trained models using dynamic search-free low-rank adaptation.arXiv preprint arXiv:2210.07558, 2022

    2022 arXiv

  39. [47]

    Eigendamage: Structured pruning in the kronecker-factored eigenbasis

    Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In International conference on machine learning , pages 6566–6575. PMLR, 2019

    2019

  40. [48]

    Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models

    Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205.12410, 1(2):4, 2022

    2022 arXiv

  41. [49]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-2...

    2022

  42. [50]

    Crash: Clus- tering, removing, and sharing enhance fine-tuning without full large language model

    Kaiyan Zhang, Ning Ding, Biqing Qi, Xuekai Zhu, Xinwei Long, and Bowen Zhou. Crash: Clus- tering, removing, and sharing enhance fine-tuning without full large language model. arXiv preprint arXiv:2310.15477, 2023

    2023 arXiv

  43. [51]

    Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023

    2023 arXiv

  44. [52]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023. A Datasets and Baselines A.1 Datasets Single...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.