Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse adapters trained on a small mask of weights can beat LoRA and full fine-tuning, and averaging their task updates merges up to 20 expert models.

desk verdict A useful scaling study with a real empirical result, but the headline merging comparison is weakened by selecting the keep-ratio on the test sets and by a sloppy merge formula. read the letter →

arxiv 2507.07140 v2 pith:PEEDDCRC submitted 2025-07-09 cs.LG

Samin Yeasar Arnob , Zhan Su , Minseon Kim , Oleksiy Ostapenko , Riyasat Ohib , Esra'a Saleh , Doina Precup , Lucas Caccia
show 1 more author
Alessandro Sordoni
This is my paper
classification cs.LG
keywords sparseadaptersmodelmergingparameter-efficientfine-tuningLoRAblocksparsityconnectionsensitivitymulti-taskNLPFLAN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that sparse adapters are practical building blocks for modular language models: instead of low-rank adapters or full fine-tuning, train only a subset of the base weights for each task, then merge the task-specific updates by simple averaging. The authors introduce a conceptually simpler way to pick that subset (keep the weights whose value and gradient agree), show that it beats LoRA and full fine-tuning on single-task fine-tuning of 20 FLAN tasks, and then merge up to 20 experts at once, far beyond the two-task settings common in earlier work. Their headline result is that element-sparse averaging reaches 51.44 mean Rouge-L on the 20 held-in tasks, compared with 45.96 for LoRA averaging and 50.21 for the best full fine-tuning merge method, while training only 2.37% of the parameters. Held-out generalization remains competitive but lags true multitask training, and weight interference inside the shared masks is identified as the main source of held-in degradation. A sympathetic reader would care because modular merging promises asynchronous, privacy-preserving multitask models without retraining on all data.

What carries the argument

The load-bearing object is the sparse adapter, a sparse increment $\Delta W = \hat{W} \cdot M$ to the base weights $W$, where $M$ is a binary mask and $\hat{W}$ starts at zero. The mask is chosen by Maximum Connection Sensitivity, $\mathrm{MCS}(w_q) = w_q \frac{\partial L}{\partial w_q}$, which rewards parameters whose learned value and gradient point the same way; in practice it is applied as a TopK selection over the QKV weights and refreshed every 100 steps during the first epoch. Merging uses an overlap-aware average, $\Delta W_m = \frac{1}{F_o} \sum_i \Delta W_i$ with $F_o = \min(\sum_i M_i, 1)$, so weights trained by several tasks are divided by the number of tasks that selected them. This mechanism identifies a small task-specific subspace, limits parameter interference at merge time, and keeps training memory low because only masked weights need a dense update.

What would settle it

Run the same 20-task merging protocol with kr chosen only on a held-out validation subset of FLAN tasks and then evaluate on never-seen test tasks; if LoRA averaging ties or beats element-sparse averaging at the newly chosen kr, the paper's central merging claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that sparse adapters—task-specific shifts that train a small subset of the base model's weights—are better building blocks for merged multi-task models than LoRA or full fine-tuning. On 20 FLAN tasks, a single sparse adapter with 99% sparsity (kr=0.01) already beats full fine-tuning, and element-sparse averaging with kr=0.1 reaches 51.44 mean Rouge-L on held-in tasks, against 45.96 for LoRA averaging and 50.21 for the best full fine-tuning merge (Ties), with only 2.37% of parameters trainable. Block-sparse variants perform nearly as well. Multitask training remains the upper bound (77.15 held-in, 54.00 held-out). The paper also characterizes the main failure mode: after merging, held-in performance drops mostly because of interference among parameters inside the shared masks, not because of changes outside them, and closing that gap remains an open direction.

Load-bearing premise

The headline merging numbers are all obtained at kr=0.1, and that keep-ratio was chosen in Section 4.2 by evaluating merged performance on the same held-in and held-out test sets that are later reported, so the comparison to LoRA could shift if a fresh task split re-tuned the sparsity level.

Editorial extensions

If this is right

  • Sparse adapters can be inserted into modular systems where expert models are trained asynchronously on private data and combined by averaging, since the overlap-aware merging rule only needs the masks and weight updates.
  • At the 20-task scale, full fine-tuning merge methods lose held-in performance, while sparse adapters stay effective, making them a candidate default building block for larger expert libraries.
  • Block-sparse adapters give nearly the same merging quality as unstructured sparse adapters, so hardware-friendly block patterns can be used without sacrificing much accuracy.
  • Because masked-region interference drives the held-in loss after merging, training methods that reduce update conflicts inside the shared masks should directly improve merged performance.
  • Even with roughly a third of layers dropped, the layer-drop sparse adapter still beats full fine-tuning merge baselines on held-in tasks, suggesting sparsity and active-layer reduction can be combined.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because kr=0.1 was selected by evaluating merged performance on the same held-in and held-out tasks later reported, the advantage over LoRA could shrink on a fresh task draw; a robust test would sweep kr on a validation split held out from the final numbers.
  • Beyond the paper: the overlap factor in Eq. (4) suggests a cheap predictor of mergeability—tasks whose masks overlap heavily should interfere more, so mask Jaccard similarity could be used to group or order experts before merging.
  • Beyond the paper: sparse adapters could be combined with routing so that only the masks of the most relevant experts are queried during inference, potentially giving the modularity of Mixture-of-Experts without its training cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a method for training sparse adapters for large language models, using a saliency-based score (maximum connection sensitivity, MCS) to select a subset of weights in the QKV layers, and a block-sparse variant. It then studies how these sparse adapters can be merged across up to 20 FLAN tasks, using a simple overlap-aware averaging scheme, and compares against LoRA, full fine-tuning, Task Arithmetic, TIES, and Breadcrumbs. The central claims are: (i) the proposed sparse adapters outperform both LoRA and full fine-tuning in single-task fine-tuning; (ii) merging sparse adapters via averaging yields superior held-in performance compared to LoRA or full model merging while achieving competitive held-out performance; (iii) sparse adapters are more scalable to many experts than full fine-tuning. The paper includes ablations on parameter selection criteria, block size, layer choice, and number of experts, as well as an analysis of weight interference during merging.

Significance. If the claims are robust, the paper makes a useful contribution by showing that simple sparse adapters, trained without an initial full fine-tuning phase, can serve as effective building blocks for parameter-efficient model merging, scaling to 20 tasks. The proposed training method is conceptually simpler than prior sparse fine-tuning approaches, and the merging analysis, including the interference decomposition in Section 4.3, is informative. The paper is also careful to report the gap to multitask training and the challenge of held-out generalization. However, the central quantitative claims are weakened by the test-set-based selection of the keep-ratio, the absence of error bars in key tables and figures, and an inconsistency in the stated merging formula. These issues currently prevent the results from being taken as established.

major comments (4)
  1. [Section 4.2, Table 1, Fig. 11, Appendix B] The keep-ratio kr = 0.1 is selected by evaluating merged performance on the same held-in (20 tasks) and held-out (10 tasks) test sets that are later used to report the headline results in Table 1. Section 4.2 states: 'We found that the sparse adapter performs best on both held-in and held-out tasks when kr = 0.1. Therefore, we use kr = 0.1 for the performance comparison in the model merging experiments.' The same test sets then appear in Table 1, so the reported advantage of sparse averaging (51.44 held-in vs 45.96 for LoRA and 50.21 for TIES) may be an upper bound from selection on the evaluation data rather than a robust property. Since Fig. 11 shows the performance curve over kr, a fair evaluation would select kr on a validation set disjoint from the final test tasks, or report results over a range of kr without claiming a single best value. This issue is load-bearing because the central merging claim is stated at kr = 0.1.
  2. [Section 3, Eq. (4)] Equation (4) defines the merged update as ΔW_m = (1/F_o) * Σ_i ΔW_i with F_o = min(Σ_i M_i, 1). The text explains that if a weight is shared across k task-specific masks, the update should be divided by k, the overlap count. However, for any weight selected by at least one task, min(Σ_i M_i, 1) equals 1, so the formula does not divide by the number of overlapping tasks; it always divides by 1 for non-zero overlap. If the intended normalization is to divide by the overlap count, the formula should use F_o = max(Σ_i M_i, 1) or an equivalent element-wise count with a floor of 1. If the capped form is intentional, the text should explain why the overlap count is not used. Because Eq. (4) is the core merging operation for sparse adapters, this inconsistency must be resolved.
  3. [Table 1, Fig. 3(a), Fig. 6] The central comparisons are reported as mean ROUGE-L values without error bars, seed counts, or per-task variance. Table 1 lists single numbers for each method, and Fig. 3(a) shows curves for sparse adapter, LoRA, and full fine-tuning without any measure of variability, making it impossible to assess whether the reported gaps (e.g., 51.44 vs 50.21 vs 45.96 in Table 1) are within run-to-run noise. Fig. 6 does report mean performance over 10 trials for the scaling analysis, but the main table and single-task figure do not. The authors should provide standard errors or confidence intervals for at least the headline results, and state the number of random seeds used.
  4. [Table 1] The merged comparison is not parameter-matched: sparse averaging uses 2.37% trainable parameters while LoRA averaging uses 1.54% (r = 128). This means the sparse adapter has about 54% more trainable parameters than the LoRA baseline, so the reported held-in advantage could partly reflect capacity rather than the benefit of sparsity or the merging scheme. The authors should add a LoRA configuration with a rank chosen to match 2.37% trainable parameters, or a sparse adapter with kr chosen to match 1.54%, to determine whether the advantage persists under a matched budget.
minor comments (5)
  1. [Section 4.2] There is a typo: 'By default we we employ simple uniform weight averaging' should read 'By default, we employ simple uniform weight averaging.'
  2. [Section 2] The phrase 'it's orthogonal sparse adapter' should be 'its orthogonal sparse adapter', and 'block-spare' should be 'block-sparse'.
  3. [Throughout] The metric is inconsistently written as 'Rouge-L' and 'RougeL'; the standard capitalization is 'ROUGE-L'. Please use one form consistently.
  4. [Section 4.2, Table 2] Table 2 reports 'MCS (Ours)' with an individual score of 77.54, but the abstract and Section 4.1 do not clarify whether the single-task performance is reported at kr = 0.1 or at the best kr; please state the sparsity level used for the individual scores in Table 2.
  5. [Appendix A, Fig. 9] The learning-rate sweep in Fig. 9 reports mean performance over five fixed tasks, but the text does not specify whether these five tasks are the same as the ones used for block-size and layer-selection sweeps; please clarify for reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; sparse-merge advantage is externally benchmarked, with only a test-set-selection risk on kr and a non-circular Eq. (4) typo.

full rationale

The paper's central claims are evaluated against external baselines (LoRA, full fine-tuning, Task Arithmetic, TIES, Breadcrumbs, multitask training) on a fixed Phi-3 base model and a fixed FLAN task split. The sparse-adapter training and merging pipeline is fully specified in Algorithm 1 and Eqs. (1)-(4), and it does not reduce to the reported outcome by construction. The MCS criterion in Eq. (2) is a defined parameter-selection rule, not a fitted value that is later renamed as a prediction. Self-citations (Arnob et al. 2021, 2024 for saliency; Ostapenko et al. 2024 for task setup) are background or benchmark choices, not load-bearing evidence for the superiority claim. The one substantive methodological concern is Section 4.2: the authors state that kr = 0.1 'performs best on both held-in and held-out tasks' and then use that same held-in and held-out evaluation to produce Table 1. That is test-set selection, which can inflate the reported advantage and is a statistical validity risk, but it is not circular reasoning because no reported quantity is defined in terms of the conclusion. A separate non-circular technical issue is Eq. (4), where Fo = min(sum M_i, 1) caps the overlap factor at 1 and therefore does not actually divide by the number of overlapping tasks as the surrounding text describes; this is an implementation/formula inconsistency, not a self-referential derivation. Overall, the sparse-merge advantage is an empirical comparison against external baselines rather than a consequence of the paper's own definitions, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on several tuned hyperparameters (kr, block size, learning rates, mask update interval) and on assumptions about evaluation (Rouge-L as the sole metric, QKV-only sparsification, representative sweep tasks, single base model). None of these are fitted constants in a theoretical derivation, but the selection of kr on the evaluation test sets is the most consequential and is a genuine free parameter affecting the headline result.

free parameters (5)
  • keep-ratio kr = 0.1 for merging (0.5 best single-task)
    Fraction of QKV weights kept trainable; chosen by comparing merged held-in and held-out Rouge-L in Fig. 11, i.e., tuned on the evaluation test sets.
  • block size B for block-sparse adapters = 16
    Selected from {8, 16, 32} on 5 fixed tasks (Appendix A, Fig. 10); no ablation on the full 20-task set.
  • learning rate per method = not explicitly stated in text
    Chosen from {1e-3, 1e-4, 1e-6} on 5 tasks (Appendix A, Fig. 9); final values used for the main experiments are not reported.
  • mask update interval = 100 steps
    Mask recomputed every 100 gradient steps during the first epoch (Algorithm 1); no ablation for this interval is shown.
  • training epochs = 5
    All fine-tuning runs for 5 epochs; fixed in the setup without ablation.
assumptions (4)
  • domain assumption Rouge-L is an appropriate quality metric for all 20 FLAN tasks.
    Used as the sole evaluation metric; several FLAN tasks are classification-style, and no other metrics (e.g., exact match) are reported.
  • domain assumption Fine-tuning only QKV attention layers provides sufficient capacity for all 20 tasks.
    Layer selection was validated on 5 tasks (Appendix A, Fig. 7), but the main claims assume QKV-only sparsification transfers to the full set.
  • ad hoc to paper The 5 tasks used for hyperparameter sweeps are representative of the 20 held-in tasks.
    Learning rate, block size, and mask interval are tuned on 5 fixed tasks; no evidence that these 5 are representative of the remaining 15.
  • domain assumption Phi-3-mini is a representative base model for the findings.
    All experiments use a single 3.8B instruction-tuned model; no second model or family is tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts." pith.science (2026). https://pith.science/paper/PEEDDCRC

@misc{pith2026250707140,
  author       = {Pith},
  title        = {Pith review of: Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEEDDCRC}},
  note         = {Machine review of arXiv:2507.07140}
}
read the original abstract

Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly adapted on the fly for specific downstream tasks, without requiring additional fine-tuning. Typically, LoRA serves as the foundational building block of such parameter-efficient modular architectures, leveraging low-rank weight structures to reduce the number of trainable parameters. In this paper, we study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures. First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature and surprisingly outperforms both LoRA and full fine-tuning in our setting. Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks, thus scaling beyond what is usually studied in the literature. Our findings demonstrate that sparse adapters yield superior in-distribution performance post-merging compared to LoRA or full model merging. Achieving strong held-out performance remains a challenge for all methods considered.

Figures

Figures reproduced from arXiv: 2507.07140 by the authors.

Figure 1
Figure 1. Left: The base model’s weights remain frozen during training, with only the sparse adapter modules updated; during each forward pass, the adapter’s outputs are combined with the fixed base weights. Right: Multiple sparse adapters—each trained separately on different tasks—are merged at inference time with the frozen base model weights [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Visualization of a weight matrix with sparsity ratio 0.5. Trainable parameters are highlighted in blue. Element-sparse and block-sparse with block size B = 2 and NB = 2. Right: A visual representation of sparse-adapter reconfiguration during fine-tuning. Color transitions indicate updates to the weight-space via masking. Subsets of trainable parameters are updated periodically. Once the mask is fixed after a f… view at source ↗
Figure 3
Figure 3. (a) Comparison of sparse adapter, LoRA, and Full FT on single task performance. We report the average rouge-L performance across 20 tasks, showing performance variations with different trainable parameter ranges by adjusting the LoRA tuning rank and the fraction of parameters for the sparse adapter in log scale. (b) Merged performance on both held-in (20 tasks) and held-out tasks (10 tasks). Section 4.3, we explore … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visual representation of sources of weight interference for task [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Impact of weight interference due to merging sparse adapters. We compare the mean Rouge-L score of the 20 held-in tasks [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: Performance of Sparse-Adapter (kr = 0.1) on training QKV, QKV-O, MLP layers in Phi-3. Mean Rouge-L computed across 20 individual tasks and merged Per￾formance for 20 held-in and 10 held-out tasks [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: Performance of different methods under varying learning-rate. We compare the mean Rouge-L performance of 5 individ￾ually trained tasks to decide the best learn￾ing rate for each method [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 11
Figure 11. Figure 11: Sparse adapter performance com￾parison across different values of kr on held-in (20 tasks) and held-out (10 tasks) after merg￾ing. Performance under Varying Sparsity [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 10 canonical work pages

  1. [1]

    Mix data or merge models? optimizing for diverse multi-task learning, 2024

    Aakanksha, Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. Mix data or merge models? optimizing for diverse multi-task learning, 2024. URL https://arxiv.org/abs/2410.10801

    arXiv 2024

  2. [2]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  3. [3]

    Evolutionary optimization of model merging recipes, 2024

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes, 2024. URL https://arxiv.org/abs/2403.13187

    arXiv 2024

  4. [4]

    Composable sparse fine-tuning for cross-lingual transfer

    Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vuli \'c . Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1778--1796, 2022

    work page 2022

  5. [5]

    Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M. Ponti. Scaling sparse fine-tuning to large language models, 2024. URL https://arxiv.org/abs/2401.16405

    arXiv 2024

  6. [6]

    Single-Shot Pruning for Offline Reinforcement Learning

    Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. Single-shot pruning for offline reinforcement learning. arXiv preprint arXiv:2112.15579, 2021

    work page Pith review arXiv 2021

  7. [7]

    Efficient reinforcement learning by discovering neural pathways

    Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, Amy Zhang, Alessandro Sordoni, and Doina Precup. Efficient reinforcement learning by discovering neural pathways. Advances in Neural Information Processing Systems, 37: 0 18660--18694, 2024

    work page 2024

  8. [8]

    Model breadcrumbs: Scaling multi-task model merging with sparse masks, 2024

    MohammadReza Davar. Model breadcrumbs: Scaling multi-task model merging with sparse masks, 2024. URL https://arxiv.org/abs/2312.06795

    arXiv 2024

Show all 47 references
  1. [9]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    Rigging the lottery: Making all tickets winners, 2021

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners, 2021. URL https://arxiv.org/abs/1911.11134

    2021 arXiv

  3. [11]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. URL https://arxiv.org/abs/1803.03635

    2019 arXiv

  4. [12]

    Megablocks: Efficient sparse training with mixture-of-experts

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5: 0 288--304, 2023

    2023

  5. [13]

    Parameter-efficient fine-tuning for large models: A comprehensive survey

    Z Han, C Gao, J Liu, J Zhang, and S Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arxiv 2024. arXiv preprint arXiv:2403.14608

    2024 arXiv

  6. [14]

    Lora+: Efficient low rank adaptation of large models (jul 2024)

    S Hayou, N Ghosh, and B Yu. Lora+: Efficient low rank adaptation of large models (jul 2024). arXiv preprint arXiv:2402.12354

    2024 arXiv

  7. [15]

    Sparseadapter: An easy approach for improving the parameter-efficiency of adapters, 2022

    Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters, 2022. URL https://arxiv.org/abs/2210.04284

    2022 arXiv

  8. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021 b . URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  9. [18]

    Lors: Efficient low-rank adaptation for sparse large language model

    Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, and Hong Chen. Lors: Efficient low-rank adaptation for sparse large language model. arXiv preprint arXiv:2501.08582, 2025

    2025 arXiv

  10. [19]

    Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024. URL https://arxiv.org/abs/2307.13269

    2024 arXiv

  11. [20]

    Editing models with task arithmetic, 2023

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023. URL https://arxiv.org/abs/2212.04089

    2023 arXiv

  12. [22]

    Dataless knowledge fusion by merging weights of language models, 2023 b

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models, 2023 b . URL https://arxiv.org/abs/2212.09849

    2023 arXiv

  13. [23]

    Snip: Single-shot network pruning based on connection sensitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018

    2018 arXiv

  14. [24]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  15. [25]

    Parameter-efficient orthogonal finetuning via butterfly factorization

    Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.06243, 2023

    2023 arXiv

  16. [26]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023. URL https://arxiv.org/abs/2301.13688

    2023 arXiv

  17. [27]

    Merging models with fisher-weighted averaging, 2022 a

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging, 2022 a . URL https://arxiv.org/abs/2111.09832

    2022 arXiv

  18. [28]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35: 0 17703--17716, 2022 b

    2022

  19. [29]

    Skeletonization: A technique for trimming the fat from a network via relevance assessment

    Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. Advances in neural information processing systems, 1, 1988

    1988

  20. [30]

    Learning to route among specialized experts for zero-shot generalization

    Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization. International Conference on Machine Learning, 2024. doi:10.48550/arXiv.2402.05859

    2024 doi arXiv

  21. [31]

    Towards modular llms by building and reusing a library of loras, 2024

    Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. Towards modular llms by building and reusing a library of loras, 2024. URL https://arxiv.org/abs/2405.11157

    2024 arXiv

  22. [32]

    Lottery ticket adaptation: Mitigating destructive interference in llms, 2024

    Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in llms, 2024. URL https://arxiv.org/abs/2406.16797

    2024 arXiv

  23. [33]

    Lora soups: Merging loras for practical skill composition tasks

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. arXiv preprint arXiv:2410.13025, 2024

    2024 arXiv

  24. [34]

    Controlling text-to-image diffusion by orthogonal finetuning

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Sch \"o lkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36: 0 79320--79362, 2023

    2023

  25. [35]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019. URL http://arxiv.org/abs/1910.10683

    1910 arXiv

  26. [36]

    Overcoming catastrophic forgetting with hard attention to the task

    Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp.\ 4548--4557. PMLR, 2018

    2018

  27. [37]

    In defense of structural sparse adapters for concurrent llm serving

    Junda Su, Zirui Liu, Zeju Qiu, Weiyang Liu, and Zhaozhuo Xu. In defense of structural sparse adapters for concurrent llm serving. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 4948--4953, 2024

    2024

  28. [38]

    Picking winning tickets before training by preserving gradient flow

    Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020

    2002 arXiv

  29. [39]

    Sampling generative networks, 2016

    Tom White. Sampling generative networks, 2016. URL https://arxiv.org/abs/1609.04468

    2016 arXiv

  30. [40]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022 arXiv

  31. [41]

    Ties-merging: Resolving interference when merging models, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models, 2023. URL https://arxiv.org/abs/2306.01708

    2023 arXiv

  32. [42]

    A survey on model moerging: Recycling and routing among specialized experts for collaborative learning

    Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.07057, 2024 a

    2024 arXiv

  33. [43]

    What matters for model merging at scale?, 2024 b

    Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale?, 2024 b . URL https://arxiv.org/abs/2410.03617

    2024 arXiv

  34. [44]

    Adamerging: Adaptive model merging for multi-task learning, 2024

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning, 2024. URL https://arxiv.org/abs/2310.02575

    2024 arXiv

  35. [45]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  36. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  37. [47]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  38. [48]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  39. [49]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.