Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Tensorized Clustered LoRA Merging for Multi-Task Interference

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TC-LoRA reduces cross-task interference in LoRA merging by clustering training samples in embedding space and jointly CP-decomposing the per-cluster adapters.

desk verdict A plausible incremental method for LoRA merging, but the supplied full text is corrupted, so the evidence is currently unverifiable; the abstract's aggregate gains do not establish the mechanism. read the letter →

arxiv 2508.03999 v1 pith:TFQPEI6J submitted 2025-08-06 cs.LG

Zhan Su , Fengran Mo , Guojun Liang , Jinghan Zhang , Bingbing Wen , Prayag Tiwari , Jian-Yun Nie This is my paper
classification cs.LG
keywords LoRAmergingtaskinterferenceCPdecompositionembeddingclusteringmulti-taskfine-tuninglow-rankadapterszero-shottransferskillcomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that task interference when merging multiple LoRA adapters into one large language model can be reduced by acting at two levels: grouping training samples by embedding-space similarity and training a dedicated adapter per cluster, then jointly decomposing all adapters with a canonical polyadic factorization that separates shared and task-specific factors. If this works, multi-task LoRA libraries can be merged without each task dragging down the others' performance. The reported evidence is that TC-LoRA beats SVD-based merging baselines by +1.4% accuracy on Phi-3 and +2.3% on Mistral-7B across out-of-domain zero-shot and skill-composition tasks.

What carries the argument

The central object is the joint Canonical Polyadic (CP) decomposition applied to the set of per-cluster LoRA adapters, combined with embedding-space clustering of training samples. The clustering defines the adapter library; the CP decomposition factorizes the stacked adapters into shared and task-specific rank-one components, which is what is claimed to disentangle task knowledge.

What would settle it

Replace the embedding-space clustering with random grouping or label-based grouping while keeping the CP decomposition fixed; if out-of-domain zero-shot accuracy does not drop below the reported TC-LoRA results, then the text-level clustering is not carrying the claimed benefit.

Watch

Extended reading notes

Core claim

The central claim is that cross-task interference in LoRA merging is addressable at both the text level and the parameter level. At the text level, clustering training samples by embedding similarity yields specialized per-cluster adapters that do not fight each other when merged. At the parameter level, a joint Canonical Polyadic (CP) decomposition of these adapters expresses them as a sum of rank-1 factor tensors, allowing shared factors to be reused while task-specific factors remain separate. The paper argues that this two-level design preserves essential task knowledge and reduces interference, and supports it with experiments showing accuracy gains over SVD-based merging baselines on P

Load-bearing premise

The load-bearing premise is that training samples that interfere with each other during merging are exactly those close in embedding space, so that per-cluster specialization plus joint factorization can separate task knowledge; if embedding similarity does not track interference, the text-level step would not help.

Editorial extensions

If this is right

  • Merging LoRA adapters for many tasks can be done without retraining the base model, making multi-task adaptation cheaper and more modular.
  • Embedding-space clustering provides a principled way to decide how many adapters to train, rather than one per task.
  • Joint factorization could allow selective pruning of task-specific factors to control which skills survive a merge.
  • The approach could extend to other parameter-efficient fine-tuning methods that produce low-rank delta matrices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The embedding-clustering assumption implies that interference is correlated with input-format similarity; a testable prediction is that clusters defined by output format or label distribution would perform worse.
  • If the CP factorization truly separates task-specific and shared factors, the shared factors should transfer to unseen tasks with minimal fine-tuning, which the paper does not test.
  • The reported gains might be larger when tasks are more heterogeneous, since interference grows with task diversity; the current experiments sample only reasoning, question answering, and coding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TC-LoRA, a method for merging LoRA adapters in multi-task settings by addressing interference at two levels. At the text level, training samples are clustered in embedding space and a specialized LoRA adapter is trained per cluster. At the parameter level, a joint Canonical Polyadic (CP) decomposition is applied to the per-cluster adapters to separate shared and task-specific factors. The abstract reports +1.4% accuracy on Phi-3 and +2.3% on Mistral-7B over strong SVD-based baselines on out-of-domain zero-shot and skill-composition tasks. The submitted full text is encoding-corrupted and unreadable, so the method details, equations, tables, and ablations cannot be inspected.

Significance. If the reported gains are real and attributable to the two named mechanisms, this is a modest but useful contribution to parameter-efficient multi-task adaptation, connecting embedding-space data similarity with tensor-factorization-based merging. The CP decomposition approach to LoRA merging is a reasonable design point and may offer computational or storage advantages. However, the evidence as presented is insufficient: single-point aggregate improvements over unnamed baselines do not establish that embedding clustering and CP disentanglement matter. The paper appears to be an empirical recipe with no visible machine-checked proofs or reproducible code, so its significance is conditional on a full, readable experimental validation.

major comments (3)
  1. [Full text (entire manuscript)] The supplied full text is unreadable: it consists of garbled/mojibake characters, so equations, algorithm boxes, tables, ablation results, and any limitation statements cannot be verified. This is a load-bearing defect because the paper's central claim is empirical. The authors must provide a properly encoded manuscript and explicitly identify the sections/tables that support the +1.4% and +2.3% accuracy claims.
  2. [Abstract (experimental claim)] The experimental claim is under-specified: it reports single-point aggregate accuracy deltas over unnamed 'strong SVD-based baselines,' with no error bars, number of seeds, significance tests, per-task results, task/dataset lists, or hyperparameters. At 1–2 percentage points, these gains are within typical run-to-run variance on zero-shot benchmarks. The grammatical fragment ('Extensive experiments ... including reasoning, question answering, and coding.') and duplicated '+2.3%' further obscure the evaluation.
  3. [Abstract (mechanism attribution)] The paper attributes the improvement to two mechanisms: embedding-space clustering and joint CP decomposition. No ablation is visible (or, due to corruption, locatable) that replaces embedding clustering with random partitions, or that uses SVD merging on the same per-cluster adapters in place of CP merging. Without these ablations, the mechanism claim is not supported; the gains could come from training separate adapters on smaller data subsets (implicit regularization) or from an increased total parameter budget. Please add ablations and a sensitivity analysis over cluster count and CP rank.
minor comments (4)
  1. [Abstract] The final sentence is a grammatical fragment; complete it and state the benchmarks, metrics, and baselines used.
  2. [Abstract] Duplicate '+2.3%' appears after 'Mistral-7B'; typo.
  3. [Abstract] 'targeting to address' is awkward; consider 'addresses task interference'.
  4. [Full text] Section numbering, references, and any limitation statements cannot be inspected due to the encoding corruption; ensure the final submission has a readable PDF with complete references and a limitations discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TC-LoRA is an empirical merging recipe evaluated on held-out benchmarks; no fitted quantity is relabeled as a prediction.

full rationale

The paper's load-bearing claim is an accuracy comparison on out-of-domain zero-shot and skill-composition benchmarks. The method has two stages: (1) clustering training samples in embedding space, then training a per-cluster LoRA adapter; (2) joint CP decomposition of the trained adapters to produce a merged model. Neither stage fits parameters to the evaluation tasks. The CP decomposition is fit to the already-trained adapters as a representation/merging step, and the target quantities (out-of-domain accuracy) are not part of that fit. No equation in the readable abstract defines cluster assignments in terms of evaluation labels, and no 'prediction' is the renamed value of a fitted parameter. The abstract contains no self-citation or uniqueness theorem invoked to forbid alternatives. The supplied full text is too corrupted to verify ablations, but unreadability is an evidence gap, not circularity. Therefore no meaningful circularity is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

Abstract-only review. The central claim rests on two unvalidated modeling premises (text-level clustering validity and parameter-level CP separability) plus standard tensor-decomposition math. The only identifiable free parameters are the cluster count and the CP rank; LoRA rank and alpha are presumably inherited from standard single-task LoRA practice. No invented entities (new particles, mediators, forces, dimensions, ledger entries) are introduced at the abstract level. Full audit requires the equations and experimental settings in the unreadable body.

free parameters (2)
  • Number of text-level sample clusters
    The abstract specifies clustering in embedding space without stating the algorithm, the number of clusters, or the selection criterion; every per-cluster LoRA adapter depends on this count.
  • CP decomposition rank
    The joint CP rank controls how much shared versus task-specific structure is retained; it is not stated in the abstract, is typically hand-set or tuned, and directly determines the reconstruction fidelity of the merged adapters.
assumptions (3)
  • domain assumption Sample-level clustering in embedding space captures input-format similarities that align with the interference structure between tasks, so per-cluster adapters merged together interfere less.
    This is the entire text-level mechanism. The abstract asserts it ('cluster the training samples in the embedding space to capture input-format similarities') without proof or stated validation, and the improvement claim depends on it.
  • domain assumption The per-cluster LoRA adapters admit a joint CP decomposition that separates task-specific factors from shared factors, and truncating or merging in this factor space preserves task knowledge while removing interference.
    The parameter-level mechanism presumes a low-rank CP structure exists across heterogeneous adapters and that this structure corresponds to task-specific versus shared knowledge. This is a modeling choice, not a consequence of tensor algebra alone.
  • standard math Standard tensor-decomposition mathematics (existence and approximation properties of CP decompositions, including non-convex fitting reaching a usable local optimum) holds as background.
    Any use of CP relies on established tensor-decomposition background; the abstract gives no sign the paper derives new decomposition theory, so standard results are assumed unstated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tensorized Clustered LoRA Merging for Multi-Task Interference." pith.science (2026). https://pith.science/paper/TFQPEI6J

@misc{pith2026250803999,
  author       = {Pith},
  title        = {Pith review of: Tensorized Clustered LoRA Merging for Multi-Task Interference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TFQPEI6J}},
  note         = {Machine review of arXiv:2508.03999}
}
read the original abstract

Despite the success of the monolithic dense paradigm of large language models (LLMs), the LoRA adapters offer an efficient solution by fine-tuning small task-specific modules and merging them with the base model. However, in multi-task settings, merging LoRA adapters trained on heterogeneous sources frequently causes \textit{task interference}, degrading downstream performance. To address this, we propose a tensorized clustered LoRA (TC-LoRA) library targeting to address the task interference at the \textit{text-level} and \textit{parameter-level}. At the \textit{text-level}, we cluster the training samples in the embedding space to capture input-format similarities, then train a specialized LoRA adapter for each cluster. At the \textit{parameter-level}, we introduce a joint Canonical Polyadic (CP) decomposition that disentangles task-specific and shared factors across LoRA adapters. This joint factorization preserves essential knowledge while reducing cross-task interference. Extensive experiments on out-of-domain zero-shot and skill-composition tasks-including reasoning, question answering, and coding. Compared to strong SVD-based baselines, TC-LoRA achieves +1.4\% accuracy on Phi-3 and +2.3\% on Mistral-7B (+2.3\%), demonstrating the effectiveness of TC-LoRA in LLM adaptation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PMDRouter selects LoRAs zero-shot by decoding scale-normalized linear response energy from one adapter-free backbone prefill, and leads most internal-signal baselines on a new multi-granularity EPM bench.

Reference graph

Works this paper leans on

53 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    A.; Awan, A

    Abdin, M.; Jacobs, S. A.; Awan, A. A.; Aneja, J.; Awadallah, A.; Awadalla, H.; Bach, N.; Bahree, A.; Bakhtiari, A.; Behl, H.; et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  3. [3]

    H.; Sanh, V.; Yong, Z.-X.; Webson, A.; Raffel, C.; Nayak, N

    Bach, S. H.; Sanh, V.; Yong, Z.-X.; Webson, A.; Raffel, C.; Nayak, N. V.; Sharma, A.; Kim, T.; Bari, M. S.; Fevry, T.; et al. 2022. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279

    arXiv 2022

  4. [4]

    Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 7432--7439

    work page 2020

  5. [5]

    Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  6. [6]

    E.; Fraser, A.; and Dodge, J

    Chronopoulou, A.; Peters, M. E.; Fraser, A.; and Dodge, J. 2023. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027

    arXiv 2023

  7. [7]

    W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al

    Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70): 1--53

    2024

  8. [8]

    Clark, C.; Lee, K.; Chang, M.-W.; Kwiatkowski, T.; Collins, M.; and Toutanova, K. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL

    work page 2019

Show all 53 references
  1. [9]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  2. [10]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Davari, M.; and Belilovsky, E. 2024. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision, 270--287. Springer

    2024

  4. [12]

    De Lathauwer, L.; De Moor, B.; and Vandewalle, J. 2000. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4): 1253--1278

    2000

  5. [13]

    Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, 10764--10799. PMLR

    2023

  6. [14]

    A.; Crisostomi, D.; Bucarelli, M

    Gargiulo, A. A.; Crisostomi, D.; Bucarelli, M. S.; Scardapane, S.; Silvestri, F.; and Rodola, E. 2025. Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference, 18695--18705

    2025

  7. [15]

    M.; and Kim, Y

    Guo, D.; Rush, A. M.; and Kim, Y. 2020. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463

    2020 arXiv

  8. [16]

    Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019 a . Parameter-efficient transfer learning for NLP. In International conference on machine learning, 2790--2799. PMLR

    2019

  9. [17]

    Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019 b . Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, 2790--2799. PMLR

    2019

  10. [18]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  11. [19]

    Y.; Pang, T.; Du, C.; and Lin, M

    Huang, C.; Liu, Q.; Lin, B. Y.; Pang, T.; Du, C.; and Lin, M. 2023. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269

    2023 arXiv

  12. [20]

    T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A

    Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    2022 arXiv

  13. [21]

    Jiang, A.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B (2023). arXiv preprint arXiv:2310.06825

    2023 arXiv

  14. [22]

    Jin, X.; Ren, X.; Preotiuc-Pietro, D.; and Cheng, P. 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849

    2022 arXiv

  15. [23]

    Lester, B.; Al-Rfou, R.; and Constant, N. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  16. [24]

    L.; and Liang, P

    Li, X. L.; and Liang, P. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190

    2021 arXiv

  17. [25]

    F.; Cheng, K.-T.; and Chen, M.-H

    Liu, S.-Y.; Wang, C.-Y.; Yin, H.; Molchanov, P.; Wang, Y.-C. F.; Cheng, K.-T.; and Chen, M.-H. 2024. DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353

    2024 arXiv

  18. [26]

    W.; Tay, Y.; Zhou, D.; Le, Q

    Longpre, S.; Hou, L.; Vu, T.; Webson, A.; Chung, H. W.; Tay, Y.; Zhou, D.; Le, Q. V.; Zoph, B.; Wei, J.; et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, 22631--22648. PMLR

    2023

  19. [27]

    Lu, Z.; Fan, C.; Wei, W.; Qu, X.; Chen, D.; and Cheng, Y. 2024. Twin-merging: Dynamic integration of modular expertise in model merging. Advances in Neural Information Processing Systems, 37: 78905--78935

    2024

  20. [28]

    D.; and van de Weijer, J

    Marczak, D.; Magistri, S.; Cygert, S.; Twardowski, B.; Bagdanov, A. D.; and van de Weijer, J. 2025. No task left behind: Isotropic model merging with common and task-specific subspaces. arXiv preprint arXiv:2502.04959

    2025 arXiv

  21. [29]

    S.; and Raffel, C

    Matena, M. S.; and Raffel, C. A. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35: 17703--17716

    2022

  22. [30]

    Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    2018 arXiv

  23. [31]

    M.; Charlin, L.; Roux, N

    Ostapenko, O.; Su, Z.; Ponti, E. M.; Charlin, L.; Roux, N. L.; Pereira, M.; Caccia, L.; and Sordoni, A. 2024. Towards modular llms by building and reusing a library of loras. arXiv preprint arXiv:2405.11157

    2024 arXiv

  24. [32]

    Pfeiffer, J.; Kamath, A.; R \"u ckl \'e , A.; Cho, K.; and Gurevych, I. 2020. AdapterFusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247

    2020 arXiv

  25. [33]

    Prabhakar, A.; Li, Y.; Narasimhan, K.; Kakade, S.; Malach, E.; and Jelassi, S. 2024. Lora soups: Merging loras for practical skill composition tasks. arXiv preprint arXiv:2410.13025

    2024 arXiv

  26. [34]

    L.; Bhagavatula, C.; and Choi, Y

    Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9): 99--106

    2021

  27. [35]

    Sheng, Y.; Cao, S.; Li, D.; Hooper, C.; Lee, N.; Yang, S.; Chou, C.; Zhu, B.; Zheng, L.; Keutzer, K.; et al. 2023. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285

    2023 arXiv

  28. [36]

    Stewart, G. W. 1993. On the early history of the singular value decomposition. SIAM review, 35(4): 551--566

    1993

  29. [37]

    Stoica, G.; Ramesh, P.; Ecsedi, B.; Choshen, L.; and Hoffman, J. 2024. Model merging with SVD to tie the Knots. arXiv preprint arXiv:2410.19735

    2024 arXiv

  30. [38]

    W.; Chowdhery, A.; Le, Q

    Suzgun, M.; Scales, N.; Sch \"a rli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q. V.; Chi, E. H.; Zhou, D.; et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  31. [39]

    Tucker, L. R. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279--311

    1966

  32. [40]

    Wang, H.; Yang, X.; Chang, J.; Jin, D.; Sun, J.; Zhang, S.; Luo, X.; and Tian, Q. 2023. Parameter-efficient tuning of large-scale multimodal foundation model. Advances in Neural Information Processing Systems, 36: 15752--15774

    2023

  33. [41]

    Wang, L.; Zhang, X.; Su, H.; and Zhu, J. 2024. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  34. [42]

    Y.; Guu, K.; Yu, A

    Wei, J.; Bosma, M.; Zhao, V. Y.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652

    2021 arXiv

  35. [43]

    Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A

    Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.; Carmon, Y.; Kornblith, S.; et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Inter...

    2022

  36. [44]

    A.; and Bansal, M

    Yadav, P.; Tam, D.; Choshen, L.; Raffel, C. A.; and Bansal, M. 2023. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36: 7093--7115

    2023

  37. [45]

    Yang, E.; Shen, L.; Guo, G.; Wang, X.; Cao, X.; Zhang, J.; and Tao, D. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666

    2024 arXiv

  38. [46]

    Yu, L.; Yu, B.; Yu, H.; Huang, F.; and Li, Y. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning

    2024

  39. [47]

    B.; Ravfogel, S.; and Goldberg, Y

    Zaken, E. B.; Ravfogel, S.; and Goldberg, Y. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199

    2021

  40. [48]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  41. [49]

    Zhang, Q.; Chen, M.; Bukharin, A.; He, P.; Cheng, Y.; Chen, W.; and Zhao, T. 2023 a . Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512

    2023 arXiv

  42. [50]

    Zhang, R.; Han, J.; Zhou, A.; Hu, X.; Yan, S.; Lu, P.; Li, H.; Gao, P.; and Qiao, Y. 2023 b . Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199

    2023 arXiv

  43. [51]

    Zhao, Z.; Gan, L.; Wang, G.; Hu, Y.; Shen, T.; Yang, H.; Kuang, K.; and Wu, F. 2024. Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning. arXiv preprint arXiv:2406.16989

    2024 arXiv

  44. [52]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.