Pith. sign in

REVIEW 2 major objections 4 minor 97 references

A forward-only controller sets how much each domain participates in multi-task LoRA fine-tuning without labels, using competence and representation drift to reward headroom and positive transfer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:53 UTC pith:AZ3Q4IZW

load-bearing objection Solid PEFT methods paper: label-free competence + drift affinity drive an online participation QP that beats static mixtures and gradient surgery at half data; calibration is load-bearing but empirically backed, not a collapse. the 2 major comments →

arxiv 2607.03522 v1 pith:AZ3Q4IZW submitted 2026-07-03 cs.LG

Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation

Wei Zhang , Lin Tang , Ming Zhao , Yuxuan Wang This is my paper
classification cs.LG
keywords multi-task learningLoRAdomain participationlabel-free signalstransfer affinitycompetenceparameter-efficient fine-tuningdata mixture
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When one shared low-rank adapter is fine-tuned on many domains at once, the domains co-learn through common parameters, so some help each other and some interfere. Most pipelines ignore this and use a fixed, uniform mix of domains. This paper shows that two answers can be read from cheap forward passes alone: how much each domain should participate, and which domains should be co-trained. A small unlabeled probe per domain yields a competence score whose level tracks remaining headroom and whose rise tracks learning speed; the drift of those probe representations yields a signed affinity that predicts pairwise transfer. Both signals feed a small entropy-regularized quadratic program on the simplex whose solution sets each domain's participation—its loss weight and its share of the data budget—favoring high-headroom, still-learning, synergistic domains and damping interfering ones. The controller adds no trainable parameters, wraps any multi-task LoRA pipeline, and on five heterogeneous domains with two backbones improves average accuracy over uniform mixing, static mixtures, gradient-surgery optimizers, and online data selection while using half the data and cutting cross-domain gradient conflict.

Core claim

Both the right amount of domain participation and the right set of domains to co-train can be estimated without labels from forward passes of the current shared adapter: competence tracks domain risk (headroom and learning speed), representation drift yields signed cross-domain affinity that predicts transfer, and folding both into an entropy-regularized quadratic program on the simplex produces participation that performs transfer-aware water-filling and measurably improves multi-domain LoRA fine-tuning at half the data budget.

What carries the argument

CoDA (co-adaptive domain adaptation): every T steps, competence and affinity are read from unlabeled probes, then an entropy-regularized quadratic program J(π) = g⊤π + η π⊤Aπ − τ ∑ π_k log π_k is solved on the simplex; its unique fixed point (a contraction under 2η∥A∥_{2} < τ) sets each domain's joint loss weight and data share, realizing transfer-aware water-filling.

Load-bearing premise

The controller assumes that, on each domain's probe, the model is sufficiently calibrated so that its confidence ranks domains by true risk; if calibration fails badly on a domain's format, competence stops being a reliable headroom signal.

What would settle it

If competence no longer correlates with per-domain accuracy (or affinity no longer predicts leave-one-out transfer) on a new set of heterogeneous domains or a severely miscalibrated backbone, and the participation program therefore fails to beat a uniform mixture at the same data budget, the central claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper frames multi-domain LoRA fine-tuning as co-adaptive multi-task learning under a shared adapter and a data budget. It introduces two label-free, forward-only signals: a per-domain competence c_k (Eq. 2) whose level and trajectory estimate headroom and learning speed, and a signed cross-domain affinity A (Eq. 3) from probe-representation drift that predicts pairwise transfer. These enter an entropy-regularized quadratic program J(π) (Eq. 5) whose solution sets each domain’s participation (loss weight and data share). The controller is parameter-free, wraps any multi-task LoRA pipeline, and is supported by three propositions (competence tracks risk under calibration; the map is a contraction with unique fixed point; the solution is transfer-aware water-filling). Empirically, on five heterogeneous domains and two 7–8B backbones at a matched 50% budget, CoDA improves average accuracy over uniform/proportional/temperature/DoReMi mixtures, PCGrad/GradNorm, and GREATS, while reducing gradient conflict and nearly matching a label-using oracle.

Significance. If the results hold, the work supplies a practical, label-free controller for the two central levers of multi-task LoRA—how much each domain participates and which domains should be co-trained—under a binding budget. The combination of a competence signal, a drift-based affinity, and a small simplex program is cleanly motivated, and the paper supplies the right supporting apparatus: correlations against external oracles the controller never sees, ablations that isolate headroom/velocity/affinity, negative controls (single-domain and shuffled affinity), sensitivity sweeps, and three propositions with proofs (including a contraction argument and a water-filling interpretation). The empirical margin over strong baselines at half the data, plus the reduction in cross-domain gradient conflict, makes the method immediately useful for multi-domain instruction tuning and composable with sample selection and post-hoc merging.

major comments (2)
  1. Proposition 1 (Appendix A.2) and the validity of g_k (Eq. 4) rest on Assumption 1 (κ_k-calibration plus an ℓ-bi-Lipschitz confidence–entropy link). The paper reports a strong empirical correlation (r≈0.91) under its warmup protocol and notes the failure mode, but never reports ECE/κ_k on the probes, never tests a deliberately miscalibrated or format-novel domain, and never shows that the ranking by c_k remains faithful when the base model has never seen the format (code, biomedical). Because Table 2 shows that the learnability term alone already accounts for most of the gain, a short calibration diagnostic (or a controlled miscalibration experiment) is needed to underwrite the central label-free claim.
  2. The affinity estimator (Eq. 3) is validated by a high correlation with leave-one-out transfer (r≈0.94, Fig. 3a), yet the paper does not report how sensitive A is to probe size, layer choice under distribution shift, or early-training noise, nor whether the sign of A remains stable when domains are added or removed mid-run. Given that the affinity term is the distinctive relational contribution (Table 2: +0.7/+1.0), a brief stability check would strengthen the claim that participation is transfer-aware rather than merely headroom-aware.
minor comments (4)
  1. Figure 1(c) and the affinity matrix in Table 6 use slightly different numerical scales/entries; a single consistent matrix (or an explicit note that one is oracle transfer and the other is recovered A) would avoid confusion.
  2. The within-domain selector (middle-difficulty band) is ablated in Table 2(b), but the exact percentile cutoffs and whether they are recomputed each round are only stated in the appendix; a one-sentence clarification in §5 would help reproducibility.
  3. Notation: the same symbol A is used for the affinity matrix and (in places) for the LoRA factor; a brief disambiguation or a different symbol for the LoRA matrix would improve readability.
  4. The contraction condition 2η∥A∥2<τ is stated in Prop. 2 and satisfied by the defaults, but a short remark on how often it is checked (or enforced) in practice would be useful.

Circularity Check

0 steps flagged

No significant circularity: competence/affinity are external-oracle-validated label-free signals; the participation program and its three propositions are self-contained analysis of an explicitly defined objective, not forced by fit or self-citation.

full rationale

The derivation chain is open and non-circular. Competence (Eq. 2) is defined as mean normalized confidence on an unlabeled probe; affinity (Eq. 3) as cosine of representation drifts. Both are validated post-hoc against external oracles the controller never sees (per-domain accuracy r≈0.91; leave-one-out transfer r≈0.94). Marginal learnability g_k (Eq. 4) multiplies headroom and velocity; the participation objective J (Eq. 5) and fixed-point (Eq. 6) are written explicitly. Proposition 1 derives an affine link to risk only under the stated Assumption 1 (κ-calibration + bi-Lipschitz); it is not definitional. Propositions 2–3 are standard convex-analysis consequences (Hessian negative-definite under 2η∥A∥₂<τ; KKT water-filling) of that same objective; they do not import uniqueness theorems or ansätze from overlapping-author citations. The sole overlapping-author citation (Tang et al. 2026) appears only in related-work discussion of mergeability and is not used in any proof or controller equation. Hyperparameters η, τ are swept and shown robust; no parameter is fitted to the final accuracy table and then re-presented as a prediction. Negative controls (shuffled affinity, single-domain) further confirm the signals are not tautological. The paper is therefore self-contained against external benchmarks; score 0 is the correct outcome.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 3 invented entities

The central claim rests on a small set of hand-chosen controller hyperparameters, a domain calibration assumption that links confidence to risk, standard convex-analysis facts for the QP, and two constructed signals (competence and drift affinity) that are validated against oracles but are still paper-defined control statistics. No new physical entities are postulated; the invented objects are algorithmic signals and the participation program.

free parameters (7)
  • affinity weight η = 0.5
    Trades per-domain learnability against the quadratic co-learning term; default 0.5, swept in sensitivity.
  • entropy temperature τ = 0.5
    Controls smoothness of π and the contraction condition 2η∥A∥₂ < τ; default 0.5.
  • learnability floor β = 0.1
    Prevents domain starvation in g_k = h_k(β + v_k); default 0.1.
  • controller period T = 100 steps
    How often participation is re-solved; default 100 optimizer steps.
  • probe size m = 256
    Unlabeled examples per domain for competence and centroids; default 256.
  • EMA coefficient for competence/centroids = 0.5
    Smoothing for velocity and drift; default 0.5.
  • within-domain difficulty band = [20, 80] percentile
    Percentile band used by the default sample selector; [20, 80].
axioms (4)
  • domain assumption On each domain probe the model is κ_k-calibrated and the confidence–entropy link is ℓ-bi-Lipschitz (Assumption 1).
    Required for Proposition 1 that competence tracks 0/1 risk and velocity tracks risk reduction; invoked in Appendix A.1–A.2.
  • standard math If 2η∥A∥₂ < τ then J is strictly concave and the softmax fixed-point map is a contraction on the simplex.
    Standard Hessian and Banach fixed-point arguments used in Proposition 2 (Appendix A.3).
  • domain assumption Domains exhibit heterogeneous non-stationary learning curves and signed pairwise transfer under a shared adapter.
    Empirical premise motivating co-adaptive control (Observations 1–4, Section 4); without it the controller reduces to near-uniform participation.
  • ad hoc to paper Representation-drift cosine of probe centroids is a faithful surrogate for leave-one-out transfer.
    Design choice of the affinity estimator (Eq. 3); supported by correlation r≈0.94 but not derived from first principles.
invented entities (3)
  • Competence signal c_k (normalized mean confidence on unlabeled probe) independent evidence
    purpose: Label-free estimate of headroom and learning speed used to form marginal learnability g_k.
    Defined in Eq. 2; validated against accuracy but is a paper-constructed control statistic, not a prior standard MTL weight.
  • Cross-domain affinity A from probe-representation drift independent evidence
    purpose: Signed, label-free estimate of pairwise transfer for the quadratic co-learning term.
    Defined in Eq. 3; predicts oracle leave-one-out transfer but is a new operationalization for online LoRA participation.
  • CoDA participation program J(π) and transfer-aware water-filling solution no independent evidence
    purpose: Maps g and A into a unique participation vector that sets loss weights and data shares.
    Eqs. 5–6 and Proposition 3; the continuous soft grouping controller is the paper’s core invented mechanism.

pith-pipeline@v1.1.0-grok45 · 27501 in / 3708 out tokens · 38044 ms · 2026-07-12T01:53:39.838421+00:00 · methodology

0 comments
read the original abstract

Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain's participation -- jointly its loss weight and its share of the sampled data -- rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.

Figures

Figures reproduced from arXiv: 2607.03522 by Lin Tang, Ming Zhao, Wei Zhang, Yuxuan Wang.

Figure 1
Figure 1. Figure 1: Co-learning a shared adapter is governed by two things, both readable without labels. (a) Per-domain accuracy [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of CODA. At each round the current shared adapter is run forward (no labels, no backprop) over a small unlabeled probe per domain to read two signals: a per-domain competence ck (its level is headroom, its rise is learning speed) and a cross-domain affinity A from the drift of probe representations (which domains pull the adapter compatibly). A co￾adaptive controller solves an entropy-regularized … view at source ↗
Figure 3
Figure 3. Figure 3: Mechanism. (a) The forward-only affinity predicts oracle leave-one-out transfer over (domain, domain, round) triples [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) Data efficiency on QWEN-2.5-7B: CODA exceeds full-data accuracy with ∼20–30% of the data. (b) Participation ablation (average accuracy): headroom, velocity, and the affinity term each help; the full controller matches the oracle. (c) Robustness to affinity weight η and temperature τ . (d) Negative controls: CODA’s gain is large across domains but vanishes within a single domain, and the affinity benefi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

97 extracted references · 13 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Energy-Based Out-of-Distribution Detection , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  2. [2]

    arXiv preprint arXiv:2401.12689 , year=

    Energy-based automated model evaluation , author=. arXiv preprint arXiv:2401.12689 , year=

    Pith/arXiv arXiv

  3. [3]

    International Conference on Learning Representations (ICLR) , year=

    Your Classifier Is Secretly an Energy Based Model and You Should Treat It Like One , author=. International Conference on Learning Representations (ICLR) , year=

  4. [4]

    International Conference on Learning Representations (ICLR) , year=

    A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , author=. International Conference on Learning Representations (ICLR) , year=

  5. [5]

    International Conference on Learning Representations (ICLR) , year=

    Leveraging Unlabeled Data to Predict Out-of-Distribution Performance , author=. International Conference on Learning Representations (ICLR) , year=

  6. [6]

    International Conference on Machine Learning (ICML) , year=

    Predicting Out-of-Distribution Error with the Projection Norm , author=. International Conference on Machine Learning (ICML) , year=

  7. [7]

    IEEE/CVF International Conference on Computer Vision (ICCV) , year=

    Predicting with Confidence on Unseen Distributions , author=. IEEE/CVF International Conference on Computer Vision (ICCV) , year=

  8. [8]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  9. [9]

    Transactions on Machine Learning Research (TMLR) , year=

    A Survey on Data Selection for Language Models , author=. Transactions on Machine Learning Research (TMLR) , year=

  10. [10]

    Zhou, Chunting and Liu, Pengfei and Xu, Puxin and Iyer, Srinivasan and Sun, Jiao and Mao, Yuning and Ma, Xuezhe and Efrat, Avia and Yu, Ping and Yu, Lili and others , booktitle=

  11. [11]

    Xia, Mengzhou and Malladi, Sadhika and Gururangan, Suchin and Arora, Sanjeev and Chen, Danqi , booktitle=

  12. [12]

    Chen, Lichang and Li, Shiyang and Yan, Jun and Wang, Hai and Gunaratna, Kalpa and Yadav, Vikas and Tang, Zheng and Srinivasan, Vijay and Zhou, Tianyi and Huang, Heng and Jin, Hongxia , booktitle=

  13. [13]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Data Selection for Language Models via Importance Resampling , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  14. [14]

    Wettig, Alexander and Gupta, Aatmik and Malik, Saumya and Chen, Danqi , booktitle=

  15. [15]

    International Conference on Learning Representations (ICLR) , year=

    What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning , author=. International Conference on Learning Representations (ICLR) , year=

  16. [16]

    International Conference on Machine Learning (ICML) , year=

    Prioritized Training on Points That Are Learnable, Worth Learning, and Not Yet Learnt , author=. International Conference on Machine Learning (ICML) , year=

  17. [17]

    Wang, Jiachen T and Wu, Tong and Song, Dawn and Mittal, Prateek and Jia, Ruoxi , booktitle=

  18. [18]

    arXiv preprint arXiv:2510.16882 , year=

    Utility-diversity aware online batch selection for llm supervised fine-tuning , author=. arXiv preprint arXiv:2510.16882 , year=

    arXiv

  19. [19]

    The Long-Term Effects of Data Selection in

    Yang, Yuxin and Zeng, Aoxiong and Yang, Xiangquan , journal=. The Long-Term Effects of Data Selection in

  20. [20]

    arXiv preprint arXiv:1511.06343 , year=

    Online Batch Selection for Faster Training of Neural Networks , author=. arXiv preprint arXiv:1511.06343 , year=

    Pith/arXiv arXiv

  21. [21]

    International Conference on Machine Learning (ICML) , year=

    Not All Samples Are Created Equal: Deep Learning with Importance Sampling , author=. International Conference on Machine Learning (ICML) , year=

  22. [22]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Deep Learning on a Data Diet: Finding Important Examples Early in Training , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  23. [23]

    International Conference on Learning Representations (ICLR) , year=

    Selection via Proxy: Efficient Data Selection for Deep Learning , author=. International Conference on Learning Representations (ICLR) , year=

  24. [24]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  25. [25]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Deduplicating Training Data Makes Language Models Better , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  26. [26]

    Tirumala, Kushal and Simig, Daniel and Aghajanyan, Armen and Morcos, Ari , booktitle=

  27. [27]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    No Train No Gain: Revisiting Efficient Training Algorithms for Transformer-Based Language Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  28. [28]

    Xie, Sang Michael and Pham, Hieu and Dong, Xuanyi and Du, Nan and Liu, Hanxiao and Lu, Yifeng and Liang, Percy S and Le, Quoc V and Ma, Tengyu and Yu, Adams Wei , booktitle=

  29. [29]

    arXiv preprint arXiv:2403.16952 , year=

    Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance , author=. arXiv preprint arXiv:2403.16952 , year=

    Pith/arXiv arXiv

  30. [30]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Chen, Mayee F and Roberts, Nicholas and Bhatia, Kush and Wang, Jue and Zhang, Ce and Sala, Frederic and R. Advances in Neural Information Processing Systems (NeurIPS) , year=

  31. [31]

    Stanford

    Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B , year=. Stanford

  32. [32]

    Chaudhary, Sahil , year=. Code

  33. [33]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Self-Instruct: Aligning Language Models with Self-Generated Instructions , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  34. [34]

    International Conference on Learning Representations (ICLR) , year=

    Finetuned Language Models Are Zero-Shot Learners , author=. International Conference on Learning Representations (ICLR) , year=

  35. [35]

    International Conference on Machine Learning (ICML) , year=

    Curriculum Learning , author=. International Conference on Machine Learning (ICML) , year=

  36. [36]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Self-Paced Learning for Latent Variable Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  37. [37]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Gradient Surgery for Multi-Task Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  38. [38]

    Chen, Zhao and Badrinarayanan, Vijay and Lee, Chen-Yu and Rabinovich, Andrew , booktitle=

  39. [39]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Conflict-Averse Gradient Descent for Multi-Task Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  40. [40]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Multi-Task Learning as Multi-Objective Optimization , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  41. [41]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

  42. [42]

    arXiv preprint arXiv:2009.09796 , year=

    Multi-Task Learning with Deep Neural Networks: A Survey , author=. arXiv preprint arXiv:2009.09796 , year=

    Pith/arXiv arXiv 2009

  43. [43]

    International Conference on Machine Learning (ICML) , year=

    Which Tasks Should Be Learned Together in Multi-Task Learning? , author=. International Conference on Machine Learning (ICML) , year=

  44. [44]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Efficiently Identifying Task Groupings for Multi-Task Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  45. [45]

    arXiv preprint arXiv:1907.05019 , year=

    Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges , author=. arXiv preprint arXiv:1907.05019 , year=

    Pith/arXiv arXiv 1907

  46. [46]

    International Conference on Learning Representations (ICLR) , year=

    Editing Models with Task Arithmetic , author=. International Conference on Learning Representations (ICLR) , year=

  47. [47]

    Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin A and Bansal, Mohit , journal=

  48. [48]

    International Conference on Machine Learning (ICML) , year=

    Language Models Are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch , author=. International Conference on Machine Learning (ICML) , year=

  49. [49]

    Merging Models with

    Matena, Michael S and Raffel, Colin A , journal=. Merging Models with

  50. [50]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  51. [51]

    arXiv preprint arXiv:2310.04742 , year=

    Parameter-Efficient Multi-Task Model Fusion with Partial Linearization , author=. arXiv preprint arXiv:2310.04742 , year=

    Pith/arXiv arXiv

  52. [52]

    International Conference on Learning Representations (ICLR) , year=

    Dataless Knowledge Fusion by Merging Weights of Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  53. [53]

    Huang, Chengsong and Liu, Qian and Lin, Bill Yuchen and Pang, Tianyu and Du, Chao and Lin, Min , booktitle=

  54. [54]

    International Conference on Machine Learning (ICML) , year=

    Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time , author=. International Conference on Machine Learning (ICML) , year=

  55. [55]

    Model Merging with

    Stoica, George and Ramesh, Pratik and Ecsedi, Boglarka and Choshen, Leshem and Hoffman, Judy , booktitle=. Model Merging with

  56. [56]

    arXiv preprint arXiv:2606.19549 , year=

    Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates , author=. arXiv preprint arXiv:2606.19549 , year=

    Pith/arXiv arXiv

  57. [57]

    Decouple and Orthogonalize: A Data-Free Framework for

    Zheng, Shenghe and Wang, Hongzhi and Huang, Chenyu and Wang, Xiaohui and Chen, Tao and Fan, Jiayuan and Hu, Shuyue and Ye, Peng , journal=. Decouple and Orthogonalize: A Data-Free Framework for

  58. [58]

    Unraveling

    Zhang, Haobo and Zhou, Jiayu , booktitle=. Unraveling

  59. [59]

    Findings of the Association for Computational Linguistics: EMNLP , year=

    Orthogonal Subspace Learning for Language Model Continual Learning , author=. Findings of the Association for Computational Linguistics: EMNLP , year=

  60. [60]

    Transactions on Machine Learning Research (TMLR) , year=

    Soft Merging of Experts with Adaptive Routing , author=. Transactions on Machine Learning Research (TMLR) , year=

  61. [61]

    Parameter-Efficient Transfer Learning for

    Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanislaw and Morrone, Bruna and De Laroussilhe, Quentin and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain , booktitle=. Parameter-Efficient Transfer Learning for

  62. [62]

    Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

  63. [63]

    Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke , journal=

  64. [64]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Prefix-Tuning: Optimizing Continuous Prompts for Generation , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  65. [65]

    Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    The Power of Scale for Parameter-Efficient Prompt Tuning , author=. Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

  66. [66]

    Zhang, Qingru and Chen, Minshuo and Bukharin, Alexander and Karampatziakis, Nikos and He, Pengcheng and Cheng, Yu and Chen, Weizhu and Zhao, Tuo , journal=

  67. [67]

    Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung , booktitle=

  68. [68]

    arXiv preprint arXiv:2311.11696 , year=

    Sparse Low-Rank Adaptation of Pre-Trained Language Models , author=. arXiv preprint arXiv:2311.11696 , year=

    Pith/arXiv arXiv

  69. [69]

    Kopiczko, Dawid Jan and Blankevoort, Tijmen and Asano, Yuki M , booktitle=

  70. [70]

    arXiv preprint arXiv:2012.13255 , year=

    Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning , author=. arXiv preprint arXiv:2012.13255 , year=

    Pith/arXiv arXiv 2012

  71. [71]

    Hu, Zhiqiang and Wang, Lei and Lan, Yihuai and Xu, Wanyu and Lim, Ee-Peng and Bing, Lidong and Xu, Xing and Poria, Soujanya and Lee, Roy Ka-Wei , booktitle=

  72. [72]

    Zou, Heming and Zang, Yunliang and Xu, Wutong and Zhu, Yao and Ji, Xiangyang , journal=

  73. [73]

    Dou, Shihan and Zhou, Enyu and Liu, Yan and Gao, Songyang and Zhao, Jun and Shen, Wei and Zhou, Yuhao and Xi, Zhiheng and Wang, Xiao and Fan, Xiaoran and others , journal=

  74. [74]

    Li, Dengchun and Ma, Yingzi and Wang, Naizheng and Ye, Zhengmao and Cheng, Zhiyuan and Tang, Yinghao and Zhang, Yan and Duan, Lei and Zuo, Jie and Yang, Cal and others , journal=

  75. [75]

    Mixture of

    Wu, Xun and Huang, Shaohan and Wei, Furu , booktitle=. Mixture of

  76. [76]

    Tian, Chunlin and Shi, Zhan and Guo, Zhijiang and Li, Li and Xu, Cheng-Zhong , journal=

  77. [77]

    Higher Layers Need More

    Gao, Chongyang and Chen, Kezhen and Rao, Jinmeng and Sun, Baochen and Liu, Ruibo and Peng, Daiyi and Zhang, Yawen and Guo, Xiaoyuan and Yang, Jie and Subrahmanian, VS , journal=. Higher Layers Need More

  78. [78]

    Liao, Mengqi and Chen, Wei and Shen, Junfeng and Guo, Shengnan and Wan, Huaiyu , booktitle=

  79. [79]

    Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    Mixture-of-Subspaces in Low-Rank Adaptation , author=. Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

  80. [80]

    Pushing Mixture of Experts to the Limit: Extremely Parameter-Efficient

    Zadouri, Ted and. Pushing Mixture of Experts to the Limit: Extremely Parameter-Efficient. International Conference on Learning Representations (ICLR) , year=

Showing first 80 references.