Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fine-tuning performance on any subset of datasets can be predicted from a single gradient pass at the base model, with under 5% error and 105x less compute, and that the resulting grouped adapter ensemble raises…

desk verdict Useful engineering extension with a reproducible ensemble recipe, but the paper overclaims the role of its gradient-based task grouping; the missing oracle/random-grouping ablation leaves the central mechanism unsupported. read the letter →

arxiv 2505.21930 v1 pith:DOSG6QPN submitted 2025-05-28 cs.LG cs.CL

Dongyue Li , Ziniu Zhang , Lu Wang , Hongyang R. Zhang This is my paper
classification cs.LGcs.CL
keywords multi-datasetfine-tuningparameter-efficientlow-rankadapterstaskaffinitygroupinggradient-basedlossestimationadapterensemblesQLoRAfederatedlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that fine-tuning a language model on many datasets at once can be made much cheaper by grouping similar datasets and ensembling one small adapter per group instead of training one shared adapter on everything. Its central lever is an empirical regularity: fine-tuned low-rank adapters such as LoRA and QLoRA stay within roughly 0.2% relative distance of the base model, so a first-order Taylor expansion around the base weights makes the loss look like a simple logistic regression in the adapter weights. That turns estimating fine-tuning performance on any subset of datasets into a single gradient pass at the base model plus a set of cheap regressions, which the paper reports achieves under 5% error while cutting compute by 105x. If true, this would let practitioners decide how to group tasks and assemble adapters for large models at a fraction of the usual fine-tuning cost, with accuracy gains over single-adapter QLoRA of up to 10 percentage points at 9% extra FLOPs.

What carries the argument

The machinery is the first-order Taylor expansion of the model output centered at the base weights, $\hat\ell(X) = \log(1+\exp(b - y\,g^\top (X-\theta^*)))$, where $b=-y\,h_{\theta^*}(s,y)$ and $g=\nabla h_{\theta^*}(s,y)$ are gradients evaluated once at the base model. Since fine-tuned adapters stay close to $\theta^*$ (relative weight distance under about 0.2%), the fine-tuned loss on any subset of datasets is approximated by solving a logistic regression in the adapter weights $X$, using random projections of the gradients as low-dimensional features. This produces the task-affinity matrix $T_{i,j}=\frac{1}{n_{i,j}}\sum_{S_k\ni i,j}\hat f_i(S_k)$, which is clustered via a semi-definite programming relaxation with trace regularization into m groups; the ensemble then averages the outputs of the group adapters, optionally adding gradient-boosting adapters fit to the negative gradients of high-loss groups.

What would settle it

Compute the actual fine-tuned loss for many random dataset subsets on a 34B-parameter model or larger, compare it with the first-order gradient estimate from the base model, and check whether the relative error stays under 5%; the approximation-error plots in the paper only show models up to 13B, so this measurement on 34B would directly test the claimed scope. A second concrete check is to measure the Taylor remainder $\|h_X - h_{\theta^*} - \nabla h_{\theta^*}^\top (X-\theta^*)\|/\|h_X\|$ for adapters trained to relative weight distances above 0.25%, where the reported residual errors begin to grow.

Watch

Extended reading notes

Core claim

The paper's central claim is that parameter-efficient fine-tuning methods stay so close to the pretrained initialization that first-order linearization of the model output is accurate enough to predict fine-tuned losses without fine-tuning. Using base-model gradients as features in a projected logistic regression, the paper estimates fine-tuning performance on random subsets of datasets, assembles an n-by-n task-affinity matrix T, and clusters the datasets into m groups. One adapter is fine-tuned per group, a few gradient-boosting steps are added, and the outputs are combined with learned weights. In experiments on ten SuperGLUE classification tasks, this ENSEMBLE LORA procedure raises QLoRA's average test accuracy by 10 percentage points (and QAdapter's by 9%) with roughly 8-9% extra FLOPs, and on a 34B Llama model it gives a 3-point improvement with 8% extra FLOPs; the paper additionally reports that small-rank adapters and adapter ensembles generalize better and converge to flatter minima.

Load-bearing premise

The load-bearing premise is that fine-tuned low-rank adapters stay so close to the base model (relative weight distance around 0.1-0.2%) that the model output, and hence the loss, is accurately linear in the adapter weights, so base-model gradients alone can predict fine-tuned loss on any dataset subset; if the loss surface curves meaningfully over the adapter update, the task-affinity estimates and the ensemble built on them would be unreliable.

Editorial extensions

If this is right

  • Fine-tuning performance of any subset of datasets can be estimated to within roughly 5% relative error without fine-tuning, at 105x lower compute than full fine-tuning.
  • Grouping datasets by these estimated affinities and ensembling group adapters improves QLoRA's average test accuracy by up to 10 percentage points and QAdapter's by 9%, with only 8-9% additional FLOPs.
  • On a 34B Llama model, the ensemble improves QLoRA accuracy by 3 percentage points with 8% additional FLOPs.
  • In a federated setting with 500 tasks, the method matches pretrain-then-fine-tune test loss while cutting compute by roughly 90% and memory by 91%.
  • Smaller low-rank adapters and ensembles of adapters show lower empirical generalization error and lower Hessian trace, indicating flatter minima, than a single larger adapter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the linearization estimate could be reused as a fast screening tool for data selection or negative-transfer detection in settings beyond fixed multi-dataset fine-tuning, such as continual learning, provided gradients remain available.
  • Our inference: because the procedure requires full access to model weights and gradients, it likely cannot be applied to closed-source models exposed only through APIs; the paper itself acknowledges this limitation.
  • Our inference: the sharpness findings suggest that ensembles of small adapters may also improve out-of-distribution robustness, but the paper does not test that and leaves it open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ENSEMBLE LORA, a method for fine-tuning a pretrained language model on multiple datasets. It partitions n datasets into m groups using a task-affinity matrix T that is estimated without fine-tuning: for random subsets S of datasets, the fine-tuned adapter weights are approximated by solving a logistic regression on base-model gradients projected to dimension d, and the fine-tuning performance f_i(S) is evaluated with the resulting approximate adapter. One adapter is then trained per group, a few gradient-boosting steps are added, and the adapters are combined with learned weights. Experiments on SuperGLUE with Llama-3.1-8B report that the estimate has 4.2-4.8% relative error at d=400 and that the method improves QLoRA's average test accuracy by up to 10 percentage points with about 9% more FLOPs. A CodeLlama-34B experiment reports a 3% improvement over QLoRA, and a 500-task federated setting is also evaluated. The paper additionally presents a sharpness analysis claiming that small-rank adapters and adapter ensembles have favorable generalization behavior.

Significance. If the linearization claim holds, the paper provides a practical and cheap recipe: one gradient pass over the training data at the base model, followed by CPU-only regression to estimate fine-tuning performance for candidate dataset groupings. The experiments are broad, covering four PEFT methods, several model families, and a federated scenario, and the code is promised publicly. The comparison against gradient-similarity clustering in Table 7 and the explicit reporting of FLOPs and memory are useful. The main caveats are that the central link from the estimated affinity matrix to the final accuracy is not directly demonstrated, and that several headline numbers (the 34B approximation error, the 105x speedup, the 1% output approximation) are stated more strongly than the reported measurements support. These are fixable with additional ablations and more careful wording, so the work is a reasonable candidate for publication after revision.

major comments (4)
  1. [Section 2.2, Algorithm 1, Table 3] The core claim that gradient-based estimates of fine-tuning performance drive the grouping is not tested. The paper never compares the clusters obtained from the estimated affinity matrix T_est with those from an oracle T computed by actual fine-tuning, and it does not ablate against random grouping. Without such an ablation, the reported gains over QLoRA could be attributable to ensembling multiple adapters and to boosting rather than to the affinity estimates. Table 7 only compares two approximate estimators, so it does not establish that either is better than chance. Please add an oracle-T baseline and a random-partition baseline, and report cluster agreement (e.g., adjusted Rand index) between T_est and T_oracle, together with the final ensemble accuracy for each.
  2. [Abstract and Section 2.2, Figure 2, Table 5] The claim that the first-order approximation 'holds with less than 1% error on models with up to 34 billion parameters' is not supported by the measurements: Figure 2's horizontal axis ends at 15 billion parameters and Table 5's largest model is Llama-2-13B. The CodeLlama-34B result in Table 6 is end-to-end and does not isolate the approximation error of h_X(s,y). Either report the approximation error at 34B or revise the claim to 'up to 13B,' and adjust the abstract and Figure 2 caption accordingly.
  3. [Section 3.2, Table 3] The '105x speedup' is the ratio of full fine-tuning FLOPs to the FLOPs of the gradient-evaluation stage only; it does not include the cost of fine-tuning the m group adapters and the boosting steps in Algorithm 2. The abstract and Section 1 describe this as 'speeding up computation compared to base fine-tuning,' which is misleading. Please state explicitly that the speedup applies to the estimation stage, and report the end-to-end cost of ENSEMBLE LORA relative to QLoRA, which from Table 4 is about 9% more FLOPs.
  4. [Section 2.2, Eq. (1), Appendix A.1] The affinity matrix T is defined asymmetrically: T_{i,j} averages f_i(S), the performance on task i, over subsets containing both i and j, while T_{j,i} averages f_j(S). However, the clustering objective v^T T v / (v^T v) is only a well-defined 'average density' for symmetric nonnegative matrices. The paper should state whether T is symmetrized before clustering and, if not, justify the use of an asymmetric matrix in the SDP objective; otherwise the clustering step is not well posed.
minor comments (6)
  1. [Section 3.2, Table 3] The text says 'Our approach achieves a relative error within 9%' for d=200 but then says increasing d to 400 reduces the error to under 5%; please make the dependence on d explicit in the first sentence.
  2. [Figure 2 and Table 5] The caption of Figure 2 says 'up to 34 billion parameters,' but the horizontal axis is labeled up to 15; align the caption, the axis, and the text.
  3. [Table 4] The header row does not clearly separate the ten dataset names; RTE, WiC, and WSC appear to be missing or merged in the printed table. Please format the table so each accuracy column maps to its dataset.
  4. [Appendix B.1 and Table 2] Model names are inconsistent between 'Llama-3-1B' and 'Llama-3.2-1B'; please standardize the naming.
  5. [Algorithm 1, line 8] The phrase 'solve a regression problem' is vague; since the derivation uses logistic loss, write 'solve a logistic regression' to match the loss defined in Section 2.2.
  6. [Section 3.3] The claim that 'test accuracy stabilizes after m reaches 3' is supported only by a sentence, with no plot or table; please include the ablation or refer to an appendix figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the gradient-based surrogate is validated against true fine-tuning and the reported ensemble gains are measured after real fine-tuning, not derived from the surrogate by construction.

full rationale

The paper's derivation chain is not circular. Algorithm 1 estimates fine-tuning performance by minimizing a first-order logistic regression on base-model gradients and evaluates the resulting surrogate on validation data; Table 3 compares these estimates against actual fine-tuning across 50 random subsets, so the approximation is tested rather than assumed. The task-affinity matrix T is an average of these estimated subset performances, and the final ensemble accuracy in Table 4 is obtained by actually fine-tuning one adapter per cluster plus boosting, so the reported gains are measured facts, not predictions forced by the construction of T. The main self-citations (Li et al. 2023, 2024a,b; Ju et al. 2022) motivate the gradient-based approach and the Hessian measure, but the paper supplies its own error measurements (Table 5, Figure 2) and compares against external baselines (QLoRA, MTL-FT, TAG), so those citations are not load-bearing. The absence of an oracle-T or random-grouping ablation weakens the attribution of the gains to the affinity estimates, and the 34B approximation claim is not directly supported by the presented error curves, but these are evidence gaps rather than circular reductions.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method rests on the linearization assumption and on several hyperparameters that are either hand-set or selected by validation/test accuracy. No new physical or architectural entities are introduced; the task affinity matrix is a derived quantity.

free parameters (8)
  • Projection dimension d = 400
    Selected from {200, 400, 800} based on minimizing estimation error in Table 3; d=400 gives under 5% relative error.
  • Number of task groups m = 3
    Chosen because 'test accuracy stabilizes after m reaches 3' (Section 3.3), i.e., selected using test accuracy.
  • Boosting steps b = 1
    One step reduces training error by 18% and adds 0.4% accuracy; b is determined by where training loss stops decreasing.
  • Number of sampled subsets k = 200 (size 3)
    Chosen by the authors for computing the affinity matrix; no sensitivity analysis is reported.
  • Clustering threshold c/n = c unspecified, threshold = c/n with c >= 1
    Used in rounding the SDP solution; the constant c is not reported.
  • Boosting learning rate eta = 0.1
    Set in Appendix A.2 for the boosting update.
  • LoRA rank = 4
    Determined by cross-validation, per Appendix B.3; used for all LoRA/QLoRA runs.
  • Learning rate and epochs = 2e-5, 10 epochs
    AdamW settings used in all experiments, chosen by hyperparameter search.
assumptions (5)
  • domain assumption Taylor expansion of the model output around base weights is accurate for fine-tuned adapters (first-order term dominates).
    Section 2.2: 'Since the approximation error is small, we next apply it to approximate the model loss.' This is the core linearity assumption.
  • domain assumption Fine-tuned adapter weights stay within 0.2% relative distance of the base model, so the expansion point is representative.
    Table 2 reports relative distances under 0.2% for Llama models up to 8B; the 34B case is not measured.
  • standard math Projected gradients preserve enough information for logistic regression (Johnson-Lindenstrauss lemma).
    Footnote 1 in Section 2.2.
  • standard math The SDP relaxation and rounding produce a good partition for the task affinity matrix.
    Appendix A.1; no guarantees on the rounding quality are stated.
  • domain assumption Validation loss on a 10% split of an already-reduced training set is a reliable proxy for fine-tuning performance.
    Section 3.1 and Appendix B.1; they use 50% of training data and 10% validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets." pith.science (2026). https://pith.science/paper/DOSG6QPN

@misc{pith2026250521930,
  author       = {Pith},
  title        = {Pith review of: Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DOSG6QPN}},
  note         = {Machine review of arXiv:2505.21930}
}
abstract

This paper develops an ensemble method for fine-tuning a language model to multiple datasets. Existing methods, such as quantized LoRA (QLoRA), are efficient when adapting to a single dataset. When training on multiple datasets of different tasks, a common setup in practice, it remains unclear how to design an efficient adaptation for fine-tuning language models. We propose to use an ensemble of multiple smaller adapters instead of a single adapter per task. We design an efficient algorithm that partitions $n$ datasets into $m$ groups, where $m$ is typically much smaller than $n$ in practice, and train one adapter for each group before taking a weighted combination to form the ensemble. The algorithm leverages a first-order approximation property of low-rank adaptation to quickly obtain the fine-tuning performances of dataset combinations since methods like LoRA stay close to the base model. Hence, we use the gradients of the base model to estimate its behavior during fine-tuning. Empirically, this approximation holds with less than $1\%$ error on models with up to $34$ billion parameters, leading to an estimation of true fine-tuning performances under $5\%$ error while speeding up computation compared to base fine-tuning by $105$ times. When applied to fine-tune Llama and GPT models on ten text classification tasks, our approach provides up to $10\%$ higher average test accuracy over QLoRA, with only $9\%$ more FLOPs. On a Llama model with $34$ billion parameters, an ensemble of QLoRA increases test accuracy by $3\%$ compared to QLoRA, with only $8\%$ more FLOPs.

Figures

Figures reproduced from arXiv: 2505.21930 by the authors.

Figure 1
Figure 1. Left: We propose an ensemble method for fine-tuning language models on multiple datasets. Given n datasets and a base adaptation method such as LoRA, we design an ensemble of adapters that applies weighted averaging to their outputs, with minimal computation and memory overhead to the base method. Middle: We partition n datasets into m groups based on the task affinities scores. Our method first estimates fine-tunin… view at source ↗
Figure 2
Figure 2. We report the approximation error for Llama [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. We compare error rate (one minus accuracy), [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustrating the empirical generalization errors and sharpness measures with respect to QLoRA weights. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: This figure compares the error rate (one average minus test accuracy), computation cost, and GPU memory [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: We show the empirical generalization er [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: We illustrate the trade-off between trans [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: We measure the average similarity between [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 36 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to pmi-based word embeddings. TACL, 4:385--399

    work page 2016

  4. [4]

    Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. 2023. Parameter-efficient fine-tuning design spaces. ICLR

    work page 2023

  5. [5]

    Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. 2024. CogBench : a large language model walks into a psychology lab. In ICML

    work page 2024

  6. [6]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA : Efficient finetuning of quantized llms. NeurIPS

    work page 2023

  7. [7]

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfeng Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence

    work page 2023

  8. [8]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR

    work page 2022

Show all 40 references
  1. [9]

    Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently identifying task groupings for multi-task learning. NeurIPS

    2021

  2. [10]

    Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. 2024. Lq- LoRA : Low-rank plus quantized matrix decomposition for efficient language model finetuning. ICLR

    2024

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. ICLR

    2021

  4. [12]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP . In ICML

    2019

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA : Low-rank adaptation of large language models. ICLR

    2021

  6. [14]

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. 2022. Datamodels: Predicting predictions from training data. ICML

    2022

  7. [15]

    Haotian Ju, Dongyue Li, and Hongyang R Zhang. 2022. Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In ICML

    2022

  8. [16]

    Giannis Karamanolakis, Daniel Hsu, and Luis Gravano. 2024. Interactive machine teaching by labeling rules and instances. TACL, 12:1441--1459

    2024

  9. [17]

    Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In ICML

    2017

  10. [18]

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshu...

    2021

  11. [19]

    Zornitsa Kozareva, Konstantin Voevodski, and Shanghua Teng. 2011. Class label enhancement via related instances. In EMNLP

    2011

  12. [20]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP

    2021

  13. [21]

    Dongyue Li, Huy L Nguyen, and Hongyang R Zhang. 2023. Identification of negative transfers in multitask learning using surrogate models. TMLR

    2023

  14. [22]

    Dongyue Li, Aneesh Sharma, and Hongyang R Zhang. 2024 a . Scalable multitask learning using gradient-based estimation of task affinity. In KDD

    2024

  15. [23]

    Dongyue Li, Ziniu Zhang, Lu Wang, and Hongyang R Zhang. 2024 b . Scalable fine-tuning from multiple data sources: A first-order approximation approach. Findings of EMNLP

    2024

  16. [24]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL

    2021

  17. [25]

    Chen Cecilia Liu, Jonas Pfeiffer, Ivan Vuli \'c , and Iryna Gurevych. 2023. Improving generalization of adapter-based cross-lingual transfer with scheduled unfreezing. NAACL

    2023

  18. [26]

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In ACL

    2019

  19. [27]

    Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1--40

    2023

  20. [28]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...

    2022

  21. [29]

    Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. 2018. A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. ICLR

    2018

  22. [30]

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 2023. TRAK : Attributing model behavior at scale. ICML

    2023

  23. [31]

    Jonas Pfeiffer, Aishwarya Kamath, Andreas R \"u ckl \'e , Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterfusion: Non-destructive task composition for transfer learning. EACL

    2020

  24. [32]

    Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. 2024. MELoRA : Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. ACL

    2024

  25. [33]

    Shagun Sodhani, Amy Zhang, and Joelle Pineau. 2021. Multi-task reinforcement learning with context-based representations. In ICML

    2021

  26. [34]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019 a . Super GLUE : A stickier benchmark for general-purpose language understanding systems. NeurIPS

    2019

  27. [35]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 b . GLUE : A multi-task benchmark and analysis platform for natural language understanding. In ICLR

    2019

  28. [36]

    Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. EMNLP

    2022

  29. [37]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022

  30. [38]

    Sen Wu, Hongyang R Zhang, and Christopher R \'e . 2020. Understanding and improving information transfer in multi-task learning. In ICLR

    2020

  31. [39]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS : Selecting influential data for targeted instruction tuning. In ICML

    2024

  32. [40]

    Fan Yang, Hongyang R Zhang, Sen Wu, Christopher R \'e , and Weijie J Su. 2020. Precise high-dimensional asymptotics for quantifying heterogeneous transfers. arXiv preprint arXiv:2010.11750

    2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.