Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

Mei-Yen Chen; Michael Hahn; M. Saquib Sarfraz; Thi Thu Uyen Hoang

REVIEW 3 major objections 5 minor 52 references

Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Recycling LoRAs fails at composing separate knowledge

desk verdict Worth reading and worth a careful referee, but the formal anchor is shaky; the empirical bridge results are the real contribution. read the letter →

arxiv 2506.13479 v1 pith:ESHDQDVJ submitted 2025-06-16 cs.CL cs.AI

Mei-Yen Chen , Thi Thu Uyen Hoang , Michael Hahn , M. Saquib Sarfraz This is my paper · ORCID

classification cs.CLcs.AI

keywords LoRAmergingmodelcompositionalgeneralizationtwo-hopreasoninglow-rankadaptersparameter-efficientfine-tuningGSM-Symbolicadapterrouting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the machine-learning community is chasing the wrong problem in LoRA recycling: merging or routing low-rank adapters (LoRAs), the small plug-in weight updates used to specialize large language models, does not reliably compose knowledge from separately fine-tuned models. The paper's central claim is that combining LoRAs works for a new task only when that task is already represented in the fine-tuning data, and that reported successes often reflect shallow pattern matching, such as entity familiarity or shared templates, rather than genuine compositional generalization. A simplified theoretical model of factual recall shows why: a LoRA trained on a one-hop fact edits the model's output only on a narrow activation subspace, so when two such adapters are summed on a two-hop prompt they return a linear mixture of the two edited entities, not their composition. Synthetic two-hop and math word-problem experiments on models up to 70B parameters support this, with two-adapter libraries failing on held-out compositions and math-specialized models even losing accuracy after routing. If the paper is right, data-free LoRA reuse is not a dependable route to new capabilities, and research should shift from inventing merger algorithms to mapping when reuse can work.

What carries the argument

The load-bearing object is the rank-one LoRA update characterized in Proposition 1. For a single fact edit, the minimal-L2 update takes the form $\Delta W \propto (i_{\tilde r(x)} - i_{r(x)}) \mathrm{ReLU}(U(V e_X + e_{\mathrm{REL}}))^T$: it attaches the new output vector to the activation subspace of the one-hop prompt. Combining two such adapters is then analyzed through the random-features kernel $k(\eta,\xi) \approx \frac{1}{m}\mathrm{ReLU}(U\eta)^T \mathrm{ReLU}(U\xi)$ of a wide ReLU MLP. On the two-hop prompt the activation $\xi = U(\frac12 V e_X + \frac12 V e_{\mathrm{REL}_1} + e_{\mathrm{REL}_2})$ has positive kernel overlap with both one-hop activation subspaces $\eta_1,\eta_2$, so the summed adapters contribute both edited entities with comparable weights instead of computing the composition. This mechanism, subspace overlap producing a mixture, is what rules out compositional integration for summing, weighted averaging, and similarity-based routing alike.

What would settle it

Record the actual attention-weighted value vectors of a real transformer on the prompt $X\ \mathrm{REL}_1\ \mathrm{REL}_2$ and check whether the induced MLP input matches the assumed $\frac12 V e_X + \frac12 V e_{\mathrm{REL}_1} + e_{\mathrm{REL}_2}$; large deviations that flip the kernel overlaps would break the theorem. Behaviorally, any data-free combination of two one-hop LoRAs that reliably solves a held-out two-hop task with no CoT bridge and no target-task expert in the library would falsify the paper's central claim.

Watch

Extended reading notes

Core claim

The paper's formal anchor is Theorem 2: if two rank-one LoRAs are each trained to edit one fact of the relations r1 and r2, then summing the adapters does not produce correct answers to the composed two-hop query. The proof shows that the minimal-L2 update for a single fact is a rank-one term $\Delta$ W = p q^T aligned with the prompt's ReLU activation; on a two-hop prompt the activation overlaps the subspaces of both one-hop adapters, so the combined model outputs a weighted linear combination of the edited entities rather than r2(r1(x)). Empirically, a library containing only the two one-hop adapters stays below 10 percent accuracy on two-hop tests across base models and entity types, while adding an expert trained on the target relation or CoT-formatted bridge examples recovers performance. In the GSM-Symbolic easy-to-hard math setting, routing LoRAs trained on simpler variants generally degraded accuracy on the harder variant, especially for math-specialized models. The paper concludes that reuse is effective only when target tasks are already represented, and advocates a research pause on new merging and routing algorithms.

Load-bearing premise

The negative theorem assumes a specific attention pattern on the two-hop prompt, namely that the MLP input is $U(\frac12 V e_X + \frac12 V e_{\mathrm{REL}_1} + e_{\mathrm{REL}_2})$, with coefficients one half, one half, and one; if real transformer attention on such prompts deviates from these coefficients, the proof's categorical conclusion about actual LLMs does not necessarily follow.

Editorial extensions

If this is right

Data-free LoRA merging and routing should not be treated as a mechanism for composing skills: with only the two one-hop adapters, two-hop accuracy stays below 10 percent across base models.
Reliable adapter reuse for a new task requires training on the target reasoning pattern; CoT-formatted bridge examples in both adapters recover high accuracy, while omitting them collapses performance.
Entity and domain familiarity is a precondition for even partial success: fake names and fake locations consistently fail, whereas real names and real locations succeed.
Routing can actively degrade a model: on GSM-P2, routing LoRAs trained on natural-language solutions dropped Qwen2.5-Math-7B-Instruct from 68 percent to 34 percent (Uniform) or 27 percent (Arrow), partly by suppressing its tool-integrated-reasoning behavior.
Weighted averaging and similarity-based routing inherit the same mixture failure as plain summing, so improving routing heuristics alone is unlikely to unlock composition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper: if the subspace-overlap mechanism is right, LoRA composition should be surface-form sensitive—adapters trained on prompts whose wording matches the two-hop prefix (e.g., ‘spouse of X lives in’) should compose better than those trained on paraphrases; the paper's template-split data could test this directly.
Going beyond the paper: the theorem is a one-layer, single-head simplification, so for real multi-layer LLMs the empirical results carry the argument; a natural extension is to test whether chain-of-thought-distilled base models can overcome the mixture failure without bridge training.
Going beyond the paper: if the position holds, LoRA hubs are better understood as task-specific assets than plug-and-play capability libraries; practical value would hinge on cataloguing what each adapter encodes and what the base model already knows, not on the routing algorithm.
Going beyond the paper: the math experiments suggest a testable asymmetry—LoRAs fine-tuned on code or Tool-Integrated-Reasoning solutions may preserve a math-specialized model's tool use, while natural-language CoT LoRAs suppress it; matching solution format between fine-tuning and base-model behavior would isolate whether the failure is compositional or a format mismatch.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper's empirical message lands, its theoretical anchor doesn't. The claim is that data-free LoRA merging and routing won't give compositional generalization unless the target task is represented in the fine-tuning data. The bridge ablation is the most useful part: training both LoRAs on CoT two-hop examples from disjoint entities makes Arrow routing jump from single digits to 85-95%, while dropping the CoT template or the one-hop tasks collapses it. That is a clean demonstration that the apparent 'composition' is really template and pattern matching. It holds across six base models and several familiarity conditions, which I find convincing.

What is new: Proposition 1 gives a closed-form rank-one LoRA update under L2 regularization, and Theorem 2 tries to show that summing adapters yields a linear mixture rather than a functional composition. The proof does not hold as written. The activation U(V/2 e_X + V/2 e_REL1 + e_REL2) is asserted, not derived from the uniform attention model stated in Section 3.1, which would give equal thirds or some other residual-stream convention. More importantly, the proof compares the adapter contribution to i_z - i_{r2(y)} rather than to the full base output W xi. The base model's actual two-hop output is i_{r2(r1(x))}, and r1(x) is not y after the edit, so the comparison target is wrong. The theorem might still be true in the toy model, but this derivation does not establish it. That is load-bearing because the abstract leans on the theorem for formal support.

Empirically, the test sets look small (100 triplets in the two-hop tasks, 100 GSM-P2 questions) and there are no error bars, and no code or data are released. Consistency across models helps, but a few random seeds would have made the main numbers much easier to trust. The broader position is broader than the evidence: two data-agnostic methods failing on synthetic two-hop tasks does not mean the entire LoRA-recycling research direction should pause. But the mechanism ablations are exactly the kind of evidence the field needs, and the direction is defensible.

Recommendation: send to peer review. A serious referee can get the theory repaired, the error bars added, and the claims scaled to the evidence. I would cite the bridge ablation even if the theorem gets weakened. This paper deserves a serious referee.

Referee Report

3 major / 5 minor

Summary. This position paper argues that the machine learning community should stop proposing new LoRA merging/routing algorithms and instead study the conditions under which reusing LoRAs actually enables compositional generalization. The authors support this position with (i) a theoretical analysis of a simplified one-layer transformer with random features and LoRA updates applied to the MLP output layer, culminating in Theorem 2, which claims that summing two LoRAs that each edit a single fact does not yield correct two-hop compositional answers, and (ii) experiments on synthetic two-hop reasoning and GSM-Symbolic math word problems, evaluating Uniform averaging and Arrow routing across Qwen and DeepSeek models from 3B to 70B parameters. The central conclusion is that combining LoRAs is ineffective for new tasks unless the target task is already represented in the fine-tuning datasets, and that entity/domain familiarity and close prompt-form matches are the key enablers of the limited success observed.

Significance. If the central claim is established, the paper provides a useful cautionary result for the growing LoRA-merging literature and a concrete research agenda focused on mechanisms rather than algorithms. The empirical study is broad: it covers multiple model families and scales, uses controlled synthetic data to avoid pretraining contamination, and includes careful ablations of bridge setups, CoT formatting, and fine-tuning layer placement. The theoretical analysis is a genuine attempt to anchor the position in a formal statement rather than only in anecdotal evidence. However, as detailed below, the formal anchor is not currently valid as written, and the empirical design does not actually test compositional generalization to unseen entity/relation combinations because the same triplets appear in train and test. These issues are load-bearing for the paper's headline claims, but they are fixable within the scope of a revision.

major comments (3)

[Appendix A.1, Eq. (13)] The proof of Theorem 2 assumes that the two-hop prompt X REL1 REL2 produces the MLP activation U(V/2 e_X + V/2 e_REL1 + e_REL2), where the coefficients 1/2 and 1 encode a specific attention pattern. This pattern is not derived from the model definition in Section 3.1: under the stated uniform softmax attention over the three context tokens, the value mixture would be (V e_X + V e_REL1 + V e_REL2)/3, while a residual-stream-plus-uniform-attention convention would give yet another pattern. The kernel overlap computations k(eta1, xi) and k(eta2, xi) in the proof depend on these coefficients, so changing the attention convention can change the magnitudes, signs, or even the identities of the contributed output terms. The theorem is therefore not proven for the model as defined; the activation formula must either be derived from an explicit attention mechanism or stated as an additional assumption whose influence on the conclusion is analyzed.
[Appendix A.1, proof of Theorem 2] The proof evaluates only the adapter contribution (DeltaW1 + DeltaW2) * ReLU(U xi) and compares it with the target i_z - i_{r2(y)}, but it never writes the full adapted output (W + DeltaW1 + DeltaW2) * ReLU(U xi). The omitted base-output term W xi is the base model's two-hop answer i_{r2(r1(x))}, and because y denotes the edited value ~r1(x), which generally differs from the original r1(x), the quantity i_z - i_{r2(y)} is not the correct error target. To establish Theorem 2, the proof must include the base-output term and show that no nonnegative or fitted combination of the adapter contributions yields the exact one-hot vector i_z for the composed relation; the current algebra only shows that the adapter contribution is a linear mixture of two one-hop update directions, which is a necessary but not sufficient observation.
[Section 4.1 and Appendix A.2.1] The two-hop experiments use a template-based split in which the same 100 triplets (A,B,C) are present in training, development, and test, and only the 50 paraphrase templates are split (46 training, 2 development, 2 test). Consequently, the test evaluation measures robustness to paraphrase variation, not compositional generalization to unseen entity combinations or unseen relation compositions. Statements in the abstract and introduction that reusing LoRAs fails on 'unseen tasks' or 'novel tasks' are therefore not supported by these experiments. The same concern applies to the GSM-Symbolic setup, where P2 questions are generated from the same seed templates used for training P1 and the base GSM-Symbolic set. The authors should either construct test sets with disjoint triplets and unseen relation pairs, or substantially weaken the claims about compositional generalization to what the data actually support.

minor comments (5)

[Appendix A.1, Eq. (15)] There is a typo in the displayed equation: 'iy−ir1(y)' should read 'iy−ir1(x)'.
[Section 3.1 and Proposition 1 proof] The LoRA regularizer is stated as ||A||_F + ||B||_F in Section 3.1, but the proof of Proposition 1 minimizes ||p||_2^2 + ||q||_2^2; please clarify whether the squared norm is intended and whether the proof's minimizer is consistent with the stated regularizer.
[Section 4.1.1] The sentence 'in the presence of of unfamiliar entities' contains a duplicated 'of'.
[Appendix A.2.1] The text says the bridge uses examples 'disjoint from those needed in testing', but the main experimental description says triplets remain constant across splits; please disambiguate whether the bridge entities/triplets are truly disjoint from the test triplets or only from the target templates.
[Section 5] The discussion of alternative methods (LoRI, LoRA Lego, Self-MoE, etc.) is brief and does not provide concrete empirical comparisons; since the paper's position is about limits, citing absence of evidence is acceptable, but a short summary of what those methods do and where they were evaluated would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the theory is a conditional toy-model argument and the empirical claims rest on held-out evaluations; the main weakness is an assumed two-hop activation pattern, which is a correctness/limitation issue rather than circularity.

full rationale

The paper's load-bearing claim is that combining LoRAs fails to compose knowledge unless the target pattern is present in the fine-tuning data. The theoretical part (Proposition 1 and Theorem 2, Appendix A.1) is derived from explicit assumptions: a single-head, uniform-attention, random-features transformer with untrained U, V, E and an L2-minimal rank-one LoRA update. Proposition 1 solves the stated regularized interpolation problem in closed form, and Theorem 2 computes kernel overlaps under a stated two-hop activation ansatz. That ansatz, with coefficients 1/2, 1/2, and 1 in Equation (13), is an assumption rather than a derivation from Section 3.1; the residual comparison also omits the base output W xi, so the theorem is not a fully rigorous anchor for the categorical conclusion. However, this is a limitation or correctness concern, not circularity: the theorem's conclusion is not built into the definition of the adapters or the optimization problem, and no fitted parameter is renamed as a prediction. The empirical sections use held-out templates for the two-hop task and held-out GSM-P2 questions for the math task, and the bridge experiments manipulate the training data rather than encoding the test outcome into the theory. The paper explicitly acknowledges that the theory applies to a single-layer transformer and that experiments are needed to test applicability to LLMs across scales. No load-bearing step is justified only by a self-citation, and the authors do not import a uniqueness theorem from their own prior work. There is therefore no circular step to exhibit, and the appropriate score is low; I set it to 1 only to acknowledge that the theoretical anchor is weaker than the paper's rhetoric suggests, not because of circularity.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theory rests on a heavily simplified transformer model and an ad hoc two-hop activation formula; there are no fitted free parameters. Detailed entries are listed below.

assumptions (6)

domain assumption A single-layer transformer with one attention head, random untrained embeddings U, V, E, uniform attention, and an MLP readout W is treated as a representative model for the theory.
The theory in Section 3.1 uses this heavily simplified model, which the authors describe as the simplest possible setup.
domain assumption The softmax attention over context tokens is exactly uniform.
Section 3.1 states 'we will take it to be exactly uniform for simplicity'; the proof's activation formula is built on this.
ad hoc to paper The two-hop prompt activation in the proof is U(V e_X/2 + V e_REL1/2 + e_REL2).
Equation (13) in Appendix A.1 specifies coefficients 1/2 and 1 that do not follow from uniform attention over three tokens; the paper does not derive them.
domain assumption The base model already performs one-hop and two-hop relations correctly before adaptation, so LoRA only edits individual facts.
Section 3.1: 'the base model already performs correctly for the given set of relations R, and W is then adapted to reflect edits.'
standard math The random-features kernel approximation applies as the MLP width m goes to infinity.
This is a standard result (Le Roux and Bengio 2007, Bach 2024), used to compute kernel overlaps in the proof of Theorem 2.
domain assumption The LoRA update is the L2-regularized minimal-norm update for the edited fact.
Proposition 1 minimizes ||A||_F^2 + ||B||_F^2; actual trained LoRAs need not reach this exact solution.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness." pith.science (2026). https://pith.science/paper/ESHDQDVJ

@misc{pith2026250613479,
  author       = {Pith},
  title        = {Pith review of: Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ESHDQDVJ}},
  note         = {Machine review of arXiv:2506.13479}
}

read the original abstract

Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.

Figures

Figures reproduced from arXiv: 2506.13479 by the authors.

**Figure 2.** (left) In the bridge setup, both LoRA experts are trained not only on one of the two hops, but [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Our methodology for LoRA routing for two-hop experiments. In standard routing, each [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Visualization of Shared and Disjoint Graphs. [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

**Figure 5.** Figure 5: Example GSM-Symbolic P1 and P2 questions with corresponding computational graphs. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

**Figure 6.** Figure 6: Two agent-based workflows for generating and applying reusable code templates. The [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: The Chain-of-Thoughts and Tool-Integrated Reasoning (TIR) prompts used for evaluation. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 39 canonical work pages

[1]

Communication-Efficient Learning of Deep Networks from Decentralized Data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data . In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 127...

2017
[2]

Building machine learning models like open source software

Colin Raffel. Building machine learning models like open source software. Commun. ACM, 66 0 (2): 0 38–40, January 2023. ISSN 0001-0782. doi:10.1145/3545111

work page doi:10.1145/3545111 2023
[3]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021

arXiv 2021
[4]

A dapter H ub playground: Simple and flexible few-shot learning with adapters

Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. A dapter H ub playground: Simple and flexible few-shot learning with adapters. In Valerio Basile, Zornitsa Kozareva, and Sanja Stajner, editors, Proceedings of the 60th Annual Meeting of the Association for Comput...

doi:10.18653/v1 2022
[5]

Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024

Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024

work page 2024
[6]

Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering, 2024 a

Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering, 2024 a

work page 2024
[7]

Towards modular LLM s by building and reusing a library of L o RA s

Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular LLM s by building and reusing a library of L o RA s. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International ...

work page 2024
[8]

Lora soups: Merging loras for practical skill composition tasks, 2024

Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks, 2024

work page 2024

Show all 52 references

[9]

L ora R etriever: Input-aware L o RA retrieval and composition for mixed tasks in the wild

Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. L ora R etriever: Input-aware L o RA retrieval and composition for mixed tasks in the wild. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Com...

2024 doi
[10]

A survey on model moerging: Recycling and routing among specialized experts for collaborative learning

Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. Transactions on Machine Learnin...

2025
[11]

Lori: Reducing cross-task interference in multi-task low-rank adaptation, 2025

Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation, 2025

2025
[12]

Realistic evaluation of model merging for compositional generalization, 2024

Derek Tam, Yash Kant, Brian Lester, Igor Gilitschenski, and Colin Raffel. Realistic evaluation of model merging for compositional generalization, 2024

2024
[13]

Lima: Less is more for alignment

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hard...

2023
[14]

Does fine-tuning llms on new knowledge encourage hallucinations?, 2024

Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations?, 2024

2024
[15]

Understanding finetuning for factual knowledge extraction

Gaurav Rohit Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the...

2024
[16]

The expressive power of low-rank adaptation, 2024

Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation, 2024

2024
[17]

On the impact of fine-tuning on chain-of-thought reasoning, 2025

Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. On the impact of fine-tuning on chain-of-thought reasoning, 2025

2025
[18]

What is being transferred in transfer learning? In H

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 512--523. Curran Associates, Inc., 2020

2020
[19]

Editing models with task arithmetic, 2023

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023

2023
[20]

Raffel, and Mohit Bansal

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023

2023
[21]

Language models are super mario: Absorbing abilities from homologous models as a free lunch

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, ed...

2024
[22]

Peters, Alexander Fraser, and Jesse Dodge

Alexandra Chronopoulou, Matthew E. Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models, 2023

2023
[23]

Exploring the benefits of training expert language models over instruction tuning

Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, IC...

2023
[24]

Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. T...

2022 doi arXiv
[25]

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199--22213, 2022

2022
[26]

Smith, Sarah Wiegreffe, and Yanai Elazar

Jack Merullo, Noah A. Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models. In The Thirteenth International Conference on Learning Representations, 2025

2025
[27]

Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024

2024
[28]

Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers, 2024

Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers, 2024

2024
[29]

Measuring and narrowing the compositionality gap in language models

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pag...

2023 doi
[30]

The two-hop curse: Llms trained on a b, b c fail to learn a c, 2025

Mikita Balesni, Tomek Korbak, and Owain Evans. The two-hop curse: Llms trained on a b, b c fail to learn a c, 2025

2025
[31]

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210--10229, 2024 a

2024
[32]

Lee, and Alberto Bietti

Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. In The Thirteenth International Conference on Learning Representations, 2025

2025
[33]

Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning

Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, 21, 2008

2008
[34]

Randomly initialized one-layer neural networks make data linearly separable

Promit Ghosal, Srinath Mahankali, and Yihang Sun. Randomly initialized one-layer neural networks make data linearly separable. arXiv preprint arXiv:2205.11716, 2022

2022 arXiv
[35]

The separation capacity of random neural networks

Sjoerd Dirksen, Martin Genzel, Laurent Jacques, and Alexander Stollenwerk. The separation capacity of random neural networks. Journal of Machine Learning Research, 23 0 (309): 0 1--47, 2022

2022
[36]

Transformer feed-forward layers are key-value memories

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495, 2021

2021
[37]

Locating and editing factual associations in gpt

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35: 0 17359--17372, 2022

2022
[38]

Qwen2.5 technical report, 2025

Qwen-team. Qwen2.5 technical report, 2025

2025
[39]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

DeepSeek-AI-team. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

2025
[40]

Dynamic cheatsheet: Test-time learning with adaptive memory, 2025

Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory, 2025

2025
[41]

Achieving >97

Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, and Bo Du. Achieving >97
[42]

Autogen: Enabling next-gen LLM applications via multi-agent conversations

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Confe...

2024
[43]

Self-moe: Towards compositional large language models with self-specialized experts

Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406.12034, 2024

2024 arXiv
[44]

Improving lo RA in privacy-preserving federated learning

Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lo RA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations, 2024

2024
[45]

PAL : Program-aided language models

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL : Program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the ...

2023
[46]

Continuous neural networks

Nicolas Le Roux and Yoshua Bengio. Continuous neural networks. In Artificial Intelligence and Statistics, pages 404--411. PMLR, 2007

2007
[47]

Learning theory from first principles

Francis Bach. Learning theory from first principles. MIT press, 2024

2024
[48]

Multi-head adapter routing for cross-task generalization, 2023

Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization, 2023

2023
[49]

Language and task arithmetic with parameter-efficient layers for zero-shot summarization, 2024

Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. Language and task arithmetic with parameter-efficient layers for zero-shot summarization, 2024

2024
[50]

Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024 b

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

2024
[51]

Training verifiers to solve math word problems, 2021

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

2021
[52]

Open llm leaderboard, 2023

Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard, 2023

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Communication-Efficient Learning of Deep Networks from Decentralized Data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data . In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 127...

2017

[2] [2]

Building machine learning models like open source software

Colin Raffel. Building machine learning models like open source software. Commun. ACM, 66 0 (2): 0 38–40, January 2023. ISSN 0001-0782. doi:10.1145/3545111

work page doi:10.1145/3545111 2023

[3] [3]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021

arXiv 2021

[4] [4]

A dapter H ub playground: Simple and flexible few-shot learning with adapters

Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. A dapter H ub playground: Simple and flexible few-shot learning with adapters. In Valerio Basile, Zornitsa Kozareva, and Sanja Stajner, editors, Proceedings of the 60th Annual Meeting of the Association for Comput...

doi:10.18653/v1 2022

[5] [5]

Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024

Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024

work page 2024

[6] [6]

Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering, 2024 a

Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering, 2024 a

work page 2024

[7] [7]

Towards modular LLM s by building and reusing a library of L o RA s

Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular LLM s by building and reusing a library of L o RA s. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International ...

work page 2024

[8] [8]

Lora soups: Merging loras for practical skill composition tasks, 2024

Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks, 2024

work page 2024