{"id":"0b63dea1-0cd6-4cfd-98d3-2884ca183694","arxiv_id":"2506.13479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Combining LoRAs generally fails to compose knowledge across disjoint tasks; reliable gains appear only when the target reasoning pattern or familiar entities are present in fine-tuning data.","lead":"This position paper argues that merging or routing LoRA adapters rarely enables genuine compositional reasoning unless the target task is already represented in the fine-tuning data. It combines a new theoretical argument with controlled synthetic reasoning and math experiments, challenging the popular promise of data-free adapter reuse.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's proof rests on an assumed two-hop activation pattern and an omitted base-output term, so the formal anchor for the position is not established by the theory alone.","rationale":"The paper's empirical core is substantial: across six base models, the two-combination library (A->B and B->C experts) scores below 10% on held-out templates, while adding a bridge containing CoT two-hop examples raises accuracy to 70-95% in most settings, and GSM-P2 routing degrades specialized math models. These results support the conditional reading of the position: data-free LoRA merging/routing does not reliably compose disjoint knowledge, and target-task representation in fine-tuning data is an important precondition. The theoretical analysis is intended to explain this, but its proof of Theorem 2 is not rigorous enough to serve as an independent anchor. The two-hop activation coefficients are assumed rather than derived, and the proof omits the base model's two-hop output when stating the required change. Both issues are fixable: the theorem likely survives for any positive activation overlap because the first adapter injects a spurious one-hot component, but the paper should state the attention/residual convention and write the full forward pass. Because the empirical evidence is consistent and the position is appropriately scoped as a call for mechanistic research, the conditional verdict stands. No ad hominem intended; this is a technical gap in the formal support.","tokens_in":22951,"tokens_out":19184,"duration_ms":193127,"concrete_test":"Implement the Section 3.1 one-layer model exactly, with explicit softmax attention and residual connection, under three attention conventions: (i) uniform attention over previous tokens only plus residual, (ii) uniform attention over all three tokens including self, and (iii) softmax attention with random K,Q. For each convention, use random Gaussian U,V and embeddings, train a base W to store the one-hop and two-hop facts, train the two rank-one LoRAs via the Proposition 1 L2 objective, and check whether (W + DeltaW1 + DeltaW2) xi equals the correct one-hot entity on X REL1 REL2. If the correct output occurs for any convention, Theorem 2 is false as stated; if it fails for all, the proof gap is cosmetic and the reader's coefficient objection does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that combining LoRAs is ineffective for new tasks unless the target task is represented in fine-tuning data, with Theorem 2 as the formal anchor. The proof in Appendix A.1 evaluates the summed adapters on X REL1 REL2 using activation U(V/2 e_X + V/2 e_REL1 + e_REL2). This pattern is not derived from the model definition in Section 3.1: with uniform softmax attention over the three context tokens the value mixture would be (V e_X + V e_REL1 + V e_REL2)/3, while a residual stream plus uniform attention over previous tokens gives e_REL2 + (V e_X + V e_REL1)/2. The text does not specify which convention is intended, and the kernel overlap computations depend on the chosen coefficients. More importantly, the proof never writes the full adapted output (W + DeltaW1 + DeltaW2) xi; it compares the adapter contribution to i_z - i_{r2(y)} instead of i_z - W xi. W xi is the base model's actual two-hop output, i_{r2(r1(x))} under the paper's assumption that the base model can answer two-hop prompts, and r1(x) generally differs from the edited y. The spurious i_y term from DeltaW1 likely prevents exact one-hot correctness for any positive overlap, so the theorem may still be true in the toy model, but the argument as written is not a valid derivation. Thus the theory alone does not anchor the categorical claim; the empirical bridge results carry it, which is why the position is conditional rather than proven.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that the machine learning community should stop proposing new LoRA merging/routing algorithms and instead study the conditions under which reusing LoRAs actually enables compositional generalization. The authors support this position with (i) a theoretical analysis of a simplified one-layer transformer with random features and LoRA updates applied to the MLP output layer, culminating in Theorem 2, which claims that summing two LoRAs that each edit a single fact does not yield correct two-hop compositional answers, and (ii) experiments on synthetic two-hop reasoning and GSM-Symbolic math word problems, evaluating Uniform averaging and Arrow routing across Qwen and DeepSeek models from 3B to 70B parameters. The central conclusion is that combining LoRAs is ineffective for new tasks unless the target task is already represented in the fine-tuning datasets, and that entity/domain familiarity and close prompt-form matches are the key enablers of the limited success observed.","tokens_in":23247,"tokens_out":3663,"duration_ms":39277,"significance":"If the central claim is established, the paper provides a useful cautionary result for the growing LoRA-merging literature and a concrete research agenda focused on mechanisms rather than algorithms. The empirical study is broad: it covers multiple model families and scales, uses controlled synthetic data to avoid pretraining contamination, and includes careful ablations of bridge setups, CoT formatting, and fine-tuning layer placement. The theoretical analysis is a genuine attempt to anchor the position in a formal statement rather than only in anecdotal evidence. However, as detailed below, the formal anchor is not currently valid as written, and the empirical design does not actually test compositional generalization to unseen entity/relation combinations because the same triplets appear in train and test. These issues are load-bearing for the paper's headline claims, but they are fixable within the scope of a revision.","major_comments":[{"comment":"The proof of Theorem 2 assumes that the two-hop prompt X REL1 REL2 produces the MLP activation U(V/2 e_X + V/2 e_REL1 + e_REL2), where the coefficients 1/2 and 1 encode a specific attention pattern. This pattern is not derived from the model definition in Section 3.1: under the stated uniform softmax attention over the three context tokens, the value mixture would be (V e_X + V e_REL1 + V e_REL2)/3, while a residual-stream-plus-uniform-attention convention would give yet another pattern. The kernel overlap computations k(eta1, xi) and k(eta2, xi) in the proof depend on these coefficients, so changing the attention convention can change the magnitudes, signs, or even the identities of the contributed output terms. The theorem is therefore not proven for the model as defined; the activation formula must either be derived from an explicit attention mechanism or stated as an additional assumption whose influence on the conclusion is analyzed.","section":"Appendix A.1, Eq. (13)"},{"comment":"The proof evaluates only the adapter contribution (DeltaW1 + DeltaW2) * ReLU(U xi) and compares it with the target i_z - i_{r2(y)}, but it never writes the full adapted output (W + DeltaW1 + DeltaW2) * ReLU(U xi). The omitted base-output term W xi is the base model's two-hop answer i_{r2(r1(x))}, and because y denotes the edited value ~r1(x), which generally differs from the original r1(x), the quantity i_z - i_{r2(y)} is not the correct error target. To establish Theorem 2, the proof must include the base-output term and show that no nonnegative or fitted combination of the adapter contributions yields the exact one-hot vector i_z for the composed relation; the current algebra only shows that the adapter contribution is a linear mixture of two one-hop update directions, which is a necessary but not sufficient observation.","section":"Appendix A.1, proof of Theorem 2"},{"comment":"The two-hop experiments use a template-based split in which the same 100 triplets (A,B,C) are present in training, development, and test, and only the 50 paraphrase templates are split (46 training, 2 development, 2 test). Consequently, the test evaluation measures robustness to paraphrase variation, not compositional generalization to unseen entity combinations or unseen relation compositions. Statements in the abstract and introduction that reusing LoRAs fails on 'unseen tasks' or 'novel tasks' are therefore not supported by these experiments. The same concern applies to the GSM-Symbolic setup, where P2 questions are generated from the same seed templates used for training P1 and the base GSM-Symbolic set. The authors should either construct test sets with disjoint triplets and unseen relation pairs, or substantially weaken the claims about compositional generalization to what the data actually support.","section":"Section 4.1 and Appendix A.2.1"}],"minor_comments":[{"comment":"There is a typo in the displayed equation: 'iy−ir1(y)' should read 'iy−ir1(x)'.","section":"Appendix A.1, Eq. (15)"},{"comment":"The LoRA regularizer is stated as ||A||_F + ||B||_F in Section 3.1, but the proof of Proposition 1 minimizes ||p||_2^2 + ||q||_2^2; please clarify whether the squared norm is intended and whether the proof's minimizer is consistent with the stated regularizer.","section":"Section 3.1 and Proposition 1 proof"},{"comment":"The sentence 'in the presence of of unfamiliar entities' contains a duplicated 'of'.","section":"Section 4.1.1"},{"comment":"The text says the bridge uses examples 'disjoint from those needed in testing', but the main experimental description says triplets remain constant across splits; please disambiguate whether the bridge entities/triplets are truly disjoint from the test triplets or only from the target templates.","section":"Appendix A.2.1"},{"comment":"The discussion of alternative methods (LoRI, LoRA Lego, Self-MoE, etc.) is brief and does not provide concrete empirical comparisons; since the paper's position is about limits, citing absence of evidence is acceptable, but a short summary of what those methods do and where they were evaluated would improve readability.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position paper, so the bar for 'proof' is not the same as for a theory paper; nevertheless, Theorem 2 is presented as the formal anchor, and the proof has a genuine gap in the activation formula and in the treatment of the base output. The empirical generalization claims are also overstated because of the template-split issue. These are fixable in revision, either by repairing the theorem under an explicit assumption or by repositioning the theory as a suggestive toy model and relying on corrected experiments. If the authors can add a disjoint-triplet two-hop test and fix the proof, the paper would be a valuable contribution to the LoRA-merging discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's empirical message lands, its theoretical anchor doesn't. The claim is that data-free LoRA merging and routing won't give compositional generalization unless the target task is represented in the fine-tuning data. The bridge ablation is the most useful part: training both LoRAs on CoT two-hop examples from disjoint entities makes Arrow routing jump from single digits to 85-95%, while dropping the CoT template or the one-hop tasks collapses it. That is a clean demonstration that the apparent 'composition' is really template and pattern matching. It holds across six base models and several familiarity conditions, which I find convincing.\n\nWhat is new: Proposition 1 gives a closed-form rank-one LoRA update under L2 regularization, and Theorem 2 tries to show that summing adapters yields a linear mixture rather than a functional composition. The proof does not hold as written. The activation U(V/2 e_X + V/2 e_REL1 + e_REL2) is asserted, not derived from the uniform attention model stated in Section 3.1, which would give equal thirds or some other residual-stream convention. More importantly, the proof compares the adapter contribution to i_z - i_{r2(y)} rather than to the full base output W xi. The base model's actual two-hop output is i_{r2(r1(x))}, and r1(x) is not y after the edit, so the comparison target is wrong. The theorem might still be true in the toy model, but this derivation does not establish it. That is load-bearing because the abstract leans on the theorem for formal support.\n\nEmpirically, the test sets look small (100 triplets in the two-hop tasks, 100 GSM-P2 questions) and there are no error bars, and no code or data are released. Consistency across models helps, but a few random seeds would have made the main numbers much easier to trust. The broader position is broader than the evidence: two data-agnostic methods failing on synthetic two-hop tasks does not mean the entire LoRA-recycling research direction should pause. But the mechanism ablations are exactly the kind of evidence the field needs, and the direction is defensible.\n\nRecommendation: send to peer review. A serious referee can get the theory repaired, the error bars added, and the claims scaled to the evidence. I would cite the bridge ablation even if the theorem gets weakened. This paper deserves a serious referee.","headline":"Worth reading and worth a careful referee, but the formal anchor is shaky; the empirical bridge results are the real contribution.","tokens_in":23761,"tokens_out":2941,"would_cite":true,"duration_ms":31537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recycling LoRAs fails at composing separate knowledge","keywords":["LoRA merging","model merging","compositional generalization","two-hop reasoning","low-rank adapters","parameter-efficient fine-tuning","GSM-Symbolic","adapter routing"],"falsifier":"Record the actual attention-weighted value vectors of a real transformer on the prompt $X\\ \\mathrm{REL}_1\\ \\mathrm{REL}_2$ and check whether the induced MLP input matches the assumed $\\frac12 V e_X + \\frac12 V e_{\\mathrm{REL}_1} + e_{\\mathrm{REL}_2}$; large deviations that flip the kernel overlaps would break the theorem. Behaviorally, any data-free combination of two one-hop LoRAs that reliably solves a held-out two-hop task with no CoT bridge and no target-task expert in the library would falsify the paper's central claim.","tokens_in":22738,"feed_emoji":"🧩","tokens_out":9550,"duration_ms":82223,"temperature":0.7,"pith_summary":"This position paper argues that the machine-learning community is chasing the wrong problem in LoRA recycling: merging or routing low-rank adapters (LoRAs), the small plug-in weight updates used to specialize large language models, does not reliably compose knowledge from separately fine-tuned models. The paper's central claim is that combining LoRAs works for a new task only when that task is already represented in the fine-tuning data, and that reported successes often reflect shallow pattern matching, such as entity familiarity or shared templates, rather than genuine compositional generalization. A simplified theoretical model of factual recall shows why: a LoRA trained on a one-hop fact edits the model's output only on a narrow activation subspace, so when two such adapters are summed on a two-hop prompt they return a linear mixture of the two edited entities, not their composition. Synthetic two-hop and math word-problem experiments on models up to 70B parameters support this, with two-adapter libraries failing on held-out compositions and math-specialized models even losing accuracy after routing. If the paper is right, data-free LoRA reuse is not a dependable route to new capabilities, and research should shift from inventing merger algorithms to mapping when reuse can work.","feed_headline":"Recycling LoRAs fails at composing separate knowledge","feed_subtitle":"Two-hop and math tests show adapters only merge when the target task already appears in fine-tuning data.","key_machinery":"The load-bearing object is the rank-one LoRA update characterized in Proposition 1. For a single fact edit, the minimal-L2 update takes the form $\\Delta W \\propto (i_{\\tilde r(x)} - i_{r(x)}) \\mathrm{ReLU}(U(V e_X + e_{\\mathrm{REL}}))^T$: it attaches the new output vector to the activation subspace of the one-hop prompt. Combining two such adapters is then analyzed through the random-features kernel $k(\\eta,\\xi) \\approx \\frac{1}{m}\\mathrm{ReLU}(U\\eta)^T \\mathrm{ReLU}(U\\xi)$ of a wide ReLU MLP. On the two-hop prompt the activation $\\xi = U(\\frac12 V e_X + \\frac12 V e_{\\mathrm{REL}_1} + e_{\\mathrm{REL}_2})$ has positive kernel overlap with both one-hop activation subspaces $\\eta_1,\\eta_2$, so the summed adapters contribute both edited entities with comparable weights instead of computing the composition. This mechanism, subspace overlap producing a mixture, is what rules out compositional integration for summing, weighted averaging, and similarity-based routing alike.","core_discovery":"The paper's formal anchor is Theorem 2: if two rank-one LoRAs are each trained to edit one fact of the relations r1 and r2, then summing the adapters does not produce correct answers to the composed two-hop query. The proof shows that the minimal-L2 update for a single fact is a rank-one term $\\Delta$ W = p q^T aligned with the prompt's ReLU activation; on a two-hop prompt the activation overlaps the subspaces of both one-hop adapters, so the combined model outputs a weighted linear combination of the edited entities rather than r2(r1(x)). Empirically, a library containing only the two one-hop adapters stays below 10 percent accuracy on two-hop tests across base models and entity types, while adding an expert trained on the target relation or CoT-formatted bridge examples recovers performance. In the GSM-Symbolic easy-to-hard math setting, routing LoRAs trained on simpler variants generally degraded accuracy on the harder variant, especially for math-specialized models. The paper concludes that reuse is effective only when target tasks are already represented, and advocates a research pause on new merging and routing algorithms.","pith_inferences":["Going beyond the paper: if the subspace-overlap mechanism is right, LoRA composition should be surface-form sensitive—adapters trained on prompts whose wording matches the two-hop prefix (e.g., ‘spouse of X lives in’) should compose better than those trained on paraphrases; the paper's template-split data could test this directly.","Going beyond the paper: the theorem is a one-layer, single-head simplification, so for real multi-layer LLMs the empirical results carry the argument; a natural extension is to test whether chain-of-thought-distilled base models can overcome the mixture failure without bridge training.","Going beyond the paper: if the position holds, LoRA hubs are better understood as task-specific assets than plug-and-play capability libraries; practical value would hinge on cataloguing what each adapter encodes and what the base model already knows, not on the routing algorithm.","Going beyond the paper: the math experiments suggest a testable asymmetry—LoRAs fine-tuned on code or Tool-Integrated-Reasoning solutions may preserve a math-specialized model's tool use, while natural-language CoT LoRAs suppress it; matching solution format between fine-tuning and base-model behavior would isolate whether the failure is compositional or a format mismatch."],"forward_implications":["Data-free LoRA merging and routing should not be treated as a mechanism for composing skills: with only the two one-hop adapters, two-hop accuracy stays below 10 percent across base models.","Reliable adapter reuse for a new task requires training on the target reasoning pattern; CoT-formatted bridge examples in both adapters recover high accuracy, while omitting them collapses performance.","Entity and domain familiarity is a precondition for even partial success: fake names and fake locations consistently fail, whereas real names and real locations succeed.","Routing can actively degrade a model: on GSM-P2, routing LoRAs trained on natural-language solutions dropped Qwen2.5-Math-7B-Instruct from 68 percent to 34 percent (Uniform) or 27 percent (Arrow), partly by suppressing its tool-integrated-reasoning behavior.","Weighted averaging and similarity-based routing inherit the same mixture failure as plain summing, so improving routing heuristics alone is unlikely to unlock composition."],"supporting_citations":[{"why":"Defines low-rank adapters (LoRA), the reusable objects whose combination the paper scrutinizes.","marker":"[Hu et al., 2021]"},{"why":"Supplies the one-layer transformer factual-recall model that the theoretical setup adapts.","marker":"[Nichani et al., 2025]"},{"why":"The closed-form rank-one edit of Proposition 1 is presented as analogous to the RoME update.","marker":"[Meng et al., 2022]"},{"why":"Provides the random-features regime used to derive the kernel overlaps in the proof.","marker":"[Rahimi and Recht, 2008]"},{"why":"Motivates the two-hop reasoning task and documents that models trained on A→B, B→C fail to learn A→C.","marker":"[Balesni et al., 2025]"},{"why":"Provides the Arrow routing method evaluated alongside uniform averaging.","marker":"[Ostapenko et al., 2024]"},{"why":"Represents the competing LoRA-concatenation approach whose claimed math composition the paper challenges.","marker":"[Prabhakar et al., 2024]"},{"why":"Provides GSM-Symbolic and the easy-to-hard GSM-P1/P2 benchmarks used in the math experiments.","marker":"[Mirzadeh et al., 2024]"}],"fun_headline_variants":["LoRA recycling fails on two-hop reasoning tests","Pause new LoRA merging: study when reuse actually works","LoRA reuse hides shallow pattern matching, not composition","Reusing LoRAs fails without target knowledge in fine-tuning","Why LoRA recycling fails: composition needs seen tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative theorem assumes a specific attention pattern on the two-hop prompt, namely that the MLP input is $U(\\frac12 V e_X + \\frac12 V e_{\\mathrm{REL}_1} + e_{\\mathrm{REL}_2})$, with coefficients one half, one half, and one; if real transformer attention on such prompts deviates from these coefficients, the proof's categorical conclusion about actual LLMs does not necessarily follow.","fun_headline_variants_meta":{"raw":{"variants":["LoRA recycling fails on two-hop reasoning tests","Pause new LoRA merging: study when reuse actually works","LoRA reuse hides shallow pattern matching, not composition","Reusing LoRAs fails without target knowledge in fine-tuning","Why LoRA recycling fails: composition needs seen tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1552,"prompt_tokens":983,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":490}},"tokens_in":599,"tokens_out":569,"duration_ms":5361,"temperature":1.0,"reasoning_tokens":490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:01:03.093055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the actual attention-weighted value vectors of a real transformer on the prompt $X\\ \\mathrm{REL}_1\\ \\mathrm{REL}_2$ and check whether the induced MLP input matches the assumed $\\frac12 V e_X + \\frac12 V e_{\\mathrm{REL}_1} + e_{\\mathrm{REL}_2}$; large deviations that flip the kernel overlaps would break the theorem. Behaviorally, any data-free combination of two one-hop LoRAs that reliably solves a held-out two-hop task with no CoT bridge and no target-task expert in the library would falsify the paper's central claim.","supporting_citations":[{"cited_title":"Lee, and Alberto Bietti","cited_arxiv_id":null,"evidence_quote":"Supplies the one-layer transformer factual-recall model that the theoretical setup adapts."},{"cited_title":"Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning","cited_arxiv_id":null,"evidence_quote":"Provides the random-features regime used to derive the kernel overlaps in the proof."},{"cited_title":"The two-hop curse: Llms trained on a b, b c fail to learn a c, 2025","cited_arxiv_id":null,"evidence_quote":"Motivates the two-hop reasoning task and documents that models trained on A→B, B→C fail to learn A→C."},{"cited_title":"Towards modular LLM s by building and reusing a library of L o RA s","cited_arxiv_id":null,"evidence_quote":"Provides the Arrow routing method evaluated alongside uniform averaging."},{"cited_title":"Lora soups: Merging loras for practical skill composition tasks, 2024","cited_arxiv_id":null,"evidence_quote":"Represents the competing LoRA-concatenation approach whose claimed math composition the paper challenges."}],"review_version":2}