{"id":"fdc9ddb7-f5ff-4f37-bae2-5f20a074adf8","arxiv_id":"2505.10939","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Subtracting a Wikipedia-trained general knowledge LoRA from task LoRAs before Arrow routing improves zero-shot generalization in modular LLMs.","lead":"This paper proposes a preprocessing step for modular LLMs: train a 'general knowledge' LoRA on Wikipedia, subtract it from each task-specific LoRA, then route the cleaned modules with the Arrow algorithm. The authors report average accuracy gains over Arrow on English, French, and German zero-shot benchmarks using Phi-3 and Phi-2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control for the general LoRA's content: gains may be a generic centering effect, not evidence of knowledge subtraction.","rationale":"The reader identified the proxy for general knowledge as the weakest assumption; I agree and sharpen it to a specific missing control. The paper's empirical gains over Arrow are plausible, but the mechanistic claim that subtracting a Wikipedia-trained LoRA removes redundant general knowledge is underdetermined. The central experiment would be to replace the general LoRA with controls that match its training regime but lack its content. If the gains persist under these controls, the results are consistent with a generic centering effect on routing prototypes, not with knowledge disentanglement. This does not change the overall assessment: the paper is a useful empirical contribution but requires additional evidence (error bars, significance, and the proposed control) to support its stated mechanism. Therefore the conditional verdict remains appropriate.","tokens_in":12343,"tokens_out":11307,"duration_ms":111501,"concrete_test":"Train three control LoRAs on Phi-3 with identical hyperparameters: (i) the same 5,000 Wikipedia segments with token order shuffled within each segment; (ii) 5,000 segments from a non-Wikipedia domain (e.g., news text); (iii) a random LoRA (Gaussian A and B) scaled to the average Frobenius norm of the Wikipedia LoRAs. Subtract each control from the same task LoRAs, run Arrow on the residuals, and compare average accuracy on the English, German, and French benchmarks (Tables 1-2) to the reported GenKnowSub numbers. Additionally, compute pairwise cosine similarity of Arrow prototypes before and after subtraction; if gains track prototype separation, a centering mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's explanatory claim rests on the hypothesis (Section 2.1) that a LoRA fine-tuned on 5,000 Wikipedia segments with a 5-word causal LM objective captures the redundant general knowledge inside task LoRAs. The paper provides no experiment separating the content of this general LoRA from its role as a generic subtracted direction. The Mean Normalization baseline subtracts the average task LoRA and fails, but it does not control for subtracting an arbitrary LoRA of similar norm or one trained on non-Wikipedia text. If GenKnowSub's gains persist when the subtracted module is a random LoRA, shuffled-text LoRA, or a news-domain LoRA, then the reported improvements are explained by a centering or regularization of Arrow's routing prototypes rather than by removal of shared general knowledge. This would invalidate the paper's core interpretation that residual modules 'focus more exclusively on task-relevant information.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GenKnowSub, a method that subtracts a general-domain LoRA (trained on small Wikipedia corpora with a causal language-modeling objective) from each task-specific LoRA in a library, then applies the zero-shot Arrow routing algorithm to select and combine the resulting residual modules. The method is evaluated on Phi-3 and Phi-2 across English, German, and French benchmarks, including multiple-choice reasoning tasks and open-ended SNI generation. The main reported results are average accuracy gains over Arrow: 1.6% on English, 3.9% on German, and 3.6% on French, plus a Rouge-L gain on SNI. The paper frames these gains as evidence that subtracting redundant general knowledge improves modularity and reusability of LLMs, and it releases code and data.","tokens_in":12510,"tokens_out":3995,"duration_ms":42295,"significance":"If the average-gain claim is robust, GenKnowSub is a simple, training-free post-processing step that could improve the reusability of existing LoRA libraries in modular LLM systems, with no changes to the routing algorithm. The paper's strengths include evaluation on multiple languages and base models, a held-out zero-shot setup with no circularity, and public code and data. However, the empirical support is currently incomplete: no variance or significance testing is reported, the 'consistent gains' claim is contradicted by several per-dataset drops, and the central interpretation that a Wikipedia-trained LoRA captures 'general knowledge' is not tested against content-free alternatives. These issues are fixable and do not invalidate the method's average-gain observation, but they need to be addressed before the paper's stronger claims can be accepted.","major_comments":[{"comment":"The core claim that GenKnowSub improves over Arrow rests on average gains of 1.6%, 3.9%, and 3.6%, but no variance, standard error, or significance test is reported. Many per-dataset differences are small; for example, PIQA-French drops from 80.20 (Arrow) to 78.78 (GenKnowSub-Fr), and ARC-Easy is lower for every GenKnowSub variant than for Arrow. Without multiple seeds, confidence intervals, or at least a paired test across datasets, it is not possible to tell whether the average improvements are systematic or noise. Please report variance or explicitly state that the results are single runs and temper the significance accordingly.","section":"§3.3, Tables 1 and 2"},{"comment":"The explanatory claim that subtracting a Wikipedia-trained LoRA removes redundant general knowledge is not supported by the current experiments. The Mean Normalization baseline subtracts the average task LoRA and fails, but it does not control for subtracting an arbitrary LoRA of comparable norm, a LoRA trained on shuffled or non-Wikipedia text, or a domain LoRA trained on news or other text. If any of these substitutes produces the same gains, the results would be explained by a generic centering or regularization effect rather than by general-knowledge subtraction. Please add controls with random LoRA, shuffled-text LoRA, and a non-Wikipedia domain LoRA to establish that the content of the general module is what matters.","section":"§2.1, §3.2"},{"comment":"The abstract and Section 3.3 state that GenKnowSub yields 'consistent performance gains,' but this is contradicted by the per-dataset numbers in Table 1: all GenKnowSub variants underperform Arrow on ARC-Easy and WinoGrande, and GenKnowSub-Fr underperforms on PIQA. Table 3 also shows Phi-2 BoolQ dropping from 81.13 (Arrow) to 80.12 (GenKnowSub-En). Please replace 'consistent' with an accurate description (e.g., average gains) and report per-dataset win/loss counts, or provide a statistical justification for calling the pattern consistent.","section":"Abstract, §3.3"}],"minor_comments":[{"comment":"The construction of LoRA_avg as the simple average of the English, French, and German general LoRAs is not justified; please explain why this average is a better approximation of 'general knowledge' than any individual language module, or present it as a convenience baseline.","section":"§3.2"},{"comment":"There is a typo in the heading: 'Resutls' should be 'Results.'","section":"Appendix B.2"},{"comment":"The Phi-2 experiments use the task LoRAs from Ostapenko et al. trained on full cluster data, whereas the Phi-3 task LoRAs are trained on a 20% subsample; please make this difference explicit in the main text when comparing cross-model conclusions, since it complicates the interpretation of the weaker Phi-2 gains.","section":"Appendix B, Tables 3 and 4"},{"comment":"The SNI Rouge-L scores and the Phi-2 Table 3 results are reported without any error bars or sensitivity analysis; adding at least one repeated-run or bootstrap estimate would strengthen confidence in these numbers.","section":"Table 3 and SNI paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its hardware limitations and releases code and data, which is commendable. The empirical average-gain observation is plausible, but the current manuscript overstates consistency and lacks controls that would distinguish the proposed knowledge-subtraction mechanism from a generic centering effect. The missing content-control experiment is the most important request; without it, the paper's title-level claim is not established even if the average gains reproduce. I am recommending major_revision rather than rejection because the issues are addressable with additional experiments and reporting changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nI've read GenKnowSub. The core move is simple: train a LoRA on Wikipedia text, subtract it from each task LoRA before running Arrow routing, and let Arrow select among the residuals. The paper reports average gains of 1.6–3.9 points over Arrow across English, German, and French, plus an honest boundary case on Phi-2. That is a real, useful empirical result if it holds. The idea is clean, the experimental narrative is clear, and the Mean Normalization baseline shows the authors are already thinking about naive alternatives.\n\nThe soft spots are two. First, the abstract says 'consistent gains,' but the tables show per-dataset drops on ARC-Challenge and PIQA-French, so the wording overstates the evidence. That is a fixable presentation issue. Second, and more substantially, the interpretation that a Wikipedia-tuned LoRA captures redundant general knowledge is under-tested. There is no control that subtracts a random LoRA, a shuffled-text LoRA, or a news-domain LoRA of similar norm. Without that, the gains could come from a generic centering or regularization effect on Arrow's routing prototypes. The Mean Norm baseline is not the right control because it subtracts a direction constructed from the task modules themselves. The stress-test note is correct to push on this.\n\nI still think the paper deserves a serious referee. The method is new, cheap, and applies to any LoRA-library pipeline, and the Phi-2 results are a sign of honest reporting. I would ask the authors to add the arbitrary-direction control and report variance across seeds. If the control shows the effect is specific to Wikipedia-text LoRAs, the knowledge story becomes convincing; if not, the method still works but the explanation changes.\n\nFor a reading group: worth a session if you work on modular LLMs, especially alongside the task arithmetic literature.","headline":"A cheap and plausible preprocessing step for modular LLMs, but the knowledge-disentanglement story needs a control before I'd believe it.","tokens_in":13036,"tokens_out":2568,"would_cite":false,"duration_ms":23539,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Subtracting a Wikipedia-trained general-knowledge LoRA from each task LoRA leaves residual modules that sharpen Arrow routing and lift zero-shot accuracy by up to 3.9 points.","keywords":["modular LLMs","LoRA subtraction","zero-shot generalization","cross-lingual transfer","task arithmetic","parameter-efficient fine-tuning","Arrow routing","general knowledge disentanglement"],"falsifier":"Train the general LoRA on scrambled or randomly shuffled Wikipedia text, or on a corpus from a completely different domain, while keeping every other detail identical; if zero-shot routing gains persist, the effect is not due to general-knowledge subtraction but to a generic parameter perturbation. A second check is to measure pairwise similarity or overlap between task LoRAs before and after subtraction; if the residual modules are not more distinct, the proposed mechanism of disentanglement is not what drives the improvement.","tokens_in":12162,"feed_emoji":"🧩","tokens_out":7729,"duration_ms":70383,"temperature":0.7,"pith_summary":"The paper argues that modular zero-shot transfer is held back because task-specific LoRA adapters still carry redundant general knowledge inherited from the base model. It proposes GenKnowSub: train a LoRA on a small Wikipedia-like corpus, then subtract that general-domain LoRA from each task LoRA to leave a residual module focused on task-specific information. These residual modules are then combined by the Arrow routing algorithm, which selects and mixes the most relevant modules for each input token without extra training. In experiments on Phi-3, this subtraction improves average accuracy over Arrow by 1.6 percentage points on English benchmarks, 3.9 points on German, and 3.6 points on French, and it also lifts Rouge-L on the open-ended SNI benchmark. The broader point is that if general and task knowledge can be separated in parameter space, modular LLM libraries become more reusable and more transferable across languages.","feed_headline":"A single vector subtraction lifts LoRA routing by up to 3.9%","feed_subtitle":"Task LoRAs stripped of shared general knowledge route better on unseen English, German, and French tasks.","key_machinery":"The central object is the residual LoRA, defined as $\\operatorname{LoRA}^i_{\\text{res}} = \\operatorname{LoRA}^i_{\\text{ts}} - \\operatorname{LoRA}_g$: a task-specific low-rank adapter minus a general-knowledge LoRA trained on a small Wikipedia corpus with a causal language-modeling objective. The general LoRA acts as a proxy for the broad linguistic and factual knowledge already present in the base model; subtracting it is meant to remove redundant content and force each module to carry only task-specific signal. The other load-bearing component is the Arrow routing algorithm, which computes the SVD of each LoRA, uses the top right singular vector as a prototype, projects input tokens onto these prototypes, and forms a softmax-weighted combination of the top $k$ modules per token. Together, the subtraction step makes modules more distinctive and the router more selective.","core_discovery":"On its own terms, the discovery is that a single subtraction operation in LoRA parameter space can disentangle general linguistic knowledge from task-specific adaptation. Formally, the paper writes each residual module as $\\operatorname{LoRA}^i_{\\text{res}} = \\operatorname{LoRA}^i_{\\text{ts}} - \\operatorname{LoRA}_g$, where $\\operatorname{LoRA}_g$ is trained on 5,000 Wikipedia segments per language with a causal language-modeling objective. The resulting residual modules are more distinct from one another, and the Arrow router, which computes a per-token weighted combination of the top-$k$ modules, can then select and compose them more effectively. The paper reports consistent gains across English, German, and French benchmarks with Phi-3, and on SNI GenKnowSub reaches a Rouge-L of 46.91 versus 45.44 for Arrow. On Phi-2, the benefit appears only when the base model has enough multilingual competence; in weak non-English settings GenKnowSub does not beat Arrow.","pith_inferences":["If the Wikipedia-trained LoRA truly recalls pretraining knowledge, then the strength of subtraction should be tunable: smaller corpora or domain-matched corpora should produce weaker or stronger disentangling. This is a testable prediction not reported in the paper.","The method could be combined with any training-free router, not just Arrow; a router that uses distances in parameter space might benefit even more from the increased distinctiveness of residual modules.","Subtracting a general module is a form of centering in LoRA space; viewed that way, GenKnowSub suggests that the shared knowledge of a model can be approximated by a single centroid, which could be learned or updated as new tasks enter the library.","An important untested boundary is low-resource languages: the paper uses only English, French, and German general LoRAs, so whether subtraction helps when the base model has little exposure to the target language remains open."],"forward_implications":["Existing LoRA libraries can be upgraded without retraining: subtracting one shared general LoRA from every task module is a single arithmetic step before routing.","Multilingual zero-shot transfer is improved when the base model has cross-lingual ability; the same residual modules route better on German and French benchmarks even when the subtracted general LoRA is English.","The method stays parameter-efficient: routing remains post-hoc and training-free, so the gains do not require joint training of experts and router.","On weaker or less multilingual base models, the expected benefit is limited to languages the base model already knows; GenKnowSub does not create multilinguality that is not there."],"supporting_citations":[{"why":"Supplies the Arrow routing algorithm, the clustered Flan task-module library, and the Phi-2 baseline that GenKnowSub builds on and is compared against.","marker":"Ostapenko et al. 2024"},{"why":"Defines LoRA, the parameter-efficient adapter format in which task-specific and general knowledge modules are trained.","marker":"Hu et al. 2021"},{"why":"Establishes the negation and task-arithmetic principle that makes subtracting one module from another a meaningful operation.","marker":"Ilharco et al. 2023"},{"why":"Shows that arithmetic operations can be applied to parameter-efficient modules, grounding the module-level subtraction.","marker":"Zhang et al. 2023"},{"why":"Provides the Flan Collection that is clustered into task groups to train the task-specific LoRAs.","marker":"Longpre et al. 2023"},{"why":"Supplies the Wikipedia segments used to train the general-knowledge LoRAs in each language.","marker":"WikimediaFoundation"},{"why":"Provides Phi-3-mini-4k-instruct, the primary base model for the English, German, and French experiments.","marker":"Abdin et al. 2024"},{"why":"Provides Phi-2, the smaller and less multilingual base model used to test how the method transfers to weaker LLMs.","marker":"Javaheripi et al. 2023"},{"why":"Provides the Super-Natural Instructions benchmark used to measure open-ended zero-shot generation with Rouge-L.","marker":"Wang et al. 2022"}],"fun_headline_variants":["Subtract general knowledge to boost LoRA routing","General knowledge subtraction sharpens task LoRAs for routing","GenKnowSub: subtract shared knowledge, improve module routing","One subtraction in LoRA space lifts multi-task routing","Single subtraction lifts LoRA routing by up to 3.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that fine-tuning a LoRA on 5,000 Wikipedia-like segments with a causal language-modeling objective actually captures the general knowledge that is redundant across task LoRAs; if that proxy fails, subtraction is just removing arbitrary parameter directions and the reported gains could come from a generic regularization effect.","fun_headline_variants_meta":{"raw":{"variants":["Subtract general knowledge to boost LoRA routing","General knowledge subtraction sharpens task LoRAs for routing","GenKnowSub: subtract shared knowledge, improve module routing","One subtraction in LoRA space lifts multi-task routing","Single subtraction lifts LoRA routing by up to 3.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3616,"prompt_tokens":981,"completion_tokens":2635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2554}},"tokens_in":597,"tokens_out":2635,"duration_ms":18168,"temperature":1.0,"reasoning_tokens":2554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:00:09.723977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the general LoRA on scrambled or randomly shuffled Wikipedia text, or on a corpus from a completely different domain, while keeping every other detail identical; if zero-shot routing gains persist, the effect is not due to general-knowledge subtraction but to a generic parameter perturbation. A second check is to measure pairwise similarity or overlap between task LoRAs before and after subtraction; if the residual modules are not more distinct, the proposed mechanism of disentanglement is not what drives the improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Arrow routing algorithm, the clustered Flan task-module library, and the Phi-2 baseline that GenKnowSub builds on and is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Phi-2, the smaller and less multilingual base model used to test how the method transfers to weaker LLMs."}],"review_version":1}