{"id":"d042a9b5-b55a-4bae-9757-df3d9e6d91c3","arxiv_id":"2505.23844","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adaptive selection and dynamic weighted fusion of source LLMs reduces knowledge interference and improves target model accuracy compared to FuseLLM.","lead":"The paper presents Fusion-X, a method that adaptively selects a subset of source language models and fuses their output probabilities to train a single target model. It reports higher accuracy and fewer degraded tasks than the FuseLLM baseline across several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 50% reduction in knowledge interference is not causally attributable to the adaptive selection network because no random or fixed-selection baseline is compared; the binary per-task degradation count is noise-sensitive and the learned selection may merely reflect a constant source…","rationale":"The reader's weakest assumption focuses on whether the selection network can learn useful scores from raw probability matrices. My concern is adjacent but more specific: the paper never demonstrates that the learned selection is the cause of the reported gains, because no random-selection or static-selection control is included. The ablation in Table 1 compares Top-2, Adaptive, and All, but Top-2 still uses the learned network's scores, so it does not isolate the value of learning. Appendix C shows stable selection frequencies, which suggests the network may learn a constant preference for one source (e.g., the one with lowest perplexity) rather than a per-input relevance signal. If that is the case, the ASN is effectively a fixed router, and the improvement over FuseLLM could be achieved by a simple heuristic. The 50% figure is also statistically fragile because it is a count of tasks with small exact-match differences, without variance or significance testing. A McNemar test on the paired 27-task degradation indicators would clarify whether the reduction is credible. This concern does not refute the method; it means the central claim is currently under-evidenced. The reader's CONDITIONAL verdict is appropriate, and I would keep it unchanged, with the condition that the authors add a random/static selection baseline and a significance test. I partially agree with the reader because the root issue is the same—whether the ASN contributes meaningfully—but my proposed test targets attribution more directly than the reader's general concern about feature learning.","tokens_in":19006,"tokens_out":6833,"duration_ms":84094,"concrete_test":"Run the identical training pipeline and hyperparameters with the ASN replaced by: (a) a frozen network with randomly initialized logits (or a fixed 50/50 random selection per sample), and (b) a static rule that always selects the top-2 source LLMs by validation perplexity on MiniPile. Keep the threshold, fusion weights, and loss terms unchanged. Compare average BBH/MMLU accuracy and the count of degraded tasks against the reported Fusion-X across at least 3 random seeds. If (a) or (b) performs within a standard error of Fusion-X, the adaptive selection mechanism is not the driver of the claimed 50% interference reduction. Additionally, apply McNemar's test to the paired 27-task degradation indicators for Fusion-X vs FuseLLM to check whether the 5-vs-10 difference is statistically significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the adaptive selection network (ASN, §4.1) reduces knowledge interference and improves accuracy over FuseLLM, with the headline '50% reduction' derived from the number of BBH tasks that fall below the target model (10 degraded tasks for FuseLLM vs 5 for Fusion-X, §5.2, Tab. 3). This evidence is not sufficient for the claim in two ways. First, FuseLLM is the only knowledge-fusion baseline; there is no control in which the ASN is replaced by random selection or by a fixed rule (e.g., always select the source model with lowest validation perplexity). Any improvement over FuseLLM could come from the weighted fusion, the continued training objective, or simply from dropping low-quality sources, rather than from learned per-input selection. Second, the 'knowledge interference' metric is a binary per-task count with no error bars or significance test; at 27 tasks, a one-task flip changes the reported reduction from 50% to 40%, and per-task exact-match scores in Tab. 3 differ by fractions of a percent. The ASN input is a flattened, token-aligned probability matrix (Eq. 4) and the paper does not analyze which features the network uses; Appendix C shows a stable selection distribution over training, which is consistent with the network learning a constant preference rather than a context-dependent relevance signal. Therefore the load-bearing assumption that learned scores are responsible for the improvement is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Fusion-X, a framework for aggregating knowledge from multiple source LLMs into a single target LLM. The main components are an adaptive selection network (ASN) that scores source models from their token-level probability matrices, a dynamic weighted fusion step that combines the selected models' distributions, and a feedback loss intended to prevent the selector from collapsing onto a single source. The method is evaluated on commonsense, BBH, MMLU, and code benchmarks across model scales from 160M to 8B, and the authors report consistent gains over continued training and FuseLLM, including a headline claim of a 50% reduction in knowledge interference measured by the number of BBH tasks that degrade relative to the target model. The paper also includes ablations over selection count, selection metric, fusion method, threshold, and the feedback loss, plus a comparison with several ensemble, MoE, and weight-merging baselines.","tokens_in":19348,"tokens_out":7279,"duration_ms":88142,"significance":"If the central claim is validated, the paper would make a useful step toward scalable and flexible integration of heterogeneous LLMs: it targets a real limitation of FuseLLM-style knowledge fusion, and the experimental scope is broad, spanning multiple model families, scales, and benchmarks. The authors provide a code repository and explicit ablations of the main design choices, which are strengths. However, the causal role of the adaptive selector is not yet established: the reported gains could come from the weighted fusion, the continued-training objective, or simple dropping of low-quality sources, and the headline '50% reduction' rests on a binary per-task count without uncertainty quantification. The significance of the contribution is therefore conditional on additional controlled experiments and reproducibility details.","major_comments":[{"comment":"The ASN architecture as written is not implementable for variable-length sequences. Eq. (4) applies linear layers f1, f2, f3 to the flattened concatenation Pcat of matrices Pi in R^{N×V}; since N, the sequence length, varies across training examples, the input dimension to f1 is not fixed. Algorithm 1 additionally reports logit tensors of shape (L, D, N), which is inconsistent with Eq. (4). Please specify whether the selector is applied per token, per sequence, or with a fixed-length window, and give the exact input/output dimensions and how variable lengths are handled. This detail is necessary to understand what the ASN actually computes and to reproduce the method.","section":"§4.1, Eq. (4); Algorithm 1"},{"comment":"The headline '50% reduction in knowledge interference' is a binary count of tasks below the target-model baseline (10 for FuseLLM vs. 5 for Fusion-X) from a single run, with no error bars, multiple seeds, or significance test. At 27 tasks, one task flip changes the reduction from 50% to 40%, and several per-task exact-match scores in Table 3 differ by less than one point. The same fragility applies to the degradation percentages in Fig. 1 and to Table 6. Please report variance over seeds and a statistical test (or at least a confidence interval) for the degradation count.","section":"§5.2, Table 3; §6"},{"comment":"The causal role of the ASN is not isolated. Within the knowledge-fusion category, FuseLLM is the only comparison, and the 'Selection count' rows of Table 1 change the selection policy together with the fusion weights and the feedback loss; there is no control in which the learned scores are replaced by random selection or by a fixed rule (e.g., always select the source with lowest validation perplexity) while the remaining components are held fixed. As a result, the gains over FuseLLM could be explained by dropping low-quality sources, by the dynamic weighted fusion, or by the continued-training objective rather than by the learned per-input selection that is the paper's main contribution. Please add such controls.","section":"§5.2, Table 3; §4.1"},{"comment":"The reported selection distribution is stable over training and concentrated on one or two sources; this is consistent with the ASN having learned a fixed preference rather than a context-dependent relevance signal. The text interprets stability as dynamic adjustment, but no analysis shows how selection varies across inputs or whether the learned scores contain information beyond average source quality. The claim of per-input adaptivity in §4.1 ('based on input data and current learning context') therefore needs supporting evidence, such as per-input selection statistics or a comparison of learned scores with a fixed ranking of source models.","section":"Appendix C, Fig. 6"}],"minor_comments":[{"comment":"The loss D(Tt, Pf) is used without defining D; please specify that it is a cross-entropy or KL divergence between the target and fused distributions.","section":"Eq. (10)"},{"comment":"The abstract and conclusion refer to 'three benchmarks', but §5.1 and Appendix G describe four (CS, BBH, MMLU, MultiPL-E); please align the count and the descriptions.","section":"Abstract and §5.1"},{"comment":"Percentage changes for tasks with near-zero baselines, such as Multistep Arithmetic Two (+1100%), are uninformative and should be accompanied by raw score differences.","section":"Table 3"},{"comment":"The statement that results have a standard deviation of −0.02 to +0.02 lacks seed information; please explain how this was computed and over how many runs.","section":"§5.2"},{"comment":"The hyperparameters λ_fuse and λ_feed are tuned on 10% of the validation set and only the best configuration is shown; please report sensitivity of the main results to these values.","section":"Appendix B"},{"comment":"There is a typo ('avaliable') in the abstract/code line, and reference [8] has incomplete author formatting; please fix these.","section":"Abstract; References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an incremental extension of FuseLLM with an adaptive selection module, and the experimental scope is substantial. My main reservation is attribution rather than novelty: the headline interference-reduction claim is not yet supported by the evidence because no control isolates the selector and no uncertainty quantification is given. The architecture description in §4.1 also needs correction. These issues are addressable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this paper extends FuseLLM-style knowledge fusion with a learned gating network and dynamic weighted fusion, and it shows consistent gains across scales and benchmarks. The ablations in Table 1 genuinely support the design choices. If you work on model fusion, it is worth reading.\n\nThe weak point is the headline claim: 'reduces knowledge interference by up to 50%.' That number is just 10 degraded tasks for FuseLLM versus 5 for Fusion-X on BBH—a binary per-task count with no error bars. One task flip changes it to 40%. More importantly, there is no control where the selection network is replaced by a fixed rule (e.g., always pick the source with lowest validation perplexity, or pick a random subset). Without that, the improvement over FuseLLM could come from the weighted fusion, or simply from dropping weak sources, rather than from learned per-input selection. The stress-test note is correct on this point.\n\nAppendix C reinforces the worry: the selection distribution is stable over training and strongly favors one source model. That is consistent with the network learning a constant preference, not a context-dependent relevance signal. The paper does not show per-input variation.\n\nWhat the paper does well: the finding that naive fusion degrades as more sources are added is demonstrated clearly (Fig 1); the method is simple and the ablations are thorough; and the comparison set is reasonable, including weight merging baselines. Missing error bars and validation-set hyperparameter tuning are standard weaknesses, not fatal ones.\n\nVerdict: the package likely works—Fusion-X beats FuseLLM by a few points across tasks—but the mechanism story is oversold. A serious referee should ask for (a) a fixed-selection control, (b) multiple-seed runs, and (c) analysis of what the ASN actually learns. With those, this would be a solid contribution. I would send it to review.","headline":"A useful extension of FuseLLM-style fusion with adaptive selection; the method appears to work, but the headline '50% interference reduction' is not causally pinned down by the experiments.","tokens_in":19842,"tokens_out":4149,"would_cite":true,"duration_ms":43134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned selector decides which source LLMs to fuse, and the paper reports it halves the number of degrading tasks.","keywords":["knowledge fusion","large language models","adaptive selection","knowledge interference","dynamic weighting","model integration","LLM merging"],"falsifier":"A reader could test the central claim by replacing the Adaptive Selection Network's scores with random scores drawn from the same distribution and keeping the same threshold and fusion weights: if random selection reproduces the 50% reduction in degrading BBH tasks, the learned signal is not load-bearing. A second check is to shuffle the token-probability entries within each source matrix before feeding them to the ASN; if benchmark gains persist, the network is not actually reading the distributions.","tokens_in":18864,"feed_emoji":"🧠","tokens_out":8476,"duration_ms":87398,"temperature":0.7,"pith_summary":"Building one stronger language model from several weaker or specialized ones usually means blending all of their next-token probability distributions into the target model. The paper argues that this all-in fusion is the source of the trouble: candidates that are irrelevant to a given task dilute what the target knows, and the paper shows that with FuseLLM the fraction of Big-Bench Hard and MMLU tasks that degrade grows as more models are added. Its proposed framework, Fusion-X, instead learns to score every source model from its token-level probability matrices, selects a subset on each training batch, and fuses only the selected candidates with normalized weights. The authors report that this reduces knowledge interference by up to 50% compared with FuseLLM, with average gains roughly doubled on BBH (2.7% to 5.3% over Llama-2-7B), while keeping the target model's parameter count and memory footprint unchanged.","feed_headline":"Adaptive selection cuts knowledge fusion interference by up to 50%","feed_subtitle":"A learnable selector picks which source models to fuse, so more models improve—not hurt—the target.","key_machinery":"The load-bearing mechanism is the Adaptive Selection Network (ASN), a three-layer linear network with GELU activations that maps the concatenated, flattened, layer-normalized probability matrices of all candidate LLMs to one logit per candidate. Softmax turns those logits into selection probabilities, and a threshold $\\tau=0.15$ converts them into a per-batch subset of candidates, with the highest-probability candidate used as a fallback so at least one source is always selected. The dynamic weighted fusion step then forms a fused distribution $P_f$ as the weighted sum of the selected candidates' probability matrices, and the feedback loss $L_{\\mathrm{feed}} = \\mathrm{CV}^2(\\{\\hat p_j\\})$ penalizes concentration of the normalized selection weights. The design work it does is to make both which models fuse and how strongly they fuse depend on the input and the training step, turning fusion into a learned, data-dependent operation.","core_discovery":"The central discovery is that knowledge fusion of LLMs is better framed as a per-sample routing problem than as a fixed blending problem. Fusing all available source models treats every candidate as equally useful, and the paper documents the failure mode: on BBH, FuseLLM finishes below the target model on 10 of 27 tasks, and the degradation ratio grows with more sources. Fusion-X's Adaptive Selection Network takes the flattened and layer-normalized probability matrices of all candidates, produces a softmax score per candidate, and keeps only those above a threshold; the selected distributions are then combined with weights proportional to the normalized scores. A feedback-driven loss, defined as the squared coefficient of variation of the candidate weights, prevents the selector from collapsing onto one small subset. In the paper's experiments this yields an average +5.3% on the 27 BBH tasks over the target model, with only five tasks degrading, which the authors count as a 50% reduction in knowledge interference relative to FuseLLM; similar gains hold at 100M, 3B, 7B, and 8B scales and on MMLU, commonsense, and code benchmarks.","pith_inferences":["An unstated consequence of the paper's design is that once the selector is trained, it could be reused at inference time to route each new input to a small subset of source models, turning the framework into a cheap conditional ensemble without further training.","The same interference mechanism should appear in knowledge distillation and multi-task continual learning, so a parallel testable extension is to downweight or skip examples from sources whose distributions disagree strongly with the target, rather than always blending them.","The paper does not test whether the selection scores match an oracle ranking of source models by per-task validation performance; a reader could check this by correlating ASN probabilities with per-source task accuracies.","Because the input to the selector is only probability matrices, the approach may transfer to other sequence models with token-level distributions, such as speech or biological sequence models, provided the token-alignment step is adapted."],"forward_implications":["Scaling the number of fused source models can improve the target rather than degrade it: Fusion-X reports rising accuracy when moving from three to five candidates at both 100M and 7B scales, while FuseLLM's degradation ratio rises.","Heterogeneous models with different architectures, tokenizers, and vocabularies can be integrated without ensemble memory overhead or weight-merging's uniform-architecture constraint.","Training becomes cheaper: the paper reports reaching the same perplexity in about half the training steps and matching FuseLLM-level performance with about three times fewer tokens.","Interference can be measured and controlled: the fraction of tasks falling below the target model on BBH drops from ten (FuseLLM) to five (Fusion-X), and the paper interprets this as up to 50% less knowledge interference.","The learned selector has interpretable behavior: in the 100M-scale run it consistently favors Tiny-Starcoder over Pythia-160M, indicating the scores track something like per-task utility rather than raw model size."],"supporting_citations":[{"why":"The knowledge-fusion baseline Fusion-X is built on and compared against; supplies the token-alignment method and the distribution-matrix fusion objective.","marker":"[49]"},{"why":"Extends FuseLLM to chat models; the paper treats it as a related knowledge-fusion approach with the same interference problem.","marker":"[50]"},{"why":"Ensemble baseline requiring all models at inference; used to contrast memory cost and scalability.","marker":"[22]"},{"why":"Big-Bench Hard benchmark used to measure the fraction of tasks that degrade, the paper's operational definition of knowledge interference.","marker":"[45]"},{"why":"MMLU benchmark used for scaling and interference analysis over 57 tasks.","marker":"[18]"},{"why":"MiniPile continual-training corpus used for all fusion runs.","marker":"[24]"},{"why":"Perplexity-optimized ensemble baseline in the method comparison table.","marker":"[36]"},{"why":"Weight-merging baseline that removes interfering delta parameters; comparison anchor for architecture-constrained methods.","marker":"[56]"}],"fun_headline_variants":["Adaptive LLM selection cuts fusion interference up to 50%","Pick the most relevant LLMs: interference drops up to 50%","Fuse only the best LLM sources to reduce interference by half","Custom LLM fusion: up to 50% less interference via adaptive selection","Dynamic LLM weighting: interference cut by 50% in knowledge aggregation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a three-layer network can learn genuinely useful selection scores from the flattened, token-aligned probability distribution matrices of the source LLMs; if those matrices carry no recoverable signal about which models would harm the target, the adaptive selector cannot outperform fixed or random fusion.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive LLM selection cuts fusion interference up to 50%","Pick the most relevant LLMs: interference drops up to 50%","Fuse only the best LLM sources to reduce interference by half","Custom LLM fusion: up to 50% less interference via adaptive selection","Dynamic LLM weighting: interference cut by 50% in knowledge aggregation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00132,"raw_usage":{"total_tokens":5401,"prompt_tokens":997,"completion_tokens":4404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":4309}},"tokens_in":613,"tokens_out":4404,"duration_ms":31824,"temperature":1.0,"reasoning_tokens":4309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:03:38.537469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the central claim by replacing the Adaptive Selection Network's scores with random scores drawn from the same distribution and keeping the same threshold and fusion weights: if random selection reproduces the 50% reduction in degrading BBH tasks, the learned signal is not load-bearing. A second check is to shuffle the token-probability entries within each source matrix before feeding them to the ASN; if benchmark gains persist, the network is not actually reading the distributions.","supporting_citations":[{"cited_title":"Knowl- edge fusion of large language models","cited_arxiv_id":null,"evidence_quote":"The knowledge-fusion baseline Fusion-X is built on and compared against; supplies the token-alignment method and the distribution-matrix fusion objective."},{"cited_title":"Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Weight-merging baseline that removes interfering delta parameters; comparison anchor for architecture-constrained methods."}],"review_version":1}