{"id":"26e9352d-7504-49fd-bf7c-1b91f4905455","arxiv_id":"2412.01289","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Concatenating visual tokens from multiple MLLMs in the same family and linearly merging their language model deltas yields a training-free fusion that improves multimodal benchmark scores over each source model.","lead":"VisionFuse combines vision encoders from several off-the-shelf multimodal AI models that share the same underlying language model, without any extra training, by pasting their visual tokens together and merging the language model weights. On standard visual question-answering benchmarks, the merged model beats each individual model, with an average gain of about 4% for a two-model combination.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported >4% average gain rests on a test-set-tuned merge coefficient λ, with no described validation split; the headline improvement may be inflated by test-set leakage.","rationale":"The reader's weakest_assumption was that the merged language model can jointly interpret visual tokens from multiple family encoders without alignment training. That is an important concern about the method's mechanism and generalizability, and the failure of cross-family merging (Table 9) shows the condition is not automatic. However, the paper provides multiple empirical demonstrations of compatibility within a family: the TextVQA interpolation curve in Figure 4c, the ablations in Table 2, and the successful integrations of MGM-8B with SLIME-8B and Eagle-8B. The mechanism is plausible even if not derived. The hyperparameter-selection issue, by contrast, directly threatens the validity of the headline number. A training-free method is not exempt from an unbiased evaluation protocol; if λ is tuned on the same test benchmarks used to compute the average, then the >4% claim has no clear statistical meaning. The proposed concrete test would settle this by recomputing with validation-selected λ and by reporting λ sensitivity. This sharpens the reason for the reader's conditional verdict without changing it: the main condition should be a clean hyperparameter-selection protocol and transparent reporting of the search procedure.","tokens_in":24777,"tokens_out":7050,"duration_ms":62047,"concrete_test":"Fix λ using a validation set disjoint from all nine reported test benchmarks (e.g., a 20% split of each benchmark's training data, or a separate benchmark such as ScienceQA). Recompute Table 1 with this validation-selected λ, including the average relative gain. Also report the gain for a fixed λ = 0.5 without any search. If the average gain over the nine benchmarks falls below the claimed >4%, the headline number is an artifact of test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—an average relative gain of over 4% for MGM-8B+SLIME-8B (Section 5, Table 1)—depends on the coefficient λ in the parameter-merge equation Θmerged = Θpre + λ Σ(Θi − Θpre) (Eq. 4). Appendix A.2 reports a search over λ ∈ {0.1, 0.3, 0.5, 0.7, 0.9, 1.0} for Task Arithmetic and for other merging methods, but it does not state that the search is performed on a validation split. Figure 4c selects α = 0.5 by maximizing accuracy on TextVQA itself, and TextVQA is one of the nine benchmarks averaged in the headline. Selecting the best of six λ values on the evaluation benchmarks constitutes test-set leakage; with such a small hyperparameter search this can materially inflate the reported gain, especially on the tuning benchmark. If λ is instead chosen on a held-out set, or if results are shown to be insensitive across the range, the claim becomes credible. As reported, the magnitude of the improvement is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VisionFuse is a training-free framework that improves the visual perception of multimodal LLMs by integrating multiple MLLMs that share the same pretrained language model. The method concatenates visual tokens from the vision encoders of several family members and merges their LLM delta parameters as Θₘₑᵣ₉ₑₐ = Θₚᵣₑ + λ·Σ(Θᵢ − Θₚᵣₑ). The central claim is that, without additional training, fusing MiniGemini-8B and SLIME-8B yields an average relative improvement of over 4% across nine multimodal benchmarks, with ablations showing that both token concatenation and LLM merging are necessary. The paper also reports results for 7B models, a high-resolution variant, an Eagle-8B combination, and three-model integration with token pruning.","tokens_in":24930,"tokens_out":4267,"duration_ms":38158,"significance":"If the reported gains are robust, VisionFuse offers a practical, low-overhead route to improving MLLM perception by reusing existing checkpoints, and it provides useful empirical evidence about the compatibility of vision encoders and LLM deltas within an MLLM family. The paper's strengths include consistent improvements across a wide benchmark suite, explicit ablations isolating the two components (Table 2), a negative control showing that simply duplicating tokens does not help (Figure 5), and a striking cross-family failure (Table 9) that supports the family-condition hypothesis. The main reservation is the selection of the merging coefficient λ on the evaluation benchmarks themselves, which affects the credibility of the headline quantitative claim.","major_comments":[{"comment":"The merging coefficient λ is selected by maximizing accuracy on TextVQA (Figure 4c), and the hyperparameter search in Table 5 is performed over the same benchmarks that are later averaged in Table 1. The paper does not state that a held-out validation split is used. Because TextVQA is one of the nine benchmarks in the headline average, the reported \"over 4%\" improvement is partly a consequence of test-set selection. Please either choose λ on a held-out validation set (e.g., a few tasks not in the evaluation suite) or report the full benchmark results across the entire λ range to show that the average improvement is not an artifact of peaking on TextVQA.","section":"Appendix A.2, Eq. (4), Figure 4c, Table 1"},{"comment":"The average \"increase of over 4%\" is an arithmetic mean of per-benchmark relative gains that range from +1.0% (MMBench) to +14.7% (Vizwiz), with MME scores and accuracy percentages mixed in the same average. With λ tuned on one of these benchmarks, the mean is not a stable statistic. Please clarify exactly how the average is computed, report the median and the range, and, ideally, provide bootstrap confidence intervals or a sensitivity table across λ to establish the stability of the central claim.","section":"Section 5, Table 1"},{"comment":"The three-model integration result relies on randomly dropping 1000 visual tokens, and this drop count is selected by maximizing TextVQA accuracy in Figure 8c. This is another free parameter chosen on the evaluation set. The three-model improvement over the two-model baseline should therefore be reported with the same caveat, and a sensitivity analysis over the number of dropped tokens should be provided so that the reader can gauge the robustness of that result.","section":"Appendix D.2, Figure 8c, Table 7"}],"minor_comments":[{"comment":"The heading \"Empircal Insights\" appears to be a typo and should read \"Empirical Insights.\"","section":"Section 3, title"},{"comment":"The word \"Comparasions\" in the caption should be \"Comparisons.\"","section":"Table 3, caption"},{"comment":"The sentence \"This limitation leeds to degraded performance\" contains a typo; \"leeds\" should be \"leads.\"","section":"Appendix D.1"},{"comment":"The phrase \"in both the the combinations\" contains a duplicated article and should be \"in both the combinations.\"","section":"Section 5, paragraph on main results"},{"comment":"Please clarify in the caption whether the accuracy curve corresponds to the merged model with both vision encoders or to a single-encoder model; the text is ambiguous about what is being interpolated.","section":"Figure 4c"},{"comment":"The comment \"Merge language model parameters using Eq. (4)\" is placed as a step; consider making it a formal numbered step to match the algorithmic description in the text.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution, but the λ-selection issue is central to the quantitative claim. If the authors can show that the improvement persists when λ is chosen on a proper validation split or that the results are insensitive across λ, the paper would likely be acceptable. I would also encourage the editor to consider whether the lack of code release hinders reproducibility assessment, though that is not a blocking issue for the review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper with a real but possibly inflated headline number. The idea is simple and genuinely useful: concatenate visual tokens from multiple MLLMs that share the same base LLM, and merge the LLM deltas so one model can read all the token streams. No training. The within-family alignment observation is the load-bearing empirical claim, and they support it with t-SNE and cross-attention IoU; that part holds up. Ablations are informative: duplicating tokens doesn't help, concatenation beats addition, and merging LLM parameters is necessary. Results are consistent across 9 benchmarks in the main table and 12 in the appendix, including Eagle-8B as a third family member, which is a nice bonus.\n\nSoft spots, in order of size. First, the merging coefficient λ is chosen by searching on the evaluation benchmarks themselves. Appendix A.2 lists the search grid but never mentions a validation split, and Figure 4c picks α by maximizing TextVQA accuracy—TextVQA is one of the nine averaged benchmarks. Six candidate values, one of them being the tuned one; that's test-set leakage, and it can inflate the reported gain on the tuning benchmark. The stress-test note is right. Second, the headline \"over 4% average improvement\" is the mean of per-benchmark relative gains, not a pooled accuracy gain; the relative gain on Vizwiz is +14.7% from a low base, which pumps the average. Third, no code or checkpoints are released, and there are no error bars, so the stability of these numbers is unknown. The three-model extension needs manually chosen token-drop counts, also selected on the eval set, so that part is more of a proof-of-concept.\n\nNone of this kills the central claim—fusion within a family clearly helps, and the mechanism is plausible—but the exact magnitude is less certain than the abstract suggests. The cross-family failure in Table 9 is honestly presented and actually supports their story.\n\nWho this is for: people building or benchmarking MLLMs, especially those who want cheap inference-time gains without retraining. It deserves a serious referee: conditional accept with requests for code, validation-based hyperparameter selection, and pooled accuracy numbers.","headline":"Training-free fusion of same-family MLLMs is a real and useful trick, but the headline gain is inflated by test-set-tuned hyperparameters.","tokens_in":25515,"tokens_out":1633,"would_cite":false,"duration_ms":14926,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VisionFuse claims that fusing same-family MLLMs by concatenating vision tokens and merging delta parameters improves multimodal benchmark scores by over 4% on average, with no training.","keywords":["multimodal large language models","training-free fusion","vision encoder ensemble","delta parameter merging","task arithmetic","visual perception","MLLM family","token concatenation"],"falsifier":"Take two same-family MLLMs with deliberately divergent visual preprocessing, for example one crop-based and one patch-based, and large pairwise delta differences; if a tuned $\\lambda$ cannot keep TextVQA accuracy above the better individual model, the compatibility premise fails. The paper's own cross-family result, MGM-8B + VILA-8B dropping TextVQA from 67.6 to 33.0, is a partial falsifier; a systematic scan across same-family pairs with measured delta distances would settle how far the claim extends.","tokens_in":89,"feed_emoji":"🧩","tokens_out":8316,"duration_ms":130025,"temperature":0.7,"pith_summary":"VisionFuse tries to establish that the visual perception of multimodal large language models (MLLMs) can be improved without any training by combining off-the-shelf models that share the same pretrained language model. The paper claims that two operations are enough: concatenate the visual tokens produced by the different models' vision encoders, and merge the language models' delta parameters by weighted interpolation toward their shared base model. Fusing Mini-Gemini-8B and SLIME-8B this way raises average performance by more than 4% over the stronger individual model across nine multimodal benchmarks, at a cost of only 3.4% more parameters from the added encoder. If the claim holds, it matters because it sidesteps the expensive re-alignment step that normally accompanies any new or additional vision encoder.","feed_headline":"No-training fusion lifts multimodal LLM scores by 4%","feed_subtitle":"Same-family vision encoders give complementary attention; merging their language-model deltas lets one LLM read all of them.","key_machinery":"The load-bearing mechanism is a two-part, training-free pipeline. A vision ensemble runs each member's preprocessing, encoder, and projector to obtain visual tokens $V_i$, then feeds the concatenation $V_F = [V_1; \\dots; V_n]$ into the language model alongside text tokens. In parallel, an LLM-merging step forms $\\Theta_{\\mathrm{merged}} = \\Theta_{\\mathrm{pre}} + \\lambda \\sum_{i=1}^M (\\Theta_i - \\Theta_{\\mathrm{pre}})$, the task-arithmetic rule of adding scaled fine-tuning deltas to the shared base model. The identity that carries the argument is this linear interpolation: it is what aligns one language model with several vision encoders that were never jointly trained, and its validity is what confines the method to a single MLLM family.","core_discovery":"The central discovery is that within an MLLM family, defined as models built on the same pretrained LLM, two previously separate techniques become compatible: visual-token concatenation and delta-parameter merging. The paper's three observations are that different MLLMs attend to different image regions for the same query, that same-family vision encoders produce feature distributions similar enough to be concatenated into one visual context, and that interpolating the language models' deltas, $\\Theta_{\\mathrm{merged}} = \\Theta_{\\mathrm{pre}} + \\lambda \\sum_i (\\Theta_i - \\Theta_{\\mathrm{pre}})$, lets a single LLM parse tokens from multiple encoders. The authors show that on TextVQA the interpolated model peaks near $\\alpha = 0.5$ between the two 8B models, and ablation evidence indicates the gain comes from genuinely richer visual information rather than from longer token sequences, since duplicating one model's tokens does not improve accuracy.","pith_inferences":["If same-family compatibility is the active ingredient, then the useful resource is families of models sharing a base LLM; quantitative measures of delta-parameter distance could predict which pairs will fuse well before running benchmarks.","The success of simple $\\lambda$-scaled delta interpolation suggests the visual-alignment knowledge in these MLLMs is concentrated in small parameter changes; testing sparse-merging variants, such as pruning or conflict-resolving merges, may make the fusion more robust and cheaper to deploy.","The token-length failure mode points to a natural extension the paper leaves open: instead of concatenating all tokens, select a per-encoder subset of the most informative or complementary visual tokens, which could preserve gains while cutting inference cost.","Because the authors' cross-family experiments show sharp drops, the method doubles as a probe: a family's merge success rate could serve as an empirical measure of how much two MLLM fine-tunings share in their visual alignment."],"forward_implications":["Fusing MGM-8B and SLIME-8B outperforms both individual models on every benchmark reported, and reaches parity with MGM-8x7B, a model with over six times as many parameters.","Combining a high-resolution encoder (MGM-HD-8B) with a low-resolution one (SLIME-8B) still improves overall accuracy, so the low-resolution encoder supplies complementary information rather than redundancy.","Both components are necessary: merging deltas without extra encoders gives small gains, while adding extra encoders without merging deltas hurts alignment and accuracy.","Naively concatenating tokens from more than two 7B models degrades performance because sequence lengths exceed the training distribution, but the same three-model fusion on 8B models, or with about 1,000 tokens pruned, improves over the two-model fusion.","The recipe transfers beyond the MGM-SLIME pair, improving MGM+Eagle-8B across TextVQA, MMBench, MMMU, Ai2d, OCRBench, and RealworldQA."],"supporting_citations":[{"why":"Supplies Mini-Gemini, one of the two fused 8B models, along with its high-resolution vision encoder and attention behavior.","marker":"[27]"},{"why":"Supplies SLIME, the other fused 8B model, whose local-patch tokens and complementary attention drive the gains.","marker":"[54]"},{"why":"Provides task-arithmetic delta merging, the exact rule used to form the merged language model in Eq. (4).","marker":"[19]"},{"why":"Establishes encoder-token concatenation as a fusion strategy and serves as the trained baseline VisionFuse improves upon training-free.","marker":"[40]"},{"why":"Defines LLaMA-3-8B-Instruct, the shared base LLM against which the 8B-family deltas are computed.","marker":"[1]"},{"why":"Defines Vicuna-v1.5, the shared base LLM for the 7B-family experiments.","marker":"[8]"},{"why":"Supplies the GPU-hour and dollar cost estimates for aligning new encoders, motivating the training-free claim.","marker":"[48]"},{"why":"Supplies VILA, the model used in cross-family merges whose failures bound the method's family-restricted validity.","marker":"[29]"}],"fun_headline_variants":["Training-free fusion of vision encoders boosts MLLMs by 4%","Merging LLM deltas lets one model read multiple vision encoders","Complementary vision encoders, no retraining, +4% multimodal scores","Fuse encoders from same family: training-free MLLM gain","One LLM, many vision encoders: no-training fusion lifts scores 4%"],"cache_read_input_tokens":27648,"weakest_assumption_plain":"The load-bearing premise is that linearly averaging the fine-tuning differences of same-family models produces a single language model that can interpret every member's visual tokens without any alignment training; the paper establishes this empirically from one family pair and observes that it fails when deltas are large or models come from different families.","fun_headline_variants_meta":{"raw":{"variants":["Training-free fusion of vision encoders boosts MLLMs by 4%","Merging LLM deltas lets one model read multiple vision encoders","Complementary vision encoders, no retraining, +4% multimodal scores","Fuse encoders from same family: training-free MLLM gain","One LLM, many vision encoders: no-training fusion lifts scores 4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":3010,"prompt_tokens":1006,"completion_tokens":2004,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1902}},"tokens_in":622,"tokens_out":2004,"duration_ms":11360,"temperature":1.0,"reasoning_tokens":1902,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:29:53.008929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two same-family MLLMs with deliberately divergent visual preprocessing, for example one crop-based and one patch-based, and large pairwise delta differences; if a tuned $\\lambda$ cannot keep TextVQA accuracy above the better individual model, the compatibility premise fails. The paper's own cross-family result, MGM-8B + VILA-8B dropping TextVQA from 67.6 to 33.0, is a partial falsifier; a systematic scan across same-family pairs with measured delta distances would settle how far the claim extends.","supporting_citations":[{"cited_title":"Editing models with task arithmetic","cited_arxiv_id":null,"evidence_quote":"Provides task-arithmetic delta merging, the exact rule used to form the merged language model in Eq. (4)."},{"cited_title":"Llama 3 model card","cited_arxiv_id":null,"evidence_quote":"Defines LLaMA-3-8B-Instruct, the shared base LLM against which the 8B-family deltas are computed."},{"cited_title":"Vila: On pre-training for visual language models","cited_arxiv_id":null,"evidence_quote":"Supplies VILA, the model used in cross-family merges whose failures bound the method's family-restricted validity."}],"review_version":1}