{"id":"c4409c84-66ae-4e44-acba-33ec3bb147f8","arxiv_id":"2505.17110","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MMER merges multiple multimodal LLMs by averaging task vectors and applying per-modality binary masks, retaining about 99% of original task performance without additional training.","lead":"This paper introduces MMER, a training-free method that combines several multimodal AI models into one by merging their language-model parts and applying binary masks to keep each modality's skills intact. It reports that the combined model retains about 99% of each original model's performance and can also protect old skills when a model is fine-tuned on a new task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"99% retention is demonstrated only on a self-fine-tuned, same-base MLLM family with per-modality thresholds tuned on the target tasks; the claimed 'existing MLLMs' use case and the decoupling assumption itself remain untested.","rationale":"The method is internally coherent and the experiments are broadly consistent with the stated mechanism; the concern is not an inconsistency in Eq. 3 but the strength of the generalization claim. The mask rule in Eq. 3 is exactly the per-coordinate L1 minimizer only at λ=1; the authors tune λ per modality on target tasks, so part of the reported retention is a curve-fitting result rather than a property of the decoupling principle. Moreover, all four MLLMs are self-fine-tuned from one base, so the shared-θ_pre precondition is satisfied by construction, and the task vectors are likely better aligned than in arbitrary public MLLMs. The paper's own Limitations says that finding such existing MLLMs is challenging, which directly narrows the 'reuse existing MLLMs' framing in the abstract and introduction. A held-out test with fixed λ and public models would settle whether the decoupling assumption, rather than the tuning and the self-constructed family, is what delivers the 99% retention. The reader's weakest assumption identified the related no-error-bound issue in the binary mask reconstruction; my emphasis is on the untested external validity and in-sample tuning of that same assumption. The conditional verdict already captures these concerns, so no verdict change is needed.","tokens_in":25595,"tokens_out":11103,"duration_ms":107062,"concrete_test":"Run MMER on four publicly released MLLMs that already share the same base LLM and were not fine-tuned by the authors (e.g., a Vicuna-7B-based LLaVA, Video-LLaVA, audio, and point model), using fixed λ=1.0 and TopK=80% for all modalities. Report per-modality retention on each model's official benchmarks. If any modality falls below 95% trimmed retention, the 99% figure is specific to the authors' self-fine-tuned family and tuned thresholds; if no such public same-base family can be assembled, that itself confirms the 'existing MLLMs' claim is untested.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central reconstruction θ_pre + m_i∘τ* ≈ θ_i (Eq. 1) assumes that an L1-thresholded binary mask of the merged task vector, with no per-coordinate rescaling, preserves task function. The paper's own Limitations section concedes that finding multiple existing MLLMs with the same architecture across modalities is challenging, and every experiment uses four MLLMs the authors fine-tuned themselves from Vicuna-7B (§4.1) with per-modality λ calibrated on target-task validation (§3.1, §B.2). The 99% retention in Table 2 is therefore a within-family, in-sample number. It does not test the headline use case of reusing off-the-shelf MLLMs, nor does it bound the failure mode where selected entries have |τ*| much larger than |τ_i| (e.g., τ* accumulating several aligned task vectors) and Eq. 1 substitutes the merged magnitude unmodified. If the L1 mask does not track functional importance for another base model or with fixed λ, the central claim reduces to a tuned demonstration on a self-constructed family.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMER, a training-free method for multimodal expansion of LLMs by merging the LLM parameters of several MLLMs into a single model while retaining each original MLLM's performance. MMER uses TIES-style task-vector merging to obtain a merged task vector τ*, then constructs per-modality binary masks by comparing each original task vector τ_i with τ* under a directional-consistency and dominant-magnitude criterion (Eq. 3). The masked merged vector is added to the pre-trained LLM to approximate each original LLM (Eq. 1), allowing modality-specific inputs to be processed with modality-specific parameters. The paper reports experiments on four self-fine-tuned Vicuna-7B MLLMs covering vision, audio, video, and point cloud: multimodal expansion across 10 input combinations (Table 1), retention of 99% of original performance averaged over 14 dual-modal tasks (Table 2), and mitigation of catastrophic forgetting in single-task and cross-modal multi-task scenarios (Table 3). Ablation and hyperparameter analyses show the importance of directional congruence, dominant significance, and the scaling factor λ.","tokens_in":25807,"tokens_out":8511,"duration_ms":78117,"significance":"If the results hold beyond the current setup, MMER is a useful contribution: it extends task-vector merging to multimodal models with a simple, train-free decoupling mechanism, and it demonstrates a practical application of model merging to catastrophic forgetting in MLLMs. The paper provides a broad experimental surface (10 input combinations, 14 retention tasks, two forgetting scenarios) and gives an exact per-coordinate derivation of the L1-optimal binary mask for λ=1 (Eqs. 2–3). The main risks are that all experiments use self-fine-tuned MLLMs with per-modality hyperparameters calibrated on the target tasks, and that the central comparison with DAMC is not run. These issues directly affect the strength of the headline '99% retention' and 'existing MLLMs' claims, but they are addressable in revision.","major_comments":[{"comment":"The headline 'retains 99% of original performance' is an in-sample, calibrated result. Section B.2 reports that λ_i is calibrated per modality, and Section 3.1 says λ_i follows the same principle as α, which is calibrated on validation sets from target tasks; moreover, all four MLLMs are self-fine-tuned from the same base (Vicuna-7B, §4.1), and the retention tasks in Table 2 are the same types of tasks used to fine-tune those models and to build the masks. The abstract and introduction say MMER 'integrates existing MLLMs' and targets a training-free expansion for off-the-shelf models, but no experiment uses an existing pre-trained MLLM. Please add an experiment with off-the-shelf same-architecture MLLMs (or an explicitly labeled surrogate) and, failing that, narrow the claims in the abstract and Section 1 to self-fine-tuned MLLMs. This is load-bearing because the 99% number is the paper's central empirical promise.","section":"§4.1, §5, Table 2, §B.2"},{"comment":"The mask formula is not the claimed L1 minimizer unless λ_i=1. The per-coordinate solution to Eq. (2) is m_i^{(p)}=1 iff |τ_i^{(p)}| ≥ 0.5|τ_*^{(p)}| and the signs agree. Equation (3) inserts λ_i, and the authors tune λ_i; Figure 6 shows that varying λ_i changes MCUB accuracy, and the ablation in Table 4 shows that λ contributes to the result. When λ_i≠1, Eq. (3) selects a mask that is suboptimal under Eq. (2), so the 'minimizing Manhattan distance' derivation in Section D.1 does not justify the actual mask. Please report the tuned λ_i values, state explicitly that Eq. (3) is a heuristic relaxation, or provide a bound relating the relaxation to the L1 objective.","section":"§3.1, Eqs. (2)–(3), §D.1"},{"comment":"DAMC is a central comparison point but is never evaluated. The Introduction frames the contribution as bypassing the NaiveMC/DAMC trade-off, and Appendix A claims MMER 'achieves similar results' to DAMC. DAMC is absent from Tables 1 and 2 and from the forgetting experiments, so the reader cannot verify the trade-off claim. If DAMC can be run under the same protocol, please include it in at least the expansion and retention experiments; otherwise, remove or soften the claim that MMER matches DAMC.","section":"§1, §2, Appendix A, Tables 1–2"},{"comment":"The decoupling assumption that selected entries of τ* can stand in for the corresponding entries of each τ_i has no error bound or failure-mode analysis. When |τ_i^{(p)}| lies between 50% and 100% of |τ_*^{(p)}|, the reconstructed parameter uses τ_*^{(p)} unmodified, so the per-coordinate error can be as large as |τ_i^{(p)}|; when the ratio is below 50%, the coordinate is dropped entirely. The paper shows empirically that this works for four self-fine-tuned Vicuna-7B models, but it does not analyze the regime where several task vectors align in sign and accumulate magnitude in τ*. I would like a discussion, or a targeted experiment, of when the no-rescaling reconstruction is expected to hold beyond this model family.","section":"§3.1, Eq. (1)"}],"minor_comments":[{"comment":"The benchmark name is spelled 'MUSIC-A VQA' in the text and references, but appears as 'MUSCI-A VQA' in Table 1 and its caption; please unify the spelling.","section":"Table 1 and throughout"},{"comment":"The role of α is confusing: Section 3.1 defines the merged parameter as θ* = θ_pre + α·τ* with α calibrated on validation sets, while Section B.2 states that α is not set because the TIES-merged model is not used in MMER. Please clarify whether α appears anywhere in MMER and, if not, remove the description or explain why the merged vector needs no scaling.","section":"§3.1 vs. §B.2"},{"comment":"The paper uses the word 'significantly' (e.g., 'MMER significantly outperforms') but reports no error bars, multiple seeds, or significance tests. The consistent improvement across tasks is visible in the tables, but the statistical language should be softened or supported.","section":"§5, Tables 1–3"},{"comment":"The notation in Eq. (2) repeats 'arg min' and uses |·| ambiguously; the expression |mi ◦ τ* − τ_i| should be defined explicitly as the ℓ1 norm over the P coordinates, with the double sum notation cleaned up.","section":"Eq. (2)"},{"comment":"The left panel of Figure 6 mixes bar plots and line plots with several y-axes; the caption states that the lines show performance retention, while the bars show mask percentages. Please use clearly separated panels and a legend that matches the caption so the reader can map each element to its axis.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is scope: the method is plausible and the experiments are extensive, but the evidence is limited to four self-fine-tuned Vicuna-7B models and the hyperparameters are tuned on the target tasks. The missing DAMC comparison is the largest concrete gap relative to the paper's own framing. I would not reject the manuscript; a revision with off-the-shelf or explicitly framed surrogate experiments, reported λ values, and either DAMC results or narrowed claims would address the load-bearing issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nMMER is a training-free way to merge MLLMs that share a base LLM. The core idea is to merge task vectors with TIES, then build per-modality binary masks by comparing each original task vector to the merged one using directional congruence and dominant significance (Eq. 3). That mask rule is a clean, L1-motivated threshold and is the real novelty. Applying it to cross-modal expansion, retention, and forgetting mitigation is new relative to NaiveMC/DAMC, TALL-masks, and Model Tailor, and the experimental sweep is wide: 10 input combinations for expansion, 14 tasks for retention, and two forgetting scenarios. The ablations honestly show that the directional congruence term is essential.\n\nWhat the paper does well: it is clearly written, the baselines are reasonable and numerous, and the 99% retention figure holds up on the trimmed average. The storage analysis and the comparison with Model Tailor are thoughtful. The L1 derivation for the mask is simple but sound.\n\nThe soft spots are real but not fatal. First, all four MLLMs were fine-tuned by the authors from Vicuna-7B for this paper. The abstract's 'reuse existing MLLMs' story is untested; the Limitations section concedes that finding existing same-architecture MLLMs across modalities is hard. So the headline use case is a proof-of-concept on a self-constructed family. Second, the per-modality λ and TopK are tuned on validation sets from the target tasks, so the numbers are tuned, not predicted. There are no error bars or significance tests. Third, the decoupling assumption has no theoretical bound: Eq. 1 substitutes tau*'s magnitude for tau_i's when tau_i is between 50% and 100% of tau*, which can be a factor-of-2 mismatch. Empirically it works on these models, but nothing guarantees it on another base model or data distribution.\n\nWho should read this: anyone working on model merging or multimodal model composition. It is a useful contribution to that subfield, not a field-reorganizing one. It deserves a serious referee. I would accept it for review with the expectation that the authors add error bars, a sensitivity analysis for fixed λ, and ideally one experiment on off-the-shelf MLLMs. Code release would also help.\n\nMy recommendation: engage with it. It is worth a slot in a good venue after revision.\n\nBest,","headline":"Training-free mask-based decoupling for MLLM merging is a clean and useful idea, but the 99% retention and 'reuse existing MLLMs' claims are only demonstrated on a self-fine-tuned same-base family with tuned thresholds.","tokens_in":26426,"tokens_out":3545,"would_cite":true,"duration_ms":28788,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes MMER, a training-free method that merges the LLM parameters of several existing MLLMs and decouples them with binary masks, claiming multimodal expansion with about 99% retention of each original model's performance and…","keywords":["model merging","task vectors","multimodal large language models","parameter decoupling","binary masks","catastrophic forgetting","training-free multimodal expansion","modality-specific routing"],"falsifier":"Compute the per-parameter relative reconstruction error $|m_i(p)\\tau^*(p) - \\tau_i(p)| / |\\tau_i(p)|$ for each modality of the paper's four MLLMs. If the audio modality, whose task-vector magnitudes are about six times smaller than vision and point, shows large error yet still retains performance, or if applying MMER to a different pair of same-backbone MLLMs with denser task vectors drops original-task retention well below 99%, the decoupling approximation would be shown to be setup-dependent rather than a general property of merged task vectors.","tokens_in":25367,"feed_emoji":"🧩","tokens_out":8389,"duration_ms":63707,"temperature":0.7,"pith_summary":"MMER is a training-free way to give one LLM multiple modality abilities by combining models that already have them. Starting from several multimodal LLMs that share the same pretrained backbone, the method merges the vectors of their fine-tuning updates (task vectors) into one vector, then builds a binary mask per modality that selects which entries of the merged vector belong to that modality. The claim is that these masks decouple the merged model well enough that each original model can be reconstructed nearly exactly (about 99% retention on the paper's benchmarks), so the same merged model can handle image, audio, video, and point-cloud inputs without retraining and without the usual performance drop of model merging. The paper also claims the same mechanism absorbs newly fine-tuned models and thereby mitigates catastrophic forgetting while adapting to new tasks.","feed_headline":"Merge four AI models into one and keep 99% of their skills","feed_subtitle":"One merged model keeps 99% of each original model's multimodal skills, with no retraining.","key_machinery":"The load-bearing objects are task vectors and modality masks. A task vector $\\tau_i$ is the difference between a fine-tuned MLLM's LLM weights and the shared pretrained weights $\\theta_{pre}$, so it represents what that modality's training added. TIES merging sparsifies these vectors and combines sign-consistent entries into $\\tau^*$. The modality mask $m_i$ is a binary vector that keeps an entry of $\\tau^*$ only when original and merged signs agree and the original entry is dominant (Eq. 3). During inference, each modality's tokens are processed through $\\theta_{pre} + m_i \\circ \\tau^*$ for attention projections, while text tokens use the average mask, so every modality gets its own effective weights without any change to the encoder stack.","core_discovery":"On its own terms, the central discovery is that a single merged task vector can carry multiple modality-specific models inside it, and that a simple comparison of the original and merged task vectors is enough to find them. For each modality $i$, MMER keeps a parameter of the merged task vector $\\tau^*$ in the reconstruction exactly when that parameter has the same sign in $\\tau_i$ and $\\tau^*$ (Directional Congruence) and when $\\tau_i$'s magnitude is at least $\\lambda\\cdot 50\\%$ of $\\tau^*$'s magnitude (Dominant Significance), with $\\lambda$ controlling the number of selected parameters. The resulting mask $m_i$ yields $\\theta_{pre} + m_i \\circ \\tau^*$ as a stand-in for the original model $\\theta_i$. The same recipe gives multimodal expansion (each input type is processed through its own decoupled weights), multimodal retention (reconstructed models score about 99% of original performance), and catastrophic-forgetting mitigation (a newly fine-tuned MLLM is treated as one more vector to merge and decouple).","pith_inferences":["If the mask-based decoupling is as general as the experiments suggest, the approach could be applied to any collection of same-backbone models that differ only by fine-tuning, not just modality-specific MLLMs; the main constraint is that base-model checkpoints must match exactly.","A testable extension is to treat the mask selection threshold $\\lambda\\cdot 50\\%$ as a per-modality calibration knob and measure whether retention degrades smoothly enough to make the method robust without validation sets.","The method's ability to absorb newly fine-tuned models implies that catastrophic forgetting can be reframed as a merging problem: instead of updating one model, keep the old vector and merge in the new one, which scales naturally to many tasks only if mask overlap stays low.","The reported gains on classification tasks for audio and point cloud suggest that merging can unlock instruction-following behavior that the original fine-tuned models lacked; whether this unlocks or merely rescues capabilities is not settled by the paper."],"forward_implications":["A single merged MLLM can accept image, audio, video, and point-cloud inputs, including combinations, with no additional training or extra inference parameters.","Original MLLMs can be reconstructed from the shared merged task vector plus per-modality masks, retaining roughly 99% of trimmed-average performance while using less storage than keeping all original models.","Adding a newly fine-tuned MLLM to the merge set preserves previous-task scores and recovers nearly all of the new task's gain, mitigating catastrophic forgetting in both single-task and cross-modal multi-task scenarios.","Because modality masks are sparse (the audio mask selects only 2.2% of parameters), the success of decoupling depends on task vectors being highly redundant, consistent with earlier findings that supervised fine-tuning adds sparse delta parameters.","Storage grows roughly linearly with the number of merged MLLMs (base LLM, merged task vector, masks, and per-modality encoders), but inference cost remains that of one model."],"supporting_citations":[{"why":"Defines the NaiveMC and DAMC frameworks that MMER builds on and supplies the MCUB benchmark and experimental setup for merging MLLMs.","marker":"Chen et al., 2024a"},{"why":"Introduces task vectors and task arithmetic, the representation MMER merges and decouples.","marker":"Ilharco et al., 2023"},{"why":"Provides the TIES merging strategy MMER uses to sparsify and sign-consistently merge task vectors.","marker":"Yadav et al., 2023"},{"why":"Supports the redundancy premise with the observation that fine-tuned task vectors are mostly redundant delta parameters, which MMER exploits via sparse masks.","marker":"Yu et al., 2024"},{"why":"Gives the theoretical grounding that task arithmetic works when weights are sufficiently disentangled, motivating the decoupling step.","marker":"Ortiz-Jimenez et al., 2023"},{"why":"The TALL-masks information-localization method whose mask-based restoration idea MMER adapts and extends to sign consistency.","marker":"Wang et al., 2024"},{"why":"Model Tailor, the same-modality catastrophic-forgetting baseline that MMER compares against in retention experiments.","marker":"Zhu et al., 2024b"},{"why":"Provides the X-InstructBLIP framework used for audio MLLM construction and as a training-based baseline.","marker":"Panagopoulou et al., 2024"}],"fun_headline_variants":["Training-free merge expands LLM modalities, keeps 99% skills","Merge multimodal models without retraining, keep 99% ability","New method merges MLLMs, retains 99% of original skills","Zero-training multimodal expansion via parameter decoupling","Merge and decouple to add modalities without forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decoupling assumption that selected entries of the merged task vector can stand in for the corresponding entries of each original task vector without any magnitude rescaling is load-bearing; when $|\\tau_i|$ is between 50% and 100% of $|\\tau^*|$, the reconstructed weight uses $\\tau^*$'s magnitude, which can be up to twice $\\tau_i$, and the paper gives no error bound for this approximation.","fun_headline_variants_meta":{"raw":{"variants":["Training-free merge expands LLM modalities, keeps 99% skills","Merge multimodal models without retraining, keep 99% ability","New method merges MLLMs, retains 99% of original skills","Zero-training multimodal expansion via parameter decoupling","Merge and decouple to add modalities without forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":3159,"prompt_tokens":968,"completion_tokens":2191,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2118}},"tokens_in":584,"tokens_out":2191,"duration_ms":13507,"temperature":1.0,"reasoning_tokens":2118,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:16:47.416833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-parameter relative reconstruction error $|m_i(p)\\tau^*(p) - \\tau_i(p)| / |\\tau_i(p)|$ for each modality of the paper's four MLLMs. If the audio modality, whose task-vector magnitudes are about six times smaller than vision and point, shows large error yet still retains performance, or if applying MMER to a different pair of same-backbone MLLMs with denser task vectors drops original-task retention well below 99%, the decoupling approximation would be shown to be setup-dependent rather than a general property of merged task vectors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces task vectors and task arithmetic, the representation MMER merges and decouples."},{"cited_title":"Raffel, and Mohit Bansal","cited_arxiv_id":null,"evidence_quote":"Provides the TIES merging strategy MMER uses to sparsify and sign-consistently merge task vectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the theoretical grounding that task arithmetic works when weights are sufficiently disentangled, motivating the decoupling step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the X-InstructBLIP framework used for audio MLLM construction and as a training-based baseline."}],"review_version":1}