{"id":"77e7dc33-f8c0-421f-b8b2-3f080a0477a1","arxiv_id":"2506.06406","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SMAR, a KL-based regularizer on per-modality expert routing, retains 86.6% of a Mixtral 8x7B's language score during visual instruction tuning with only 2.5% pure-text data.","lead":"This paper introduces SMAR, a training-time regularizer that pushes image and text tokens toward different experts inside a Mixture-of-Experts multimodal model, while keeping the two routing patterns a controlled distance apart. If the results hold, it would let multimodal models keep most of their language ability while using almost no text-only data during visual instruction tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline language-retention advantage is not robust: the 86.6% vs 81.6%/82.8% gap is driven almost entirely by MBPP; excluding MBPP from Table 2 reduces the retention gap to about 0.5 percentage points.","rationale":"The reader's CONDITIONAL verdict is correct, and I would keep it unchanged. I agree with the broad concern that the retention claim is fragile, but I place the load-bearing weight on benchmark construction rather than on the absence of a theoretical account of the KL regularizer. The reported 86.6% retention number is an average over six benchmarks, and removing MBPP - where SMAR's score is nearly triple the baseline - collapses the retention advantage to about half a percentage point. Since MBPP is a code-formatting-sensitive benchmark and the paper itself explains the MBPP gain via code-format preservation and instruction-following rather than language knowledge retention, the current evidence does not support the abstract's broad 'preserves language capabilities and outperforms baselines' statement. The SMAR thresholds were also tuned on the same language benchmarks used for the headline Table 2, which further inflates confidence. The method may still be a useful cheap regularizer, and the multimodal results are roughly flat, but the central language-retention comparison needs either a robustness analysis across benchmark subsets and multiple seeds or a revised claim that does not rest on MBPP.","tokens_in":12782,"tokens_out":8913,"duration_ms":86649,"concrete_test":"Recompute language-capability retention from the published Table 2 numbers with MBPP removed from both the six-benchmark average and the base-model denominator. If the SMAR retention advantage over the baseline falls below 1 percentage point (a direct calculation gives about 0.5 pp), then the abstract claim of outperforming baselines in language retention should be downgraded or re-benchmarked; this desk-check settles whether the headline comparison is an artifact of one benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing problem is that SMAR's headline advantage over baselines is an artifact of the specific benchmark average chosen. In Table 2, SMAR's retention rate is 86.6%, versus 81.6% for the no-auxiliary-loss baseline and 82.8% for load-balancing loss. Effectively all of that gap comes from MBPP: SMAR scores 28.4, while the baseline and load-balancing baseline score 10.4 and 14.8. Excluding MBPP from the same six-benchmark average, the retention ratios become 90.7% (SMAR), 90.2% (baseline), and 90.3% (load-balancing), a difference of roughly 0.5 percentage points rather than 4-5 points. MBPP is a code-completion benchmark strongly affected by output formatting, and the paper itself attributes its MBPP gains to preservation of code formatting and instruction-following, not to preservation of parametric linguistic knowledge. Because there is only one run, no error bars, the retention benchmark set excludes HumanEval and IFEval without a formal criterion, and the SMAR thresholds in Table 4 were selected using the same language benchmarks reported in Table 2, the central claim that SMAR 'outperforms' the baselines in language retention is not yet supported. This is a metric-construction weakness, not primarily a theoretical one: the causal mechanism of the KL regularizer could be exactly as proposed and the headline claim would still fail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMAR, a regularization loss that penalizes deviations of the symmetric KL divergence between batch-level vision and text routing distributions from a target band, aiming to preserve language capabilities when adapting MoE-based LLMs to multimodal instruction tuning. The method is tested on a Mixtral 8x7B-based VITA-style model using only 2.5% pure-text data, reporting an 86.6% language retention rate versus 81.6% for the no-auxiliary-loss baseline and 82.8% for the load-balancing-loss baseline, and is also applied to MoE-LLaVA. The loss formulation, training recipe, and dataset composition are clearly specified.","tokens_in":13102,"tokens_out":7296,"duration_ms":61893,"significance":"If the retention result were robust, SMAR would be a cheap and practical way to preserve language skills during multimodal adaptation, avoiding the need for large text-only corpora or architectural overhauls. The paper provides a clearly specified loss, a two-stage training protocol, ablation of thresholds and components, and a transfer experiment to another MoE architecture. However, the headline claim is not yet supported because the retention advantage is largely an artifact of benchmark selection (MBPP) and the experiments are single runs without error bars or significance tests.","major_comments":[{"comment":"The headline retention advantage (86.6% vs 81.6% and 82.8%) is almost entirely due to the inclusion of MBPP in the six-benchmark average. If MBPP is excluded, the retention ratios become approximately 90.7% (SMAR), 90.2% (no-auxiliary baseline), and 90.3% (load-balancing), a gap of about 0.5 percentage points. Because the paper itself attributes the MBPP gain to 'preservation of code formatting' and instruction-following (Section 4.3), MBPP is not a clean measure of parametric language knowledge. The authors should report retention with and without MBPP, and justify why MBPP is included while HumanEval and IFEval are excluded from the retention average.","section":"Section 4.3, Table 2"},{"comment":"The SMAR hyperparameters [dmin, dmax] and beta are selected by evaluating the same language benchmarks (MMLU, GSM8K, BBH, MBPP, HumanEval) that are used in the headline retention claim. This means the reported 86.6% reflects a fitted choice, not an out-of-sample prediction. The authors should select thresholds on a validation split or report retention averaged over the whole grid, and show the sensitivity of the headline difference to the threshold choice.","section":"Section 4.4, Table 4"},{"comment":"All experiments are single runs with no error bars or significance tests. After removing MBPP, the difference between SMAR and the baselines is only about 0.5 percentage points, which is within typical run-to-run variance for this type of evaluation. At least three seeds, or an appropriate significance test, are needed to support the claim that SMAR 'outperforms' the baselines in language retention.","section":"Section 4.2, Table 2"},{"comment":"The proposed causal mechanism—that separating routing distributions preserves linguistic knowledge—is not demonstrated. The ablation in Section 4.4 shows that the relation between the MRD distance band and retention is non-monotonic (best at [1.5,2.0], worse at lower bands, with routing collapse at [0.1,0.5]), and the paper provides no analysis linking the achieved layer-wise MRD distance to the retention numbers. A control experiment (e.g., applying the same KL penalty to a random modality split, or measuring retention against the actually achieved distance) would strengthen the causal interpretation.","section":"Section 3.2, Section 4.4"}],"minor_comments":[{"comment":"The claim 'without modifying model architecture' is inconsistent with the introduction of trainable modality-aware bias vectors bv and bt. Please clarify that the core MoE structure is unchanged but that router parameters are added.","section":"Abstract, Section 1, Eqs. (1)-(2)"},{"comment":"The sentence listing gains over LLaVA-1.5-13B gives five percentages (5.4%, 7.0%, 7.4%, 12.8%, 3.0%) for six benchmarks (SQA I, MME, MMBench, MM-Vet, VQAT, VQAv2). Please correct the mismatch.","section":"Section 4.3"},{"comment":"The 'Avg.' retention ratio is computed as the ratio of the averaged raw scores, not the average of the per-benchmark ratios. Please state this formula explicitly to avoid ambiguity.","section":"Table 2, Section 4.3"},{"comment":"The column header 'VQAT' is not defined in the caption; please spell out the benchmark name (e.g., TextVQA) in the caption.","section":"Table 1"},{"comment":"The threshold grid is coarse and the best setting [1.5,2.0] is at the upper end of the tested range; consider extending the grid to show where performance peaks.","section":"Section 4.4, Table 4"},{"comment":"The limitations section correctly acknowledges the two hyperparameters and limited sensitivity exploration; consider also noting the benchmark-composition sensitivity of the headline retention number.","section":"Section 5, Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a multimodal LLM venue. The central weakness is the benchmark-construction issue: the headline advantage disappears when MBPP is excluded, and the hyperparameters are selected on the same test suite. The lack of multi-seed results further weakens the claim. I recommend major revision with a strong request to report the MBPP-excluded numbers and to add statistical robustness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clearly written methods contribution. The new piece is real: a batch-level Modality Routing Distribution (MRD), a symmetric KL divergence between vision and text routing, and a tolerance band that penalizes only violations, plus a trainable modality bias. That combination is not in Flex-MoE (per-token cross-entropy) or Mono-InternVL (hard partitioning), and it is presented cleanly, with a sensible ablation that separates the bias from the loss and shows what happens when the band is set poorly. The two-stage training recipe is standard, and the generalizability experiment on MoE-LLaVA, though modest, is the right kind of check.\n\nThe soft spot is not the method—it is the headline claim. The 86.6% retention number beats the baselines by roughly five points on the paper's own six-benchmark average, but the stress-test arithmetic is right: exclude MBPP and the gap collapses to about 0.5 percentage points. MBPP is a code-completion benchmark, sensitive to format, and the paper itself attributes the MBPP gain to instruction-following, not to preservation of parametric language knowledge. Including a benchmark that behaves like an instruction-following test in a six-benchmark average, while excluding IFEval and HumanEval without a formal criterion, makes the retention metric a selective construction. On top of that, the results are single runs with no error bars, and the threshold pair (dmin, dmax) is chosen by looking at the same language benchmarks that later appear in Table 2, so some of the headline is fitted hyperparameters.\n\nI would not call this a load-bearing flaw in the training objective itself. The method may still be doing exactly what the authors say. But the paper as written overstates what it demonstrates. A revision that reports multiple seeds, releases code, and either justifies the benchmark selection or uses a pre-specified retention set would make the claim credible.\n\nFor whom: people working on MoE-based MLLMs and on data-efficient ways to preserve language ability during multimodal adaptation. It is a niche but an active one, and the method is cheap enough to try. It deserves a serious referee, but with the expectation of major revision, not acceptance as is.","headline":"A clean, well-specified regularizer with a headline result that mostly evaporates once MBPP is set aside; the method deserves peer review, but the language-retention claim needs better support.","tokens_in":13660,"tokens_out":2840,"would_cite":false,"duration_ms":27767,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A soft routing regularizer keeps a multimodal MoE's language skills mostly intact while training on only 2.5% text.","keywords":["mixture of experts","multimodal large language models","modality-aware routing","language capability retention","KL divergence regularization","visual instruction tuning","expert specialization","routing distribution"],"falsifier":"Train the identical model with SMAR, but compute the MRD distance on shuffled modality labels so the loss has no true signal about which tokens are visual or textual; if language retention still reaches about 86.6%, the reported preservation is not caused by modality separation.","tokens_in":12579,"feed_emoji":"🧠","tokens_out":9569,"duration_ms":79548,"temperature":0.7,"pith_summary":"Most routes to turning a pretrained mixture-of-experts language model into a vision-language model either mix in a large share of text-only data or freeze parts of the backbone, both of which cost efficiency or multimodal quality. This paper proposes a regularizer that tracks the routing distribution of image tokens and text tokens over experts and softly enforces how far apart those two distributions can be. On visual instruction tuning with only 2.5% pure text, the method retains 86.6% of the backbone's language score across six benchmarks, against 81.6% without the auxiliary loss and 82.8% with a standard load-balancing loss. The underlying point is that a small, architecture-free loss term can make experts specialize by modality and thereby protect language competence without a large text corpus.","feed_headline":"A soft routing regularizer preserves 86.6% of language skill","feed_subtitle":"A KL penalty on expert routing keeps most text skills intact in a multimodal MoE with only 2.5% pure-text data.","key_machinery":"The central object is the modality routing distribution (MRD): for each modality $m\\in\\{v,t\\}$, the batch-level probability mass $\\tilde{q}_m$ over experts, built from the frequency with which each expert is selected by tokens of that modality and the average normalized routing weight. The SMAR loss is a hinge on the symmetric KL divergence $d_{\\mathrm{sym\\text{-}KL}}(\\tilde{q}_v,\\tilde{q}_t)$, penalizing distances only when they fall outside $[d_{\\min}, d_{\\max}]$; small trainable modality-aware biases $b_v, b_t \\in \\mathbb{R}^E$ are added to the router logits to make the separation learnable. Together these steer the router toward modality-specialized experts while leaving the transformer architecture, visual encoder, and data untouched.","core_discovery":"The paper's claim is that language degradation during multimodal fine-tuning of a MoE model can be curbed by controlling the gap between the routing distributions of vision and text tokens, rather than by adding text data or freezing layers. Concretely, it defines a modality routing distribution (MRD) per modality, measures the symmetric KL divergence between the vision and text MRDs, and adds a hinge penalty whenever that distance leaves a chosen band $[d_{\\min}, d_{\\max}]$. The resulting SMAR loss steers some experts toward text, some toward images, and leaves others as mixed fusion experts, without hard-assigning experts to modalities. With this loss, the paper reports 86.6% language-capability retention on a six-benchmark average using only 2.5% pure text, while holding or slightly improving most multimodal benchmark scores relative to the baseline and to the load-balancing-only variant.","pith_inferences":["The 86.6% retention number is an average over six benchmarks that includes MBPP, where SMAR's gain is much larger than on the other tasks; averaging only knowledge and reasoning benchmarks would make the advantage over baselines much smaller.","If modality-separated routing is the causal mechanism, an analogous band on pairwise routing distances could also protect other competences such as code or multilingual skill during multimodal expansion, which the paper does not test.","A direct causal test would be to keep the same loss but shuffle the modality labels used to compute the MRDs; if language retention still rose, the effect would not be about true modality separation."],"forward_implications":["With only 2.5% pure text in the instruction-tuning mix, SMAR reaches 86.6% language retention, beating the 81.6% no-auxiliary-loss baseline and the 82.8% load-balancing-only baseline.","SMAR carries over to a second MoE architecture (MoE-LLaVA) when applied during its expert-expansion stage, improving several language benchmarks without hurting multimodal scores.","The tolerance band matters: setting $d_{\\min}=0.1, d_{\\max}=0.5$ causes routing collapse and the worst overall performance, while $[1.5, 2.0]$ gives the best language result.","The final configuration drops the standard load-balancing loss; the paper's ablation shows that adding load balancing to SMAR degrades the result.","The method requires no architecture changes and no frozen backbone, so it can be plugged into existing MoE-MLLM training loops."],"supporting_citations":[{"why":"Supplies Mixtral 8×7B, the pretrained MoE backbone whose router and expert structure SMAR regularizes.","marker":"Jiang et al., 2024"},{"why":"VITA is the base design SMAR extends and the source of the ~20% text-data strategy it aims to surpass.","marker":"Fu et al., 2024"},{"why":"Defines the load-balancing loss used as a baseline and as the auxiliary term in the full objective.","marker":"Fedus et al., 2022"},{"why":"MoE-LLaVA provides the second architecture for generalization tests and the training-data composition.","marker":"Lin et al., 2024"},{"why":"LLaVA-1.5 contributes the two-stage curriculum and the visual instruction tuning datasets.","marker":"Liu et al., 2023b"},{"why":"Supports the premise that pretrained MoE experts already have differentiated knowledge, motivating the third adaptation route.","marker":"Lo et al., 2024"},{"why":"Observed modality preferences in MoE-MLLM routing, the empirical motivation for controlling MRD distance.","marker":"Li et al., 2025"}],"fun_headline_variants":["SMAR: keep 86.6% language skills with 2.5% text","KL penalty on expert routing saves text skills in MoE","Modality-aware soft routing preserves language ability","SMAR: routing regularizer keeps language intact with 2.5% text","Expert routing tuned by KL keeps language skills at 86.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that making image tokens and text tokens flow through noticeably different sets of experts is what actually preserves the model's language ability, so controlling that gap through a distance penalty is sufficient—rather than the gap being only a by-product of better training or of benchmark selection.","fun_headline_variants_meta":{"raw":{"variants":["SMAR: keep 86.6% language skills with 2.5% text","KL penalty on expert routing saves text skills in MoE","Modality-aware soft routing preserves language ability","SMAR: routing regularizer keeps language intact with 2.5% text","Expert routing tuned by KL keeps language skills at 86.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3036,"prompt_tokens":880,"completion_tokens":2156,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2063}},"tokens_in":496,"tokens_out":2156,"duration_ms":13586,"temperature":1.0,"reasoning_tokens":2063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T06:01:59.748322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical model with SMAR, but compute the MRD distance on shuffled modality labels so the loss has no true signal about which tokens are visual or textual; if language retention still reaches about 86.6%, the reported preservation is not caused by modality separation.","supporting_citations":[],"review_version":1}