{"id":"64860bd9-70b6-4107-b82d-42f8c11f36c1","arxiv_id":"2608.04454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Phase-aware expert merging based on routing role profiles preserves more MoE-VLM accuracy than global routing aggregation at matched compression ratios.","lead":"The authors propose RoleMerge, a training-free method that compresses mixture-of-experts vision-language models by merging experts that serve similar roles across image, question, and answer phases. It preserves more benchmark accuracy than existing routing-based merging methods at matched expert-retention ratios, with up to 9.6 percent relative improvement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RoleMerge's InfoVQA scores on DeepSeek-VL2-Tiny exceed the Full Model at ρ=.625 and .75 (50.7 and 59.0 vs 49.3) and exactly match it at ρ=.50, indicating a likely evaluation inconsistency that could inflate the reported Avg gains.","rationale":"I read the paper as making an empirical claim: phase-conditioned roles improve merging, and RoleMerge preserves more performance than baselines. The load-bearing condition is that the evaluation numbers are trustworthy. Table 1 violates this condition: a lossy compression method should not beat the full model on a task, and exact equality at ρ=.50 is statistically implausible. This is more directly load-bearing than the reader's phase-boundary concern, because even if phases are perfectly defined, an inconsistent table cannot support the claim. The phase-boundary issue remains a valid secondary concern, but it is about interpretation, not arithmetic. My proposed test settles whether the anomaly is real; if it is, the paper must be revised. Therefore I keep the reader's CONDITIONAL verdict pending that check.","tokens_in":25240,"tokens_out":11296,"duration_ms":128210,"concrete_test":"Re-run the Full and RoleMerge checkpoints on DeepSeek-VL2-Tiny InfoVQA (infovqa_val_lite) with the exact LMMS-Eval harness, and compare per-example predictions. Verify (1) whether the ρ=.50 RoleMerge outputs are byte-identical to Full (copy-paste signature), and (2) whether ρ=.75 can genuinely outscore Full by 9.7 ANLS points. Compute paired bootstrap 95% CIs for the Full-vs-RoleMerge difference at all three retention ratios. If the anomaly disappears, Table 1's Avg values and the 'best at all ratios' claim must be recomputed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: RoleMerge preserves more performance than baselines. That claim rests on Table 1. In DeepSeek-VL2-Tiny, RoleMerge's InfoVQA scores are 49.3, 50.7, and 59.0 at ρ=.50, .625, .75, while the Full Model scores 49.3. A lossy, training-free compression method outperforming the uncompressed reference on the same fixed evaluation examples is implausible, and an exact match at ρ=.50 is a near-certain sign of a score-recording or example-alignment error. If InfoVQA is wrong, the six-task Avg is wrong: at ρ=.75, the 9.7-point InfoVQA surplus contributes 1.6 points to the Avg; lowering that value could reverse the reported best-vs-baseline margin (NAMEx 57.2). The paper provides paired bootstrap intervals for DeepSeek-VL2-Small and Qwen3-VL but not for Tiny, so this anomaly is unquantified. Because the headline conclusions across all three models cite Tiny results, this inconsistency threatens the empirical core of the paper, independent of the phase-boundary conceptual concern.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that training-free expert merging in mixture-of-experts vision-language models should be based on phase-conditioned expert roles rather than globally aggregated routing statistics. The authors propose RoleMerge, which constructs a Routing Role Profile for each expert from phase-normalized routing counts over image-context, question, and answer phases, assesses merge compatibility via an expert–phase information loss, and augments the merge score with a signed collision adjustment to preserve answer-decoding distinctions. Expert and router parameters are then merged with shared within-group weights. Experiments on DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and Qwen3-VL-30B-A3B-Instruct across six benchmarks and three retention ratios report that RoleMerge retains more macro-average performance than several existing methods, with relative improvements up to 9.6%.","tokens_in":25499,"tokens_out":9306,"duration_ms":93527,"significance":"The proposed reformulation is well-motivated and the implementation is careful: matched baselines under a common protocol, paired bootstrap confidence intervals for two of the three models, component ablations, calibration-source/size/composition analyses, and a deterministic algorithm. If the reported results are correct, the paper provides a simple, training-free improvement to MoE-VLM compression and evidence that phase structure is a relevant signal for expert specialization. The main weakness is the lack of released code/data and the fact that one set of reported scores (InfoVQA on DeepSeek-VL2-Tiny) appears internally inconsistent and is not covered by uncertainty quantification; this must be resolved before the empirical claims can be fully credited.","major_comments":[{"comment":"RoleMerge's reported InfoVQA scores on DeepSeek-VL2-Tiny exceed the uncompressed Full Model at ρ=.625 and ρ=.75 (50.7 and 59.0 vs. 49.3) and exactly match it at ρ=.50. Because merging is a lossy, training-free compression, this is implausible and suggests an evaluation inconsistency such as mismatched example lists, a score-recording error, or different decoding settings. The paper provides no paired uncertainty or re-check for Tiny, unlike the Small and Qwen3-VL comparisons, so the anomaly is unquantified. These values contribute to the reported Avg (the 9.7-point surplus at ρ=.75 adds about 1.6 points), and thus the retention percentages and margins over the best baseline depend on them. Please verify the InfoVQA rows, correct or explain them, and add uncertainty estimates for Tiny.","section":"Table 1 (Sec. 5.2)"},{"comment":"The baseline implementations are acknowledged as adapted ports; for REAM, the paper states that the calibration replay does not retain the complete visual sequence and attention state required by REAM's sequential mode. This means the REAM rows in Table 1 may reflect a weakened version of the method, potentially biasing the comparison. Because REAM is included in the main table, please assess the impact of the incomplete visual/attention state on REAM's grouping quality, provide a more faithful port if possible, and discuss how the comparison changes.","section":"Appendix C.3"},{"comment":"The method's justification rests on the deterministic partition of routed tokens into image-context, question, and answer phases, but the paper provides no sensitivity analysis to this phase-boundary definition (e.g., moving the system prompt or interleaved text to a different bucket) and no validation that the buckets correspond to functional inference phases. A control experiment varying the phase boundaries would help establish that the reported gains are due to phase-conditioned roles rather than an artifact of the specific token split.","section":"Appendix B.1 / Sec. 3.2"}],"minor_comments":[{"comment":"The value of ε is never specified; please state the exact value used in the experiments.","section":"Algorithm 1 / Sec. 4"},{"comment":"The calibration-set sizes for DeepSeek-VL2-Small and Qwen3-VL are not stated in the main text (the appendix mentions 64 per source for Small and 128 per source for Tiny; the Qwen3-VL size is unclear). Please make the calibration budgets explicit for all models.","section":"Sec. 5.1 / Appendix C"},{"comment":"ACR is introduced in the main text before it is defined; the definition appears only in Appendix A.3. Please provide a one-sentence definition when it is first mentioned.","section":"Sec. 4.3"},{"comment":"Consider using a cleaner table format, e.g., separating the Full Model row and using distinct symbols for Best/Second-best, to improve readability.","section":"Table 1"},{"comment":"The statement that 'relative improvements of up to 9.6%' is based on one setting; consider reporting the range of relative gains across all model–retention combinations.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The Tiny InfoVQA anomaly is the main obstacle to acceptance. If the authors can provide logs or a corrected table, the paper is likely salvageable, as the qualitative conclusion (RoleMerge best among compressed methods) appears robust to plausible corrections. I would also encourage the editor to ask for code or detailed routing-trace logs to support reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: the phase-conditioned role profiling idea is genuinely new and mostly well-executed, but Table 1 contains an anomaly that undermines the empirical core as printed.\n\nWhat's new: instead of globally aggregating routing statistics, RoleMerge builds per-expert Routing Role Profiles from phase-normalized counts, and merges based on expert-phase information loss plus a signed answer-collision adjustment. That's a real departure from the cited methods, and the 'w/o phase split' ablation shows the phase signal matters. The experimental design is also better than typical for this area: matched baselines under a common protocol, paired bootstrap CIs for two of the three models, calibration-set sensitivity sweeps, component ablations, and honest disclosure that baseline numbers are adapted ports rather than original reproductions. The efficiency and storage measurements are concrete.\n\nThe soft spot is DeepSeek-VL2-Tiny in Table 1. RoleMerge's InfoVQA at rho=.625 and .75 (50.7 and 59.0) exceeds the Full Model's 49.3, and at rho=.50 it exactly equals 49.3. For a lossy, training-free compression method evaluated with greedy decoding on fixed examples, beating the uncompressed model by 9.7 points is implausible, and the exact match looks like a score-recording or example-alignment error. The paper provides no paired uncertainty for Tiny, so this is unquantified. The stress-test note says this could reverse the .75 margin over NAMEx; that's slightly overstated—you'd need RoleMerge's InfoVQA below roughly 40 to actually reverse it, not just back to 49.3—but the cell is still wrong on its face and the table cannot be taken at face value. If InfoVQA is corrected, the Tiny Avg values change and the 'best at all nine settings' claim needs re-auditing. This is the kind of thing that should be verified before publication.\n\nA secondary concern: the phase boundary is a deterministic heuristic (visual positions vs. remaining prompt vs. generated tokens), and there's no sensitivity analysis on that partition. That's addressable and minor-to-moderate.\n\nBottom line: the method is worth engaging and the paper deserves a serious referee, but the Tiny numbers need to be confirmed or corrected, and code/data release would help a lot. If the anomaly is a typo, the rest mostly holds.","headline":"Phase-aware merging is a real idea with careful experiments, but the Tiny InfoVQA cells in Table 1 are implausible and need fixing before the headline numbers can be trusted.","tokens_in":25996,"tokens_out":3890,"would_cite":false,"duration_ms":38095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Expert merging for MoE-VLMs should follow phase-conditioned roles, not global routing aggregation; RoleMerge demonstrates the gains.","keywords":["mixture-of-experts","vision-language models","expert merging","model compression","routing statistics","phase-conditioned roles","training-free compression"],"falsifier":"Re-run the matched comparisons after deliberately shifting the phase boundary—for example, counting the first decoded token as part of the question phase—and check whether RoleMerge's advantage over global aggregation survives; if a small boundary shift erases or reverses the gain, the reported benefit depends on the specific partition rather than on phase-conditioned expert roles.","tokens_in":25061,"feed_emoji":"🧩","tokens_out":9731,"duration_ms":81904,"temperature":0.7,"pith_summary":"Mixture-of-experts vision-language models keep every expert in memory even though each token activates only a few, so shrinking the expert pool is a real deployment gain. This paper argues that the usual ground for choosing which experts to merge—aggregating routing statistics over all tokens—obscures the phase structure of multimodal inference, where image-context, question, and answer tokens route differently and image tokens dominate the counts. RoleMerge instead gives each expert a phase-normalized Routing Role Profile and merges experts with compatible profiles while protecting answer-decoding traffic, all without retraining. Across three models and six benchmarks, it retains more of the full model's macro-average than alternative merging methods at matched expert-retention ratios, with relative gains up to 9.6 percent.","feed_headline":"Phase-aware expert roles beat global routing stats in MoE-VLM merging","feed_subtitle":"RoleMerge pairs experts by phase role, keeping up to 96.2 percent of full accuracy.","key_machinery":"The load-bearing object is the Routing Role Profile (RRP): for each expert, the conditional distribution over the three inference phases, computed from phase-normalized routing counts with a uniform phase prior. It converts raw routing traces into a description of what each expert does during image-context processing, question processing, and answer decoding. The expert–phase information loss—the KL-divergence-based drop in mutual information when two experts are merged—turns role differences into a merge score, and the answer-aware score adds a signed collision term that preserves answer-decoding expert distinctions.","core_discovery":"The central claim is that merge compatibility in MoE-VLMs should be judged by phase-conditioned expert roles, not by globally aggregated routing statistics. RoleMerge encodes each expert's role as a Routing Role Profile (RRP), the conditional distribution over the three inference phases (image-context, question, answer) under a uniform phase prior, and measures the cost of merging two experts as the induced decrease in expert–phase mutual information, $d_{EP,\\ell}(i,j) = P_\\ell(i)D_{KL}(q_{\\ell,i}\\|m_{\\ell,ij}) + P_\\ell(j)D_{KL}(q_{\\ell,j}\\|m_{\\ell,ij})$. A signed collision adjustment built from answer-decoding routing mass delays merges that would collapse distinct experts carrying concentrated answer-generation traffic. Once the grouping is fixed, expert parameters and the corresponding router entries are averaged with the same within-group weights, so the compression is achieved without retraining either the experts or the router.","pith_inferences":["The same phase-conditioned profiles could be repurposed for expert pruning: experts whose routing concentrates in low-token phases such as answer decoding are prime pruning candidates, and phase-conditioned saliency could plausibly beat activation-based saliency.","Because the paper focuses on merge compatibility, combining expert–phase information loss with parameter-conflict-aware fusion (for example subspace- or curvature-based) is a natural next step that directly addresses the acknowledged limitation of parameter interference inside merged groups.","The phase partition is itself a modeling choice; a stronger stress test would vary the boundary between question and answer tokens and check whether the reported gains are stable, since the paper's own calibration audit shows source identity matters as much as source count.","The gains are measured on fixed QA-style benchmarks with greedy decoding; applying RoleMerge to open-ended or multilingual generation, where phase structure is less clean, would require re-examining the phase definitions."],"forward_implications":["At every evaluated expert-retention ratio, RoleMerge retains more of the full model's six-task macro-average than all comparison methods in all nine model–retention settings, with relative gains up to 9.6 percent.","Because RoleMerge merges only routed expert parameters and router output entries, the visual encoder, attention modules, shared parameters, and decoding interface stay fixed, so the compression is training-free.","The ablations show that both phase separation and the correspondence between each expert and its own profile are load-bearing: pooling phases or permuting profiles degrades answer-carrier recoverability and multimodal accuracy.","On the largest model, Qwen3-VL-30B-A3B-Instruct at retention .625, the method cuts routed-expert parameters by 37.5 percent and total bf16 storage by 35.0 percent while keeping 91.9 percent of the full model's macro-average."],"supporting_citations":[{"why":"Establishes the sparse MoE routing mechanism whose expert-selection traces RoleMerge analyzes for phase-conditioned profiles.","marker":"Shazeer et al. 2017"},{"why":"MC-SMoE: routing-policy-based merging that RoleMerge extends with phase normalization, and the source of the frequency-weighted within-group parameter averaging.","marker":"Li et al. 2024c"},{"why":"HC-SMoE baseline using hierarchical clustering for retraining-free expert grouping.","marker":"Chen et al. 2025"},{"why":"Sub-MoE baseline, a strong comparison method across all three evaluated models.","marker":"Li et al. 2026"},{"why":"NAMEx baseline formulating expert fusion as a Nash-bargaining problem.","marker":"Nguyen et al. 2026"},{"why":"DeepSeek-VL2, one of the two MoE-VLM families providing the evaluated models.","marker":"Wu et al. 2024"},{"why":"Qwen3-VL, the second MoE-VLM family, including the largest evaluated model.","marker":"Bai et al. 2025"},{"why":"Documents context-dependent expert routing in MoEs, supporting the premise that global usage statistics are insufficient.","marker":"Xue et al. 2024a"}],"fun_headline_variants":["Phase-aware expert merging preserves up to 96.2% of MoE-VLM accuracy","RoleMerge: judge experts by phase role, not global routing stats","MoE-VLM merging: phase-conditioned roles outperform global aggregation","Expert merging for MoE-VLMs: phase roles beat global routing stats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's correctness rests on the assumption that the deterministic token partition into image-context, question, and answer phases (Appendix B.1) correctly identifies the functional phases of MoE-VLM inference, and that routing traces from the TCS calibration set are representative of routing on the six evaluation benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Phase-aware expert merging preserves up to 96.2% of MoE-VLM accuracy","RoleMerge: judge experts by phase role, not global routing stats","MoE-VLM merging: phase-conditioned roles outperform global aggregation","Expert merging for MoE-VLMs: phase roles beat global routing stats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":2070,"prompt_tokens":1031,"completion_tokens":1039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":957}},"tokens_in":647,"tokens_out":1039,"duration_ms":9480,"temperature":1.0,"reasoning_tokens":957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:29.546503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the matched comparisons after deliberately shifting the phase boundary—for example, counting the first decoded token as part of the question phase—and check whether RoleMerge's advantage over global aggregation survives; if a small boundary shift erases or reverses the gain, the reported benefit depends on the specific partition rather than on phase-conditioned expert roles.","supporting_citations":[],"review_version":1}