{"id":"8285bd3b-5cb0-4f7c-9495-cc8cfe407591","arxiv_id":"2505.24541","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Splitting the vision encoder of an MLLM into domain-specific experts with a lightweight router yields small benchmark improvements at near-zero extra inference cost.","lead":"This paper proposes Mixpert, a way to split a multimodal model's image encoder into one shared part and several specialized experts (for charts, documents, math, OCR, and general images), with a small router that picks the best expert per image. The reported gains are modest but the architecture adds little extra compute, which could make domain specialization cheaper for multimodal systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Router validation and tau selection use the same test benchmarks as final evaluation, so the reported gains may reflect benchmark-specific routing rather than a general solution.","rationale":"I agree with the reader that the weakest point is the router: its accuracy is only measured on validation data that overlaps the final evaluation benchmarks, and the end-to-end effect of routing errors is never directly measured. I would emphasize the evaluation-protocol contamination as the load-bearing issue: Section 4.1 explicitly samples router validation images from ChartQA, DocVQA, MathVerse, MathVista, IIIT5K/HME100K, and COCO Caption, which are the same benchmarks used for final results. Table 5 then selects tau by sweeping on those benchmarks. This is not an internal contradiction, but it is a correctness risk: the favorable numbers in Tables 3, 4, 5, and 8 may be inflated by distribution overlap, and the paper provides no evidence that the router would sustain similar accuracy on truly novel or mixed-domain inputs. The versatile expert does not rescue confident misclassifications, so the central claim of seamless integration into arbitrary MLLMs and arbitrary visual domains rests on an unproven transfer assumption. The concrete test I propose isolates router generalization from expert quality by comparing learned routing against an oracle router on a held-out domain. The reader's verdict of CONDITIONAL already captures this uncertainty, so I do not recommend changing the verdict; the paper should be revised to provide cleaner router evaluation and a held-out routing test.","tokens_in":15130,"tokens_out":6740,"duration_ms":85003,"concrete_test":"Fix the full Mixpert pipeline (experts and router) as trained in the paper, but set tau by tuning only on a held-out validation set that is disjoint from every final evaluation benchmark and from the router training set. Then evaluate on a held-out set of images that includes at least one domain not in the five training categories (or controlled mixtures of two domains), with ground-truth labels. Compare three conditions: baseline LLaVA-OV, Mixpert with the fixed tau, and Mixpert with an oracle router. If Mixpert with learned routing does not exceed the baseline on this held-out set, while oracle routing does, then the router is the bottleneck and the central claim of general applicability is not established. The same check should be run for at least one open-weights model (e.g., Qwen2-VL) with a held-out domain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Mixpert generalizes across arbitrary visual domains with minimal overhead. The support for this claim depends on the router choosing the right expert at inference. Section 4.1 constructs the router validation set by sampling 5K images from ChartQA test/val, DocVQA test, MathVerse/MathVista test, IIIT5K/HME100K test, and COCO Caption test. These are the same benchmarks (or close variants) used for final evaluation in Table 8. The threshold tau is then selected in Table 5 by sweeping on ChartQA test, DocVQA test, OCRBench test, MME test, and MathVista test. This creates a selection leak: both the router's reported 89-95% accuracy and the chosen tau are optimized for the evaluation distribution. The paper never measures the end-to-end effect of routing errors on samples outside this distribution. The versatile expert only catches low-confidence cases; confidently wrong routes go to a specialized expert that was fine-tuned on a different domain and may be worse than the joint-SFT baseline. If routing accuracy drops on truly novel or mixed-domain images, the claimed gains could reverse. Table 7 rules out the 'extra fine-tuning' alternative explanation, but it does not address this routing-generalization and evaluation-protocol issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mixpert, a mixture-of-vision-experts architecture for multimodal LLMs. The vision encoder is split into a frozen shared shallow component and a set of domain experts (chart, document, math, OCR, general) plus a versatile fallback expert, initialized from the joint-SFT checkpoint. A lightweight two-layer MLP router, operating on global-average-pooled features from the shared component, selects the expert at inference using a score-difference threshold. Experts are fine-tuned on domain-specific data while the shared component and LLM remain frozen. Experiments on LLaVA-OV, InternVL2-8B, and Qwen2-VL-7B report consistent gains on ChartQA, DocVQA, OCRBench, MME, MathVerse, MathVista, and MMBench, with minimal additional activated parameters and FLOPs. The paper also includes ablations on the number of MoE layers, router training data size, routing strategies, and the routing threshold.","tokens_in":15366,"tokens_out":4488,"duration_ms":53793,"significance":"If the results hold, Mixpert provides a practical way to mitigate domain conflicts in MLLMs at near-single-encoder inference cost, which would be a useful contribution. The paper gives a clear motivation experiment (Fig. 2), a thorough efficiency analysis (Table 6), and ablations for layer placement, routing strategy, and threshold. It also attempts a control for extra fine-tuning (Table 7). However, the central evaluation protocol has a selection leak: the router validation set and the threshold sweep use the same test benchmarks that later measure the headline gains. This makes the reported improvements difficult to interpret as evidence of general routing capability. The open-weights comparisons additionally introduce 5.5M training samples not available to the baselines, weakening the integration claims. These issues are fixable with re-designed evaluation and controls, but they are load-bearing for the paper's main claims.","major_comments":[{"comment":"The router validation set is sampled from the final evaluation test distributions. Specifically, Sec 4.1 constructs the 25K router validation set from ChartQA test/val, DocVQA test, MathVerse/MathVista test, IIIT5K/HME100K test, and COCO Caption test; Table 8 then evaluates on ChartQA test, DocVQA test, MathVerse mini-vision, and MathVista testmini. Moreover, the routing threshold tau=0.6 in Table 5 is selected by sweeping on ChartQA test, DocVQA test, OCRBench test, MME test, and MathVista testmini. This means the reported router accuracy (~89–95%) and the chosen threshold are fitted to the evaluation distribution. The paper should hold out the final benchmarks entirely from router validation and threshold selection, for example by using a leave-one-benchmark-out procedure or by training the router only on the training corpora and selecting tau on a separate validation split that is disjoint from the test sets used in Table 8.","section":"Sec 4.1, Tables 4–5, Table 8"},{"comment":"The end-to-end effect of routing errors is never measured. Table 3 reports router classification accuracy on the 25K validation set, but this accuracy is not linked to final MLLM performance. The paper assumes that the versatile expert handles low-confidence and misrouted images, but no experiment quantifies what happens when the router misroutes an image to a specialized expert. I ask for an oracle-routing versus router-routing comparison on the final benchmarks, and ideally an explicit corruption experiment (e.g., forcing random or systematically wrong expert assignments) to show the robustness of the fallback mechanism. Without this, the reported gains could be driven by the router's benchmark-specific behavior rather than a general solution.","section":"Sec 3.3, Table 3, Sec 4.3"},{"comment":"The open-weights comparisons are not controlled for the extra training data. For InternVL2-8B and Qwen2-VL-7B, Mixpert is trained on an additional 5.5M collected samples plus 3M router training images, while the base models are evaluated without these data. Table 7 provides a fine-tuned control for LLaVA-OV-7B and InternVL2-8B, but no corresponding control is shown for Qwen2-VL-7B. Even for InternVL2-8B, the control result is reported only in Table 7 and, as presented, does not establish that the improvement in Table 8 is due to the MoE routing rather than the extra fine-tuning data. To support the claim that Mixpert 'integrates seamlessly into any MLLM,' the authors should report a same-data, single-expert fine-tuned control for each backbone, with identical training budgets and data mixtures.","section":"Sec 4.4, Table 8, Table 7"},{"comment":"The layer-selection ablation uses manual expert assignment based on the test benchmarks: the caption of Table 2 states that 'based on the characteristics of testing samples, we manually selected the specific expert for each evaluation benchmark.' This introduces test-set information into the architecture choice (number of MoE layers and whether to include the projector). Since the chosen configuration (projector + last two ViT layers) is derived from this ablation, the reported efficiency/performance trade-off is partly tuned on the final test sets. The authors should either use a held-out validation set for this selection or clearly acknowledge that the layer choice is optimized on the evaluation benchmarks and show that the conclusion is stable under this selection.","section":"Table 2, Sec 4.3"}],"minor_comments":[{"comment":"The phrase 'some navie routing strategies' contains a typo; it should be 'naive routing strategies.'","section":"Sec 4.3"},{"comment":"The sentence 'during inference phrase' should read 'during inference phase.'","section":"Sec 4.3, 'Additional Cost' paragraph"},{"comment":"The dataset name 'IIITK5K' appears in Sec 4.1, but the reference [42] is titled 'Scene Text Recognition Using Higher Order Language Priors' for the IIIT5K dataset; the paper uses 'IIITK5K' inconsistently.","section":"Sec 4.1"},{"comment":"The routing equations use the notation s_d for the score difference but do not define s^(1) and s^(2) explicitly as the highest and second-highest softmax scores; please add a brief definition for clarity.","section":"Sec 3.3, Eq. (1)–(2)"},{"comment":"The dataset names in Table 1, such as 'K12Printing,' 'MA VIS,' and 'RoBUT WTQ,' are not self-explanatory; consider adding a footnote or reference for each unusual dataset name.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about test-set leakage in router validation and threshold selection is well-founded and lands directly on the paper's main claim. The reported improvements may reflect benchmark-specific routing rather than a general solution. The missing Qwen2-VL fine-tuned control is another substantive problem. These issues are fixable by re-running the routing evaluation with properly held-out benchmarks and by adding same-data control models, so I recommend major revision rather than rejection. I would also encourage the authors to release code and checkpoints to enable reproducibility of the routing experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Give this one a serious look. Mixpert is a clean, pragmatic idea: keep the shallow layers of a vision encoder shared, turn the deep layers and projector into domain experts (chart, doc, math, OCR, general) plus a versatile fallback, and route each image with a cheap two-layer MLP. The motivation section makes its point well: after joint SFT, further task-specific SFT helps each domain, while SFT from scratch hurts, which demonstrates real conflicts and motivates the both/and design.\n\nThe paper does several things right. The efficiency accounting is honest: only 237M additional parameters total, but only 1.3M extra activated parameters and negligible FLOPs at inference. Table 7 is a useful control: fine-tuning the same layers on all data without routing does not reproduce the gains, so routing is the active ingredient. The layer-scanning ablation is informative, and the writing is easy to follow.\n\nThe main soft spot is the evaluation protocol. The router validation set is sampled from the same test benchmarks used for final evaluation (ChartQA test/val, DocVQA test, MathVerse/MathVista test, IIIT5K/HME100K test, COCO Caption test), and the routing threshold tau is tuned on those test sets in Table 5. So both the reported router accuracy and the chosen tau are fitted to the evaluation distribution. The paper never measures end-to-end performance on images outside that distribution. This is not a fatal flaw—tau is fairly insensitive between 0.5 and 0.7—but it weakens the generalizability claim. The fix is straightforward: hold out a validation set that is disjoint from all evaluation benchmarks, fix tau on that, and add a few benchmarks the router never saw.\n\nA second issue is the open-weights comparison: Mixpert on InternVL2 and Qwen2-VL is trained on 5.5M extra public samples, while the baselines are left as-is. That makes the comparison unbalanced and should be disclosed more prominently or controlled for.\n\nThe gains are real but modest on several benchmarks (0.1–1.6 points), larger on OCRBench and MME. The practical significance is believable, though not dramatic.\n\nOverall, this is a legitimate addition to the MoE-for-vision line, and the paper is honest enough to be fixable. I'd send it to peer review with a request for major revision on the evaluation protocol, and I'd want code and data splits released. The stress-test concern from the reader lands; the central idea still holds up as a plausible mechanism.","headline":"Solid idea with a clean architecture, but the evaluation protocol leaks test-set signal into router tuning; worth a serious referee with major revisions.","tokens_in":15926,"tokens_out":4074,"would_cite":true,"duration_ms":46556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One vision encoder, split into routed experts, beats multi-task training","keywords":["multimodal large language models","mixture-of-experts","vision encoder","domain conflict","dynamic routing","visual instruction tuning","efficient fine-tuning"],"falsifier":"Take a collection of images that mixes the five domains in unseen proportions, including hybrid images such as a math chart or a scanned document with handwriting that the router is likely to find ambiguous, and compare Mixpert's end-to-end benchmark accuracy against the single-encoder baseline at the same compute budget. If the images whose top-two router scores differ by less than the threshold perform no better than random expert selection, or if accuracy on correctly routed images does not exceed the baseline, then the central claim that routing delivers the gains is falsified.","tokens_in":14881,"feed_emoji":"🖼️","tokens_out":5217,"duration_ms":60980,"temperature":0.7,"pith_summary":"The paper tries to establish that the performance ceiling of a multimodal LLM on diverse visual tasks comes less from encoder capacity than from how the encoder's parameters are organized. Its proposal, Mixpert, keeps a single jointly fine-tuned vision encoder but restructures the deep layers and projector into multiple domain-specific experts, with a lightweight router sending each image to the most suitable expert. This is claimed to preserve the shared-knowledge benefits of joint training while removing the inter-domain conflicts that hold a single encoder back, and to do so at nearly single-encoder inference cost. On LLaVA-OV, InternVL2, and Qwen2-VL, the paper reports consistent gains on chart, document, OCR, math, and general benchmarks, with router category accuracy around 89–95% on validation samples.","feed_headline":"One vision encoder, split into routed experts, beats multi-task training","feed_subtitle":"Routing each image to the right vision expert lifts chart, document, OCR, and math benchmarks at near-zero extra inference cost.","key_machinery":"The load-bearing object is the mixture-of-vision-experts layer stack: the last two ViT transformer layers and the projector of an existing vision encoder are replicated into multiple domain experts, all fed by the same frozen shallow trunk, with a lightweight two-layer MLP router that sees global-average-pooled trunk features and a score-difference threshold that sends ambiguous images to the versatile expert. This design lets the model keep one shared representation at the bottom, specialize only at the top, and never activate more than one expert plus the router per image.","core_discovery":"Starting from a jointly supervised-fine-tuned MLLM, Mixpert partitions the vision encoder into a frozen shared trunk (the shallow ViT layers) and a mixture of vision experts (the projector plus the last two ViT layers), where each of five experts is trained on one manually defined domain (chart, document, math, OCR, general) and a sixth 'versatile' expert keeps the original joint-SFT weights as a fallback. A two-layer MLP router, fed by average-pooled features from the shared trunk, picks the highest-scoring expert unless the gap between the top two routing scores falls below a threshold, in which case the versatile expert handles the image. The paper's central claim is that this restructured encoder reconciles joint optimization with task specialization: it inherits common knowledge from joint SFT, removes per-domain conflicts through decoupled expert tuning, and adds only about 0.001 G FLOPs of routing overhead at inference. On benchmarks this yields improvements such as +0.6 on ChartQA and DocVQA, +31 on OCRBench, +34 on MME, and gains on MathVista, MathVerse, AI2D, and MMBench over LLaVA-OV-7B, with analogous gains when the same recipe is applied to InternVL2-8B and Qwen2-VL-7B.","pith_inferences":["The paper measures router accuracy only as a classifier on validation samples from the same task families used in training; a natural extension would measure end-to-end answer quality conditional on the routing decision, which would reveal whether the versatile expert fully absorbs misroutes.","The manual five-way taxonomy bundles images into chart, document, math, OCR, and general, but the same architecture could be probed with a coarser or finer partition, or with categories discovered from data clusters instead of human judgment, and the gains might shift accordingly.","Because the router uses a global average-pooled representation of the whole image, very small but decisive visual details such as a single formula or a printed stamp may be underweighted; a patch-level or attention-weighted routing signal is a testable variant the paper does not explore.","If the benefit stems from decoupled expert tuning rather than routing itself, a single encoder fine-tuned jointly on all domains with per-domain low-rank adapters should show a similar pattern, and comparing those two mechanisms would isolate the true source of the gains."],"forward_implications":["Any MLLM with a ViT-style vision encoder and an MLP projector can adopt Mixpert by restructuring only the projector and last two encoder layers, so the recipe transfers without changing the LLM or the training data.","Because each expert is fine-tuned independently on its own domain, data-balance concerns in multi-task SFT disappear: a rare domain can be trained as aggressively as a common one without harming other tasks.","Inference stays close to single-encoder cost because only the router (about 1.3M parameters and 0.001 G FLOPs) is added per image, while total stored parameters grow by about 237M.","Images that mix multiple domain characteristics are handled by the versatile expert rather than forced into a single category, so routing mistakes on ambiguous inputs degrade performance less than direct top-1 routing.","When dynamic resolution splits an image into sub-images, applying the routing decision independently to each sub-image yields further gains, extending the method to high-resolution and multi-image settings."],"supporting_citations":[{"why":"Supplies the LLaVA-OV baseline, its joint SFT checkpoint, and the domain-sorted training corpora used for expert and router training.","marker":"[24]"},{"why":"Introduces the mixture-of-experts principle that Mixpert adapts to the vision encoder.","marker":"[16]"},{"why":"Formalizes hierarchical mixtures of experts, the MoE foundation the routing design builds on.","marker":"[17]"},{"why":"Provides an MoE-in-LLM MLLM design whose placement Mixpert explicitly contrasts with its own vision-encoder placement.","marker":"[13]"},{"why":"Another MoE-in-LLM MLLM design used as a point of comparison for where the mixture is applied.","marker":"[28]"},{"why":"Represents the multiple-independent-encoders approach that Mixpert claims to outperform with lower cost.","marker":"[15]"},{"why":"Represents a multiple-encoder vision pipeline used as a comparison point for visual perception quality.","marker":"[51]"},{"why":"Represents a mixture-of-encoders design in the comparison set for multimodal benchmarks.","marker":"[49]"}],"fun_headline_variants":["One vision encoder, many routed experts: Mixpert fixes multi-task conflicts","Mixpert: split a single ViT into experts, route each image smartly","Routing images to the right vision expert lifts MLLM benchmarks at low cost","From one encoder to many experts: Mixpert's dynamic routing improves MLLMs","Efficient vision expert routing: Mixpert reconciles joint training and specialization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the five hand-defined domain categories capture the real distribution of images and that the router's roughly 89–95% category accuracy on benchmark validation samples carries over to every new image, since routing mistakes are never measured end-to-end and would, if frequent or harmful, erase the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["One vision encoder, many routed experts: Mixpert fixes multi-task conflicts","Mixpert: split a single ViT into experts, route each image smartly","Routing images to the right vision expert lifts MLLM benchmarks at low cost","From one encoder to many experts: Mixpert's dynamic routing improves MLLMs","Efficient vision expert routing: Mixpert reconciles joint training and specialization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2208,"prompt_tokens":1010,"completion_tokens":1198,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":1097}},"tokens_in":626,"tokens_out":1198,"duration_ms":13758,"temperature":1.0,"reasoning_tokens":1097,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:18:46.112683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a collection of images that mixes the five domains in unseen proportions, including hybrid images such as a math chart or a scanned document with handwriting that the router is likely to find ambiguous, and compare Mixpert's end-to-end benchmark accuracy against the single-encoder baseline at the same compute budget. If the images whose top-two router scores differ by less than the threshold perform no better than random expert selection, or if accuracy on correctly routed images does not exceed the baseline, then the central claim that routing delivers the gains is falsified.","supporting_citations":[{"cited_title":"Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991","cited_arxiv_id":null,"evidence_quote":"Introduces the mixture-of-experts principle that Mixpert adapts to the vision encoder."},{"cited_title":"Hierarchical mixtures of experts and the em algorithm.Neural computation, 6(2): 181–214, 1994","cited_arxiv_id":null,"evidence_quote":"Formalizes hierarchical mixtures of experts, the MoE foundation the routing design builds on."}],"review_version":1}