{"id":"caf76e8b-8417-4262-b45c-ee43694cc860","arxiv_id":"2506.08356","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MedMoE inserts report-conditioned mixture-of-experts into a GLoRIA-style medical vision-language model, reporting accuracy gains on several radiology benchmarks.","lead":"MedMoE is a medical vision-language model that adds a mixture-of-experts module, routing image features through expert branches selected by a report-conditioned router. It claims better image-text alignment and zero-shot accuracy across X-ray, MRI, CT, and ultrasound benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never isolates the MoE module: MedMoE changes backbone, pretraining data, and adds report-type supervision at once, so the reported gains do not demonstrate that expert routing helps.","rationale":"The reader identifies the same load-bearing weak spot: the absence of a same-backbone, same-data, no-MoE baseline. My reading of the paper confirms that the comparison set changes multiple variables at once, so the empirical results cannot isolate the contribution of the Mixture-of-Experts module. This is not a minor methodological nit; it directly undermines the only novel architectural claim in the paper. The architecture's dependence on a report embedding during zero-shot classification is also unspecified, and if class prompts carry modality information, the router could be exploiting label leakage rather than learning modality-specialized visual representations. These concerns are consistent with the reader's high correctness risk assessment. The paper does have some independent value as an organized exposition of an MoE-style medical VLM, and the efficiency argument about hard routing is plausible, but no code, no hyperparameter details, and no ablation are provided. Given that the central claim is unsupported by the current evidence, the reader's REJECT verdict stands.","tokens_in":8032,"tokens_out":5051,"duration_ms":59255,"concrete_test":"Run an ablation on the identical UniMed 5.3M pretraining set and Swin-T backbone with K=1 expert, keeping global contrastive, local contrastive, and auxiliary report-type losses and all other hyperparameters identical; evaluate on CheXpert, RSNA, Thyroid, ACL, Meniscus, and MediMeTA axial under the same zero-shot and linear-probing protocols. If K=1 matches or exceeds K=4 on these benchmarks, the MoE routing is not the source of the reported gains. Also report the t_g used in zero-shot classification, specifying whether class prompts contain modality names, so the routing input is unambiguous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that report-conditioned MoE routing produces modality-specialized visual representations and outperforms static multi-scale local contrastive learning. The evidence in Tables 1-2 cannot support that causal attribution. MedMoE is compared against UniMed-CLIP, which uses a ViT-B/16 backbone, while MedMoE uses Swin-T + MoE; against GLoRIA, which uses ResNet50 and CheXpert data. No baseline trains a single shared local feature head (K=1, no router) on the same Swin-T backbone, the same UniMed 5.3M data, and the same global+local+auxiliary losses. Consequently the reported differences (e.g., +6.67 on RSNA and -10.40 on Meniscus vs UniMed-CLIP in Table 1) could be caused by backbone, data, training recipe, or the auxiliary modality classifier, not by MoE routing. The omission is load-bearing because the MoE module is the paper's only novel contribution. A second unresolved ambiguity compounds it: the router is a function of the global report embedding t_g, but zero-shot classification has no report. The paper never states what t_g is in this setting; if the class prompts encode modality names, the router receives modality information at test time and the comparison is no longer a fair test of modality-agnostic inference. Internal mismatches (e.g., §5.2 claims state-of-the-art on all three CT views while Table 1 shows UniMed-CLIP better on sagittal, 31.72 vs 26.83) further weaken confidence in the reported numbers. The central mechanism therefore remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MedMoE, a medical vision-language model that augments a Swin Transformer backbone with a report-conditioned Mixture-of-Experts (MoE) module. The router selects one of several convolutional expert branches based on the global report embedding, and the selected expert fuses multi-scale image features through cross-scale attention. Training uses global and local contrastive losses plus an auxiliary report-type classification loss. The authors report zero-shot classification and linear probing results across X-ray, ultrasound, MRI, and CT benchmarks, claiming improvements over generalist baselines such as UniMed-CLIP and PMC-CLIP, along with qualitative attention visualizations and a computational cost comparison.","tokens_in":8398,"tokens_out":4015,"duration_ms":44399,"significance":"If the central claim were fully supported, the idea of modality-specialized expert routing inside the visual encoder would be a useful contribution to medical vision-language pretraining, where static encoders are the norm. The paper uses publicly available data and benchmarks, compares against several strong baselines, and provides a parameter/FLOP analysis showing that hard routing keeps inference cost close to a single Swin-T backbone. However, the experimental design does not currently isolate the MoE mechanism, and several text/table contradictions undermine confidence in the reported numbers. The contribution is therefore promising but not demonstrated in the present manuscript.","major_comments":[{"comment":"The central claim that report-conditioned MoE routing improves performance is not supported by any ablation that isolates the MoE module. MedMoE differs from UniMed-CLIP in backbone (Swin-T vs ViT-B/16), initialization (MedCLIP-pretrained weights), and training objective (addition of the auxiliary report-type loss). A K=1 baseline without a router, trained on the same Swin-T backbone, the same UniMed 5.3M data, and the same global/local/auxiliary losses, is needed to attribute the reported gains to the MoE mechanism. Without it, the differences in Table 1 (e.g., +6.67 on RSNA, -10.40 on Meniscus relative to UniMed-CLIP) could be caused by any of these changes.","section":"§5.2, Table 1"},{"comment":"The router is conditioned on the global report embedding t_g and trained with an auxiliary loss using the ground-truth report type y, but zero-shot classification (Table 1) is performed without a report. The paper never states what t_g is during zero-shot inference. If the class prompts encode modality names (e.g., 'a CT image of ...'), then the router is receiving modality information at test time, contradicting the abstract's claim that the method works 'without requiring modality-specific supervision at inference.' This ambiguity is load-bearing because it determines whether the zero-shot comparison is a fair test of the proposed mechanism.","section":"§4.3, §4.5, Abstract"},{"comment":"Several empirical claims contradict the reported tables. Section 5.2 states that MedMoE delivers state-of-the-art results on all three CT views, but Table 1 shows UniMed-CLIP achieving 31.72 on sagittal CT versus MedMoE's 26.83. The same paragraph says MedMoE attains the best performance on 6/9 datasets, whereas Table 1 shows best results on 5/9 (CheXpert, RSNA, Breast, axial CT, coronal CT). Section 5.3 also claims that for MRI (ACL) and CT (MediMeTA Axial) MedMoE matches or exceeds all generalist baselines, but Table 2 shows UniMed-CLIP reaching 97.28 on ACL 100% versus MedMoE's 92.84, and a tie on MediMeTA Axial 100% at 76.38. These inconsistencies need to be corrected or explained.","section":"§5.2, Tables 1 and 2"},{"comment":"No error bars, standard deviations, or significance tests are reported for any experiment. Many comparisons in Table 1 are small (e.g., 66.03 vs 65.90 on CheXpert), and without repeated runs it is impossible to assess whether the reported gains are meaningful. At minimum, the authors should report variance over multiple seeds or a statistical test for the main zero-shot and linear probing comparisons.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The section numbering is duplicated: '3. Method' is immediately followed by '4. Methodology,' and the latter contains the actual method description. The duplicate header should be removed and sections renumbered.","section":"§3/§4"},{"comment":"The Figure 2 caption states that the router selects an expert 'based on the global image embedding,' while §4.3 defines the router as conditioned on the global report embedding t_g. These are different inputs; the inconsistency should be resolved.","section":"Figure 2 and §4.3"},{"comment":"Reference [4] is cited for 'MedCLIP-pretrained weights,' but the reference is the ImageNet paper by Deng et al. The citation is incorrect or the provenance of the initialization weights needs a proper source.","section":"§5.1, References"},{"comment":"The sentence 'outperforming the strongest generalist baseline (PMC-CLIP) by 3.33% and 0.56%, respectively' is unclear: PMC-CLIP is not the strongest generalist baseline in Table 1 (UniMed-CLIP has higher average accuracy), and the two percentages are not tied to specific datasets.","section":"§5.2"},{"comment":"The column header 'Avg.CheXpert(5x200)' appears to be a formatting error, merging the average column with the CheXpert column. The table would be easier to read with separate 'Avg.' and 'CheXpert' headers.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is in scope for a computer vision journal and the proposed idea is reasonable, but the experimental section needs substantial work before it can be accepted. The absence of a K=1 ablation is the most serious issue; the router/test-time ambiguity and the text/table contradictions can be fixed in revision. I would be willing to review a revised version if the authors add the missing ablation and clarify the inference-time routing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is worth a paper, but this version doesn't demonstrate it. The MoE module is a reasonable extension of GLoRIA, and the cross-scale attention is a nice detail, but the experiments are set up so the MoE effect can't be separated from a better backbone and larger pretraining data. Also, the text overclaims some results, and the router's input at zero-shot time is unspecified.\n\nWhat's genuinely new: applying report-conditioned MoE to the local feature extractor of a medical VLM, with hard routing to a single expert at inference. The computational cost table (Table 3) is honest about the overhead. The paper also collects a nice set of zero-shot and linear probing benchmarks across four modalities.\n\nThe soft spots are load-bearing. The central claim is that the MoE routing itself improves alignment, but there is no baseline with the same Swin-T backbone, the same UniMed 5.3M data, and a single shared local head. MedMoE changes backbone, data, and adds an auxiliary classifier at the same time, so the +6.67 on RSNA and -10.40 on Meniscus could come from any of those. Without a K=1 no-router control on the same training setup, the paper doesn't show what it claims.\n\nSecond, the router is a function of the global report embedding t_g, but zero-shot classification has no report. Unless the authors state what they feed the router in that setting, the comparison may be leaking modality information through the text prompts. That is a straightforward question, but it matters.\n\nThird, the text says state-of-the-art on all three CT views, but Table 1 shows UniMed-CLIP better on sagittal (31.72 vs 26.83). That kind of mismatch makes the reader double-check every other number. No error bars or significance tests on any of the results also doesn't help.\n\nWho this is for: people working on medical vision-language pretraining, especially those interested in MoE or modality-adaptive encoders. A serious referee could help the authors redo the experiments properly, so yes, I'd send it to review, but I'd expect major revision and a real ablation. I wouldn't cite it in its current form.","headline":"A useful architectural idea undermined by an experimental design that never isolates the MoE module.","tokens_in":8866,"tokens_out":2386,"would_cite":false,"duration_ms":25659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Report-conditioned MoE routing improves medical image-text alignment.","keywords":["Medical vision-language models","Mixture-of-Experts","Modality specialization","Global-local contrastive learning","Zero-shot classification","Medical image retrieval","Swin Transformer","Multi-scale features"],"falsifier":"Train MedMoE with the MoE module ablated—replacing the expert branches with a single shared head that averages the same multi-scale features—while keeping the Swin Transformer, UniMed data, and all losses identical; if zero-shot accuracy on RSNA, ACL, and CT benchmarks does not drop by a substantial margin, the central claim that routing specializes representations is not supported.","tokens_in":7885,"feed_emoji":"🩻","tokens_out":4617,"duration_ms":49274,"temperature":0.7,"pith_summary":"MedMoE asks whether medical vision-language models should treat every imaging modality the same way when aligning images with reports. The paper's claim is that they should not: a mixture-of-experts module, routed by the diagnostic report, lets the model specialize its visual attention to the granularity of each modality—local lesions for MRI and CT, global patterns for X-ray and ultrasound. The authors build this into a global-local contrastive learning pipeline, train on the UniMed dataset, and report gains in zero-shot classification and linear probing across nine benchmarks spanning four modalities. If the claim holds, modality-specialized routing is a cheap and effective way to improve medical image-text alignment without extra modality labels at inference.","feed_headline":"Report-conditioned experts sharpen medical image-text alignment","feed_subtitle":"MedMoE routes image features by report type, lifting zero-shot accuracy on 6 of 9 radiology benchmarks.","key_machinery":"The central mechanism is the report-conditioned Mixture-of-Experts module over multi-scale Swin features. Each of K convolutional experts takes the feature pyramid $\\{F^{(1)},\\dots,F^{(L)}\\}$, projects every scale to a shared space, and fuses them with per-location soft attention weights $\\beta_k^{(\\ell)}$; a lightweight MLP maps the global report embedding $t_g$ to a softmax over experts, and hard routing (argmax) selects one expert at inference. The selected local feature grid is then aligned with word-level text embeddings via a GLoRIA-style local contrastive loss, while a cross-entropy auxiliary head on the report embedding supervises expert specialization. This two-level specialization—across scales and across experts—is what carries the argument that modality-adaptive local grounding improves medical vision-language alignment.","core_discovery":"MedMoE's central claim is that inserting a report-conditioned Mixture-of-Experts module into the local visual feature extractor—rather than at the fusion or output stage—yields better image-text alignment than a static, modality-agnostic encoder. Each expert is a small convolutional head that fuses Swin Transformer feature pyramids via cross-scale attention, and a hard router selects exactly one expert per input based on the global report embedding. An auxiliary loss over the report type nudges experts toward diagnostic modalities, so at inference the model needs only the report text, not a modality label, to route. The paper reports that this design improves or matches generalist and specialist baselines on most of nine zero-shot benchmarks and several linear-probing settings, with attention visualizations showing experts concentrating on modality-appropriate anatomy.","pith_inferences":["A direct ablation that disables the MoE while keeping the same Swin backbone and UniMed pretraining data would make the mechanism's contribution measurable; the paper's tables compare against different backbones and datasets.","Report-conditioned routing could be extended to new modalities such as fundus or pathology by appending new expert branches and training only those branches, keeping inference cost flat.","Because only one expert is active per input, MedMoE naturally supports adding modalities at runtime without scaling inference, and may enable continual learning by adding experts for new modalities."],"forward_implications":["Zero-shot classification improves on 6 of 9 radiology benchmarks, including CT views where prior generalist models were weakest.","Hard routing keeps inference FLOPs near a single Swin-Tiny branch (7.8 GFLOPs, ~37M parameters) despite training many experts, so specialization does not cost runtime.","The router needs only the report text at inference, not a modality label, meaning the model can specialize without extra annotation.","Freezing MedMoE's encoder and fitting a linear head transfers well in low-data regimes, beating prior state of the art on RSNA at 1% training data.","Attention maps show each expert attends to modality-relevant anatomy, supporting the claim that routing changes spatial grounding rather than just scale."],"supporting_citations":[{"why":"Supplies the global-local contrastive objectives and the base paradigm MedMoE extends with MoE routing.","marker":"[6]"},{"why":"Provides the UniMed pretraining dataset and the strongest generalist baseline, UniMed-CLIP, that MedMoE compares against.","marker":"[9]"},{"why":"Defines the Swin Transformer backbone whose hierarchical multi-scale feature maps feed the expert branches.","marker":"[14]"},{"why":"Serves as the global-only baseline with a static SWIN encoder, against which the benefit of specialized routing is measured.","marker":"[22]"}],"fun_headline_variants":["MedMoE: Report-conditioned experts align medical images and text","Modality-specialist experts lift zero-shot medical image-text retrieval","Report-type routing selects experts for sharper medical vision-language","MedMoE's report-driven MoE improves multimodal medical alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim is that the reported accuracy gains come from the report-conditioned MoE routing, but no experiment removes the MoE while keeping the same Swin backbone and the same UniMed pretraining data, so the gains could in principle come from the backbone or the data.","fun_headline_variants_meta":{"raw":{"variants":["MedMoE: Report-conditioned experts align medical images and text","Modality-specialist experts lift zero-shot medical image-text retrieval","Report-type routing selects experts for sharper medical vision-language","MedMoE's report-driven MoE improves multimodal medical alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1297,"prompt_tokens":877,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":493,"tokens_out":420,"duration_ms":5304,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:13:06.248871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MedMoE with the MoE module ablated—replacing the expert branches with a single shared head that averages the same multi-scale features—while keeping the Swin Transformer, UniMed data, and all losses identical; if zero-shot accuracy on RSNA, ACL, and CT benchmarks does not drop by a substantial margin, the central claim that routing specializes representations is not supported.","supporting_citations":[{"cited_title":"Lungren, and Serena Yeung","cited_arxiv_id":null,"evidence_quote":"Supplies the global-local contrastive objectives and the base paradigm MedMoE extends with MoE routing."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Defines the Swin Transformer backbone whose hierarchical multi-scale feature maps feed the expert branches."},{"cited_title":"Medclip: Contrastive learning from unpaired medical images and text","cited_arxiv_id":null,"evidence_quote":"Serves as the global-only baseline with a static SWIN encoder, against which the benefit of specialized routing is measured."}],"review_version":1}