{"id":"4e23789a-407a-49be-84f5-71d0b089681d","arxiv_id":"2412.13736","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MedCoT uses a hierarchical chain of large-language-model experts and a sparse mixture-of-experts decoder to improve medical VQA accuracy and interpretability.","lead":"MedCoT is a medical visual question answering method that first has a large language model write a diagnostic rationale, then has a second model check and fix it, and finally uses a small local model with a mixture of experts to read the question, image, and rationale to give the final answer. It reports topping several specialized medical VQA benchmarks with a model a fraction of the size of LLaVA-Med, while also printing the reasoning path.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported accuracy gains may be driven by Gemini memorizing public benchmark answers; no contamination check or no-rationale baseline is provided.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Gemini-generated rationales may be contaminated by benchmark memorization. This is the most fundamental threat to the central claim because it does not merely affect the magnitude of the reported gains—it could eliminate them entirely. If the rationales leak the answer, then the contribution of the hierarchical expert design and sparse MoE is not real visual-language reasoning but a form of answer extraction from the LLM's memory. The paper gives no evidence to rule this out. The ablation and expert-number grid search are informative about the pipeline's internal components, but they do not address the provenance of the rationale's information. The lack of a no-rationale baseline is especially important: without it, one cannot tell whether the 87.50% accuracy on VQA-RAD comes from the Diagnostic Specialist's own multimodal understanding or from a rationale that already contains the answer. A text-only probe with Gemini and a no-rationale Diagnostic Specialist would settle this concern directly. The hyperparameter tuning on test sets is a secondary methodological weakness, but it is less fundamental and can be repaired by using validation splits; the contamination issue, if real, would invalidate the paper's headline. The reader's conditional accept is appropriate: the paper is promising but the evidence is incomplete. No change to the verdict is needed, provided the authors can address the contamination concern with the proposed checks.","tokens_in":12242,"tokens_out":3477,"duration_ms":32248,"concrete_test":"Run a text-only probe: feed only the question and answer options (no image) to Gemini Pro 1.5 with the same Initial/Follow-up prompts, and measure how often the generated rationale implies the correct answer. If this text-only accuracy is substantially above chance (e.g., >70% on VQA-RAD closed questions), benchmark contamination is confirmed. Additionally, train the Diagnostic Specialist without any rationale (question+image only) and compare accuracy; if the no-rationale baseline nearly matches MedCoT, the rationale is not the source of gain; if it is much lower, then the rationale matters and its contamination would invalidate the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MedCoT's hierarchical expert design plus sparse MoE surpasses SOTA Med-VQA models—rests entirely on the accuracy numbers reported in Figure 4 and Table 1. The Initial and Follow-up Specialists are Gemini Pro 1.5, a proprietary model whose pretraining data likely include the public VQA-RAD, SLAKE-EN, Med-VQA-2019, and PathVQA benchmarks. If Gemini has memorized the correct answers, its generated rationales can implicitly encode the ground truth (e.g., 'there is no localized mass'), turning the Diagnostic Specialist into a near-trivial answer extractor. The paper provides no contamination analysis, no evaluation on a private or newly collected dataset, and no no-rationale baseline (Diagnostic Specialist trained without any rationale). Consequently, the measured improvements—5.52% and 4.09% over LLaVA-Med—could reflect benchmark leakage rather than the proposed architecture. The ablation in Table 1 shows that removing Follow-up and MoE reduces accuracy, but it does not isolate whether the gain comes from the architecture or from the information content of the rationale. This concern is load-bearing because if contamination is present, the headline claim about surpassing SOTA on standard benchmarks is an artifact of memorization and would not transfer to unseen clinical data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MedCoT, a hierarchical expert reasoning framework for medical visual question answering. An Initial Specialist (Gemini Pro 1.5) generates a preliminary rationale; a Follow-up Specialist (also Gemini Pro 1.5) validates and refines the rationale and produces an image caption; a locally trained Diagnostic Specialist (Flan-T5 + DETR with sparse MoE and feature-level majority voting) ingests the image, question, rationale, and caption and outputs the final answer. Experiments on VQA-RAD, SLAKE-EN, Med-VQA-2019, and PathVQA report accuracy gains over previous state-of-the-art methods, including a 256M-parameter model outperforming LLaVA-Med (7B) by 5.52% and 4.09% on VQA-RAD and SLAKE-EN. The paper claims both improved accuracy and interpretability.","tokens_in":12425,"tokens_out":4755,"duration_ms":42797,"significance":"The core idea is plausible and practically relevant: using a proprietary LLM as a rationale generator and a small local sparse-MoE model as a diagnostic aggregator is a reasonable way to inject interpretable reasoning into Med-VQA. If the empirical claims hold, the method would be a useful lightweight alternative to large medical VLMs and would demonstrate that sparse MoE voting improves robustness. The paper releases code and presents qualitative examples. However, the central empirical claims are currently undermined by post-hoc test-set hyperparameter selection, the absence of a no-rationale baseline, potential contamination of the public benchmarks by Gemini's pretraining data, and the lack of error bars. These issues are fixable but must be addressed before the state-of-the-art claims can be accepted.","major_comments":[{"comment":"The optimal number of experts (6, 10, 5, 5 for VQA-RAD, SLAKE-EN, Med-2019, and PathVQA) and the optimal k value (2) are selected per dataset on the test sets, as described in the Grid Search paragraph. This is post-hoc selection on the test data, so the reported accuracies in Figure 4 and Table 1 are optimistic and do not estimate generalization honestly. Please select hyperparameters on a validation split (or with nested cross-validation) and report test accuracy for the selected configuration, together with error bars over multiple runs or bootstrapping.","section":"Section 4.3, Grid Search"},{"comment":"The ablation varies only the Follow-up Specialist and the MoE components; it never removes the rationale/caption input entirely. Since the Diagnostic Specialist is trained on Gemini-generated rationales and captions, the 9.93-point gap between the no-MoE/no-Follow-up condition (77.57) and full MedCoT (87.50) on VQA-RAD could be largely due to the additional textual information rather than the Follow-up refinement or the MoE architecture. Add a baseline with no rationale/caption input and a baseline with only the Initial Specialist rationale, so that the contribution of each component is isolated.","section":"Table 1 and Section 4.3, Ablation Study"},{"comment":"Gemini Pro 1.5 is used as the Initial and Follow-up Specialist on four public benchmarks (VQA-RAD, SLAKE-EN, Med-VQA-2019, PathVQA). Because Gemini's pretraining data likely include these public datasets, the generated rationales may implicitly encode ground-truth answers, and the Diagnostic Specialist may be reading the answer out of the rationale rather than performing genuine visual reasoning. The paper provides no contamination analysis, no evaluation on a private or newly collected dataset, and no control such as comparing Gemini rationales on original versus visually perturbed images. This is load-bearing for the generalization claim in the abstract; please add such an analysis or explicitly scope the claims to settings where the rationales are independently verified.","section":"Sections 4.1 and 4.2"},{"comment":"The main results are reported as single runs without error bars or significance tests. For differences of 4-6% against LLaVA-Med, it is possible that confidence intervals overlap. Report multiple random seeds (or bootstrap confidence intervals) and, where feasible, a significance test, so that the claimed improvements can be assessed quantitatively.","section":"Figure 4 and Table 1"}],"minor_comments":[{"comment":"The notation for the MoE feature-level majority vote is under-specified: Wi is defined as softmax(Vtop k)_i over k selected experts, but the dimensions of Vtop k, Ei,Ff, and EFf are not stated. Please define these terms explicitly and clarify how the weighted average operates on feature vectors.","section":"Equations (4)-(6)"},{"comment":"The symbol f is used both for the rationale generator in the optimization objective and later for the fused feature Ff; this collision makes the notation hard to follow. Please rename one of these uses.","section":"Equation (1)"},{"comment":"The captions of Figures 2 and 5 are nearly identical, and the duplicated text should be differentiated. Figure 5's caption says 'context' but the figure shows two Diagnostic Specialist boxes; please clarify the intended message.","section":"Figures 2 and 5"},{"comment":"The dataset name 'Med-VQA-2019' is presumably VQA-Med 2019 from ImageCLEF; please use the standard name and cite the original source consistently throughout.","section":"Section 4.1"},{"comment":"The abstract states that MedCoT 'surpasses existing state-of-the-art approaches' without noting that the main-text comparisons are for closed-end questions, with open-end results in the appendix. Please state this qualification in the abstract or in the main-text summary.","section":"Abstract and Section 4.2"},{"comment":"The limitation section acknowledges LLM hallucination but does not discuss the possibility that the public benchmark rationales are contaminated by Gemini's pretraining data; adding this discussion would strengthen the paper's treatment of external validity.","section":"Limitation section"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution with a plausible architecture, but the validation methodology needs substantial strengthening before the SOTA claims can be accepted. The most serious issues are the test-set hyperparameter selection and the missing no-rationale baseline; the contamination concern, while not provable from the text, is a real risk given the use of Gemini Pro on public benchmarks. I would ask for a revision that adds the missing controls and proper statistical reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MedCoT is a sensible new combination for Med-VQA: Gemini generates chain-of-thought rationales, a follow-up specialist validates and refines them, and a local 256M sparse-MoE decoder produces the final answer. The pieces are not new—CoT prompting, self-reflection, and sparse MoE are all established—but the three-tier expert arrangement is a legitimate contribution, and the paper earns credit for targeting interpretability rather than raw accuracy alone.\n\nThe ablation results are internally consistent with the claims: removing the follow-up specialist drops accuracy by about 6.6 points on VQA-RAD, and removing the MoE drops it by about 4.8 points. The organ-specific expert visualization in Figure 6 is a nice piece of evidence that the sparse MoE specializes. Code is released, which helps reproducibility.\n\nBut the evidence is weaker than the abstract implies. Section 4.3 grid-searches the number of experts and the top-k value per dataset on the test sets; selecting hyperparameters that way inflates accuracy. There are no error bars or significance tests anywhere, so the headline gains over LLaVA-Med could be partly noise. More importantly, the ablation never removes the rationale entirely: the comparison is Initial+Diagnostic versus Follow-up+Diagnostic, so the gain could come from having any extra text, not from the hierarchical expert logic specifically.\n\nThe stress-test concern about Gemini contamination is fair. All four datasets are public, Gemini's pretraining data very likely include them, and the paper provides no no-rationale baseline, no contamination check, and no private or freshly collected evaluation. If Gemini has memorized the answers, the rationale implicitly leaks the label, and the local decoder becomes a near-trivial answer extractor. The paper honestly acknowledges hallucination risk in the Limitation section, but it does not address leakage.\n\nThe central accuracy claim is not contradicted by the paper's own data, but its external validity is unproven. I would not desk-reject this. It deserves a serious referee, who should ask for multiple seeds, a no-rationale baseline, validation-based hyperparameter selection, and a contamination test or private dataset. If the numbers survive those checks, this is a useful contribution to Med-VQA. For now, treat the headline as promising but unverified.","headline":"A useful new combination for Med-VQA—hierarchical expert rationales plus a local sparse-MoE decoder—but the headline accuracy gains are not yet trustworthy because of test-set hyperparameter tuning, missing error bars, and no contamination check on the Gemini-generated rationales.","tokens_in":13030,"tokens_out":2253,"would_cite":false,"duration_ms":22027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedCoT claims a 256M-parameter hierarchical expert chain with sparse-MoE voting beats a 7B medical VQA model by up to 5.5 points.","keywords":["medical visual question answering","chain-of-thought reasoning","hierarchical expert verification","sparse mixture of experts","interpretability","rationale generation","self-reflection","multimodal reasoning"],"falsifier":"Replace the medical image with a blank or semantically unrelated image while keeping the question and options identical; if the Initial and Follow-up Specialists still produce matching rationales and the Diagnostic Specialist still answers correctly, then the chain is relying on textual priors or memorized benchmark content rather than on visual evidence.","tokens_in":11990,"feed_emoji":"🩺","tokens_out":8754,"duration_ms":72237,"temperature":0.7,"pith_summary":"MedCoT is a medical visual question answering method built around a three-tier expert chain: an Initial Specialist proposes a diagnostic rationale, a Follow-up Specialist critiques and corrects that rationale and writes an image caption, and a locally hosted Diagnostic Specialist—a multimodal T5 with a sparse mixture of experts—votes on the final answer. The paper's central claim is that this hierarchy produces both more accurate answers and readable reasoning paths without any manually annotated rationales. On closed questions it reports 87.50% accuracy on VQA-RAD and 87.26% on SLAKE-EN, and with roughly 256M trainable parameters it outperforms the 7B-parameter LLaVA-Med by 5.52% and 4.09% respectively. The significance is that medical diagnosis is modeled as a multi-expert consultation rather than a single-model shortcut, making the reasoning chain inspectable.","feed_headline":"Small model beats 7B medical VQA via expert chain","feed_subtitle":"Three-tier reasoning chain with expert voting raises accuracy and exposes the diagnostic rationale.","key_machinery":"The central mechanism is the hierarchical expert verification chain. The Initial Specialist is a general-purpose multimodal LLM prompted to produce a step-by-step rationale; the Follow-up Specialist uses self-reflection to keep or revise that rationale and to generate a caption that bridges the image-text modality gap; the Diagnostic Specialist is a locally hosted multimodal T5 whose visual features come from DETR and whose text features come from Flan-T5, fused by cross-attention and a sparse top-k Mixture of Experts. The sparse MoE selects k = 2 experts per input, aggregates their outputs with a softmax-weighted feature-level majority vote, and gates the final fusion between textual and attended visual features; empirically the experts auto-specialize, with different experts handling head, chest, and abdomen questions. The chain is what carries the argument: it converts an opaque single-model answer into a traceable rationale that is checked by a second pass and then voted on by specialist modules.","core_discovery":"MedCoT's claim is that an explicit, verifiable reasoning chain is both more accurate and more interpretable than single-shot Med-VQA. The pipeline starts with an Initial Specialist (a proprietary LLM, Gemini Pro 1.5) that proposes a rationale from the image and question; a Follow-up Specialist (the same LLM) classifies the rationale as effective or ineffective, rewrites it if needed, and adds an image caption; a local Diagnostic Specialist—a multimodal T5 with a sparse MoE and feature-level majority vote—then produces the final answer from the image, question, refined rationale, and caption. The paper reports that on closed questions this three-tier chain reaches 87.50% on VQA-RAD and 87.26% on SLAKE-EN, beating the 7B-parameter LLaVA-Med by 5.52 and 4.09 points, and that ablations attribute 6.62 points to the Follow-up Specialist and 4.78 points to the MoE on VQA-RAD.","pith_inferences":["The paper's reported gains do not yet separate the architecture's contribution from the memory of the proprietary LLM; running the pipeline with the image removed or with rephrased questions would settle whether the rationales are truly vision-grounded.","If the result holds, the three-tier draft-verify-decide pattern transfers naturally to other high-stakes multimodal question answering settings such as pathology slides, satellite imagery, or industrial inspection, where a small local model makes the final call and an LLM supplies the rationale.","The appendix case where both specialists agree on a wrong pneumomediastinum diagnosis shows the chain inherits a single-point-of-failure risk in the LLM; a second, vision-only verification signal beyond the Follow-up Specialist might catch errors the current design misses.","Since k=2 was optimal on every dataset while the best expert count varied from 5 to 10, the computational cost is essentially fixed by the number of experts, and expert count is a dataset-level hyperparameter rather than a universal constant."],"forward_implications":["A roughly 256M-parameter local Diagnostic Specialist is enough to outperform a 7B end-to-end medical VQA model on VQA-RAD and SLAKE-EN closed questions, which would cut deployment cost if the result holds.","The Follow-up Specialist's self-reflection contributes 6.62 accuracy points on VQA-RAD (87.50% vs 80.88%), so rationale quality is a first-order accuracy lever rather than a cosmetic addition.","The sparse MoE contributes another 4.78 points on VQA-RAD (87.50% vs 82.72%), and its experts specialize by organ, giving per-case traceability of which expert handled a question.","Human rationale annotation is not required; the chain generates and verifies rationales by prompting an LLM, removing a major bottleneck for interpretable Med-VQA.","On open-ended questions MedCoT reports higher Rouge and BLEU scores than MedThink, indicating the reasoning chain also improves generative answer quality, not only closed-form accuracy."],"supporting_citations":[{"why":"MEVF baseline that MedCoT compares against and whose visual feature setup later CLIP-based methods build on.","marker":"Nguyen et al., 2019"},{"why":"MM-CoT; the two-stage rationale-then-answer multimodal chain-of-thought approach MedCoT extends to medicine.","marker":"Zhang et al., 2023b"},{"why":"LLaVA-Med; the 7B-parameter large vision-language baseline MedCoT claims to beat by 5.52% and 4.09%.","marker":"Li et al., 2024"},{"why":"T5; supplies the TextualEncoder and TextualDecoder for the Diagnostic Specialist.","marker":"Raffel et al., 2020"},{"why":"DETR; supplies the VisualEncoder that turns medical images into visual features.","marker":"Carion et al., 2020"},{"why":"Switch Transformers; the top-k sparse expert routing and load-balancing approach used in the sparse MoE.","marker":"Fedus et al., 2022b"},{"why":"VQA-RAD; one of the four benchmark datasets and the source of the 87.50% headline accuracy.","marker":"Lau et al., 2018"},{"why":"SLAKE-EN; the second benchmark dataset that anchors the cross-dataset accuracy claims.","marker":"Liu et al., 2021"}],"fun_headline_variants":["Expert chain lets small medical VQA outscore 7B","Three-tier reasoning beats single-shot medical VQA","Medical VQA with expert voting is more accurate and explainable","Hierarchical expert review sharpens medical image answers","MedCoT: smaller model wins via multi-expert diagnosis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy gains assume that the rationales produced by the external language model come from genuine visual understanding of each image, not from the model having memorized the benchmark questions and answers.","fun_headline_variants_meta":{"raw":{"variants":["Expert chain lets small medical VQA outscore 7B","Three-tier reasoning beats single-shot medical VQA","Medical VQA with expert voting is more accurate and explainable","Hierarchical expert review sharpens medical image answers","MedCoT: smaller model wins via multi-expert diagnosis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1617,"prompt_tokens":943,"completion_tokens":674,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":594}},"tokens_in":559,"tokens_out":674,"duration_ms":6881,"temperature":1.0,"reasoning_tokens":594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:49:59.106232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the medical image with a blank or semantically unrelated image while keeping the question and options identical; if the Initial and Follow-up Specialists still produce matching rationales and the Diagnostic Specialist still answers correctly, then the chain is relying on textual priors or memorized benchmark content rather than on visual evidence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MEVF baseline that MedCoT compares against and whose visual feature setup later CLIP-based methods build on."}],"review_version":1}