{"id":"a216622e-214d-4706-a523-39809a3f8cfb","arxiv_id":"2507.07902","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A medical multimodal RAG pipeline with rethink-and-rearrange and online search; the claimed SOTA is contradicted by the paper's own PMC-VQA numbers.","lead":"MIRA is a medical question-answering system that adds retrieval from both a fixed database and live web search, plus a self-critique loop, to an image-and-text model. The paper claims new state-of-the-art results, but its own tables show it trailing large models on PMC-VQA and it compares against a narrow set of baselines on a 1,000-image MIMIC-CXR subset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Tables 2 and 3 contradict the abstract's SOTA claim: MIRA trails Qwen2.5-VL-72B and OpenAI o1 on PMC-VQA, and the MIMIC-CXR comparison omits the key RAG baselines.","rationale":"I read the paper in good faith. The framework is described at a high level, the appendix provides hyperparameters and sample outputs, and the release of code is a positive. However, the paper's own results undermine the abstract's claim. The MIMIC-CXR comparison is on a small subset and missing key baselines; the PMC-VQA results directly show MIRA below two baselines. The RL description in Section 3.3 is inconsistent with the cross-entropy-only training described in Section 3.4. These are internal inconsistencies between claims and evidence, not disagreements with external consensus. The reader's data-contamination concern is plausible but secondary: even if no contamination exists, the SOTA claim still fails. Therefore the REJECT verdict is appropriate; a revision that narrows the claims, adds missing baselines, and clarifies the training objective could be reconsidered.","tokens_in":18652,"tokens_out":5567,"duration_ms":49877,"concrete_test":"Run the released MIRA code on the official PMC-VQA test split (1,297 conversational and 971 detailed questions) using the paper's evaluation protocol and compare exact-match accuracy to the Table 3 baselines. If the reproduced scores are <=0.66/0.39, MIRA does not surpass Qwen2.5-VL-72B (0.69/0.47) or OpenAI o1 (0.69/0.49), and the abstract's 'new state-of-the-art' claim is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MIRA 'achiev[es] new state-of-the-art results' (Abstract). This is contradicted by the paper's own evaluation. In Table 3, MIRA obtains correctness scores of 0.66 (conversational) and 0.39 (detailed) on PMC-VQA, while Qwen2.5-VL-72B scores 0.69/0.47 and OpenAI o1 scores 0.69/0.49. Section 5.5 acknowledges that MIRA 'trails Qwen2.5-VL-72B and OpenAI o1,' yet the abstract and contributions section repeat the unqualified SOTA claim. Table 2 (MIMIC-CXR) is run on a 1,000-sample subset and omits the two most relevant RAG baselines (MMed-RAG, RULE); the included baselines are non-RAG models or zero-shot variants, so the comparison cannot support a SOTA claim among medical RAG systems. Additionally, Section 3.3 introduces a policy-gradient RL objective (Eqs. 7-8) and a reward function, but Section 3.4 states 'All training stages are using cross-entropy loss' (Eq. 9); the released code would need to contain an RL loop for this description to be accurate. The load-bearing issue is that the empirical evidence in the manuscript falsifies the headline performance claim, independent of any data-contamination concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIRA, a multimodal retrieval-augmented generation framework for medical VQA and report generation. MIRA combines a LLaVA-style vision-language backbone with offline and online retrieval, a query-rewrite module, and a 'Rethink-Rearrange' (RTRA) module that iteratively selects and refines retrieved evidence through chain-of-thought reasoning. The manuscript claims new state-of-the-art results on medical benchmarks, a 9x inference speedup over 72B models, and the first multimodal RAG framework for MedVQA. Evaluations are reported on MIMIC-CXR report generation and PMC-VQA, with ablations on retrieval components and a preference study.","tokens_in":18902,"tokens_out":3300,"duration_ms":36471,"significance":"If the claims were supported, MIRA would be a practically useful contribution: dynamic retrieval selection and iterative refinement are reasonable mechanisms for grounding medical generation, and the paper releases code, which aids reproducibility. The paper also provides ablations (Table 5) showing the value of both online and offline retrieval pathways. However, the central performance claim is contradicted by the paper's own results, and the evaluation design contains a training/evaluation source-overlap risk. The contribution is therefore not currently established at the level required for the claimed state-of-the-art status.","major_comments":[{"comment":"The abstract and Section 1 claim 'new state-of-the-art results,' but Table 3 shows MIRA trailing both Qwen2.5-VL-72B and OpenAI o1 on both PMC-VQA splits: 0.66 vs 0.69 conversational and 0.39 vs 0.47/0.49 detailed. Section 5.5 acknowledges this gap, so the unqualified SOTA claim is unsupported by the manuscript's own evidence. The claim should either be removed or restricted to same-scale models.","section":"Abstract / Table 3"},{"comment":"The MIMIC-CXR comparison is run on a 1,000-sample subset and omits MMed-RAG and RULE, the two most relevant RAG baselines cited in Section 2. The included baselines are non-RAG or zero-shot models, so Table 2 cannot support a state-of-the-art claim among medical RAG systems. The statement in Section 5.4 that MMed-RAG was excluded due to unavailable code does not justify claiming superiority; at minimum, results from published papers should be reported for these baselines or the SOTA claim should be withdrawn.","section":"Section 5.4 / Table 2"},{"comment":"The fine-tuning set described in Section 3.3 includes 50k instances curated from PubMedVision and VQA-Med, while the main evaluation in Section 5.5 uses PMC-VQA, which is built from the same source. The statement that MIRA had 'no pretraining on PMC-VQA' does not rule out overlap or near-duplicates between the training data and the test set. The paper needs an explicit overlap analysis or an evaluation on a disjoint held-out set; without this, the reported PMC-VQA numbers may reflect memorization rather than generalization.","section":"Section 3.3 / Section 5.5"},{"comment":"Equations (7) and (8) define a reward function and a policy-gradient update for 'Reinforced CoT Generation,' but Section 3.4 states that 'All training stages are using cross-entropy loss' and gives Eq. (9) as the only training objective. This is an internal contradiction. The authors must clarify whether RL is actually used; if so, the RL objective and training details must be reported, and if not, Eqs. (7)-(8) should be removed.","section":"Section 3.3 / Section 3.4"},{"comment":"The paper repeatedly claims a '9x faster inference than 72B models' advantage, but no latency, throughput, hardware, or measurement protocol is provided anywhere in the manuscript. As a quantitative efficiency claim, it needs a concrete experimental setup and results; otherwise it should be removed from the abstract and contributions.","section":"Section 1 / Section 5.5"}],"minor_comments":[{"comment":"Section 5.2 says results are reported as mean and standard deviation over three random seeds, but Tables 2, 3, and 5 contain only point estimates. Either include the standard deviations or remove the promise.","section":"Section 5.2"},{"comment":"Section 3.1 describes the vision encoder as CLIP-ViT-Large, but Table 6 lists vision_tower as google/siglip-so400m-patch14-384. Since the actual configuration determines the model, this discrepancy should be resolved in the main text.","section":"Section 3.1 / Appendix Table 6"},{"comment":"Section 5.3 says exact match accuracy is the principal metric for PMC-VQA, but Table 3 reports 'correctness' scores without defining how they differ from exact match. The metric should be defined precisely.","section":"Section 5.3"},{"comment":"There are several typographical and phrasing issues, including 'Evaulation Metrics' in Section 5.3, 'iamges' in the Figure 3 caption, and repeated references [31] and [32] to the same MMed-RAG paper, which should be consolidated.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript's headline performance claim is contradicted by its own Table 3, and the evaluation has a training/evaluation source-overlap issue in the PMC-VQA experiments. The MIMIC-CXR comparison also omits the most relevant RAG baselines. These issues affect the core contribution and cannot be fixed with local edits; the claims need substantial re-scoping and re-evaluation. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The framework itself is sensible: query rewriting, offline plus online retrieval, a rethink-rearrange loop for dynamic k-selection, and attention-based modality fusion. The ablations show that both retrieval pathways matter, and releasing code is a step in the right direction. If I needed a medical RAG baseline, I'd look at this after a cleanup.\n\nBut the paper's central claims overreach. The abstract says \"new state-of-the-art results,\" yet Table 3 shows MIRA trailing Qwen2.5-VL-72B and OpenAI o1 on PMC-VQA. The MIMIC-CXR comparison is on a 1,000-sample subset and omits MMed-RAG and RULE, the two RAG baselines that matter most; \"code unavailable\" does not justify ignoring the closest comparisons. The \"first multimodal RAG framework\" line is undercut by the cited MMed-RAG and RULE. The RL description is internally inconsistent: Section 3.3 gives policy-gradient equations and a reward function, but Section 3.4 says all training uses cross-entropy loss. The PMC-VQA evaluation is contaminated in principle because the fine-tuning set is curated from PubMedVision, the same source as the test set; the paper only says there was \"no pretraining on PMC-VQA,\" which is not the same as no overlap. And the \"9x faster\" claim has no measurement in the paper.\n\nThese are load-bearing problems for the empirical story. The engineering is not the issue; the claims and evaluation are. A revision that narrows the claims, adds the missing baselines, reports standard deviations, measures latency, and clearly separates training and test data would be worth reconsidering.\n\nI would send this to peer review rather than desk reject, because the pipeline is real and the code is public. But I would expect major revisions, and I would not accept the SOTA claim until the evidence supports it.","headline":"A real engineering pipeline whose headline claims are contradicted by its own tables; worth engaging on revision, not as it stands.","tokens_in":19530,"tokens_out":2963,"would_cite":false,"duration_ms":33890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MIRA is a retrieval-augmented generation framework that grounds medical answers in retrieved images and text, using a rethink-rearrange loop to cut hallucination and factual error.","keywords":["retrieval-augmented generation","medical visual question answering","multimodal large language models","hallucination mitigation","chain-of-thought reasoning","radiology report generation","factual accuracy","dynamic retrieval"],"falsifier":"Check for near-duplicate questions or images between the 50k PubMedVision/VQA-Med fine-tuning instances described in Section 3.3 and the PMC-VQA test set, then rerun inference on the non-overlapping subset; if the 0.66 and 0.39 scores drop substantially, the reported generalization is overstated. For the report-generation claim, rerun on a random 1,000-image MIMIC-CXR sample and compare against the available MMed-RAG system, which the paper notes was excluded because its code and weights were unavailable.","tokens_in":18366,"feed_emoji":"🩻","tokens_out":6262,"duration_ms":67153,"temperature":0.7,"pith_summary":"The paper introduces MIRA, a retrieval-augmented generation framework for medical visual question answering built to reduce factual errors and hallucination. Its central claim is that an 8-billion-parameter vision-language model can reach clinically useful accuracy by retrieving both text and images from offline medical databases and live web search, dynamically choosing how many retrieved contexts to include, and running a rethink-rearrange loop in which the model writes an initial answer, critiques it against the retrieved evidence, and rewrites a final answer. The paper reports that this design outperforms earlier medical vision-language models on a 1,000-image MIMIC-CXR report generation subset and approaches the accuracy of 72-billion-parameter frontier models on PMC-VQA while claiming roughly nine times faster inference. The motivation matters because medical question answering can surface fabricated findings or unsupported treatment advice when a model relies only on its internal knowledge.","feed_headline":"Dynamic retrieval plus self-critique lifts medical VQA accuracy","feed_subtitle":"MIRA's 8B model grounds answers in retrieved images and text, staying near 72B systems at nine times the speed.","key_machinery":"The load-bearing mechanism is the Rethink-Rearrange (RTRA) module: a three-stage validation loop in which the model first drafts an answer from retrieved multimodal evidence, then critiques that answer via chain-of-thought reasoning against the evidence, and finally writes a refined response, with a filtering step that drops retrieved chunks judged irrelevant while keeping at least one text and one image reference. Around it sits a multimodal retrieval pipeline that encodes queries and database entries in a shared embedding space, retrieves from both an offline medical database and an online search API, and fuses the two modalities through a learned attention weight so that image-centric and text-centric questions are handled differently. A query rewrite module rewrites user questions before encoding, and a reinforcement-style reward combining a factual score and a coherence score steers the refinement.","core_discovery":"MIRA is a multimodal RAG framework whose central claim is that medical answers should be grounded in retrieved multimodal evidence rather than memory alone, with the amount of evidence adapted per query. The multimodal retrieval path jointly pulls three text chunks, two images, and one online text paragraph into a joint embedding space, then a dynamic attention mechanism weights image and text contributions according to the query. The Rethink-Rearrange module then generates an initial answer, runs a chain-of-thought self-critique against the retrieved material, filters out irrelevant retrieved chunks, and produces a final refined answer. The paper reports that with no pretraining on PMC-VQA, this recipe scores 0.66 on true/false questions and 0.39 on detailed questions on PMC-VQA, outperforming earlier medical vision-language models while trailing two much larger systems, and that it achieves the best scores among the compared models on the MIMIC-CXR report generation subset. The authors describe the results as substantially improving factual accuracy and establishing new state-of-the-art performance.","pith_inferences":["The state-of-the-art claim rests mainly on the MIMIC-CXR subset, since the paper's own PMC-VQA table shows MIRA trailing two much larger models, and the paper notes that the most relevant RAG baseline, MMed-RAG, was excluded because its code and weights were unavailable.","The statement that MIRA had no pretraining on PMC-VQA is weaker than a no-overlap guarantee: the fine-tuning set is curated from PubMedVision, and PMC-VQA is built from the same source, so a leakage check on near-duplicate questions or images would settle whether the reported accuracy measures generalization.","The appendix shows online-RAG snippets containing clearly irrelevant web-page text, such as navigation artifacts, suggesting the online retrieval path can inject noise into the evidence; measuring retrieval precision against end-task accuracy would be a natural extension.","The dynamic-k claim could be made causal by comparing fixed-k retrieval settings against the Rearrange-filtered adaptive k on the same evaluation splits, since the paper's ablations vary the online/offline path and the modality but not the number of retrieved chunks directly."],"forward_implications":["If MIRA works as described, medical VQA systems can reduce hallucination risk by conditioning every answer on retrieved evidence and by deleting irrelevant retrieved chunks before generation.","The online search path means the knowledge base is not frozen, so recommendations can reflect current guidelines, though the quality of retrieved web sources then becomes part of the system's reliability.","The 8-billion-parameter scale with roughly nine times faster inference than 72-billion-parameter models suggests evidence-grounded medical assistants could run at interactive speeds without a frontier-model budget.","The paper's preference analysis, in which MIRA is preferred by an external judge model in 46% of comparisons against one large model and 59.5% against another, indicates the design is competitive in open-ended clinical questions, not just closed-form VQA.","The reported gains on MIMIC-CXR report generation imply that the same retrieval-and-rethink loop can be applied to radiology reporting, not only to short-answer questions."],"supporting_citations":[{"why":"Provides the vision-language base architecture and 500k-image alignment data that MIRA extends with retrieval and rethink-rearrange reasoning.","marker":"[16]"},{"why":"Supplies the PubMedVision benchmark and the HuatuoGPT-Vision baseline that MIRA compares against on PMC-VQA.","marker":"[6]"},{"why":"MMed-RAG is the multimodal RAG baseline the paper positions against, though it was excluded from the MIMIC-CXR comparison for lack of code and weights.","marker":"[31]"},{"why":"RULE is the reliable multimodal RAG baseline whose static-retrieval limitation MIRA's online and adaptive retrieval is meant to address.","marker":"[33]"},{"why":"MIMIC-CXR supplies the radiology report generation data and the 1,000-sample subset used for the main report-generation comparison.","marker":"[14]"},{"why":"XrayGPT serves as a chest-radiology vision-language baseline for both report generation and PMC-VQA comparisons.","marker":"[28]"},{"why":"BiomedGPT provides a generalist biomedical baseline whose scores appear in both the MIMIC-CXR and PMC-VQA tables.","marker":"[35]"},{"why":"Kimi-K1.5 is one of the larger models used in the preference analysis and in the PMC-VQA correctness comparison.","marker":"[27]"}],"fun_headline_variants":["Adaptive retrieval + self-critique sharpens medical AI answers","MIRA: medical AI grounds answers in images and text, adaptively","Self-critical RAG boosts medical VQA accuracy at 9x speed","Dynamic RAG with self-critique yields state-of-the-art medical results","Medical MLLM: adaptive retrieval and self-critique improve facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation-generalization claim depends on the assumption that the PMC-VQA test set is not contaminated by the fine-tuning data drawn from PubMedVision, so the reported accuracy measures out-of-distribution ability rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive retrieval + self-critique sharpens medical AI answers","MIRA: medical AI grounds answers in images and text, adaptively","Self-critical RAG boosts medical VQA accuracy at 9x speed","Dynamic RAG with self-critique yields state-of-the-art medical results","Medical MLLM: adaptive retrieval and self-critique improve facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1437,"prompt_tokens":992,"completion_tokens":445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":608,"tokens_out":445,"duration_ms":4413,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:29:46.367056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check for near-duplicate questions or images between the 50k PubMedVision/VQA-Med fine-tuning instances described in Section 3.3 and the PMC-VQA test set, then rerun inference on the non-overlapping subset; if the 0.66 and 0.39 scores drop substantially, the reported generalization is overstated. For the report-generation claim, rerun on a random 1,000-image MIMIC-CXR sample and compare against the available MMed-RAG system, which the paper notes was excluded because its code and weights were unavailable.","supporting_citations":[],"review_version":1}