{"id":"049c06d2-1a09-4333-b688-37e3a96f3ef8","arxiv_id":"2505.06898","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A fine-tuned 8B medical vision-language model that claims explainable grounding, uncertainty estimates and cancer prognosis, but its core uncertainty formula is mathematically inconsistent and key comparisons use the authors' prior model.","lead":"XMedGPT is a medical AI assistant that combines image, text and audio inputs to produce diagnoses, visual grounding and survival predictions. The paper reports strong benchmark results but contains a flawed mathematical definition of its uncertainty measure and compares against the authors' own earlier model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (5) is algebraically false, so the reported uncertainty AUCs do not evaluate the proposed reliability index.","rationale":"The reader's weakest_assumption correctly isolates Eq. (5) as the critical unsupported step. The entire 'Precise Uncertainty Quantification' section, the abstract's uncertainty AUC claims, and the proposed Reliability Indexing mechanism depend on the H defined by Eqs. (5)-(7). Because Eq. (5) is algebraically false, both interpretations of H_c lead to a quantity that is not the semantic entropy the paper claims to compute. This is an internal inconsistency, so the reported uncertainty results cannot be interpreted as validating the proposed estimator. The prognostic claims based on cohorts of n=49-163 are an additional limitation, but the authors explicitly flag small-cohort external validation in the Discussion; the Eq. (5) defect is not flagged and is more fundamental. The paper has broad benchmarking and a large training corpus as independent evidence for some perception claims, but that does not repair the formal failure of the uncertainty contribution. The reader's REJECT verdict therefore stands, and no verdict adjustment is needed.","tokens_in":42422,"tokens_out":3661,"duration_ms":38414,"concrete_test":"Independently re-derive Eq. (5) with a one-cluster example: M=5 sampled sequences, all semantically equivalent, with cluster-normalized weights {0.50, 0.25, 0.15, 0.06, 0.04}. The left-hand side -P(c) log P(c) equals 0 because P(c)=1, while the right-hand side is the Shannon entropy of those weights, about 1.93 nats. The identity therefore fails. Then recompute the uncertainty AUC on SLAKE and the MIMIC-CXR subset using a corrected, well-defined semantic-entropy estimator; if the resulting AUC differs materially from 0.862 and 0.764, the reliability-index headline is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (5) asserts -P(c|x,q) log P(c|x,q) = -Σ_{s∈c} [P(s|x,q)/P(c|x,q)] log(P(s|x,q)/P(c|x,q)), where P(c|x,q) is defined by Eq. (2) as Σ_{s∈c} P(s|x,q). The right-hand side is the entropy of the normalized within-cluster distribution; the left-hand side is a single term of a cluster-entropy sum. These two quantities are equal only in the degenerate case where one within-cluster probability absorbs all mass. If H_c is read literally as -P(c)log P(c), then Eq. (6) evaluates -Σ P(C_i)^2 log P(C_i), which is not semantic entropy. If H_c is instead read as the within-cluster entropy on the right-hand side, Eq. (6) becomes a weighted average of within-cluster entropies and discards the between-cluster uncertainty, systematically under-estimating uncertainty whenever samples are spread across clusters. Either way, the reliability index whose AUC is reported as 0.862 on VQA and 0.764 on report generation is not the quantity defined in the paper. This is an internal inconsistency in a central contribution, not merely a disagreement with an external consensus. The small-cohort prognostic concern raised by the reader is real but secondary, and it is already acknowledged in the Discussion; the uncertainty estimator failure is the load-bearing problem. Code and data are not yet available, so there is no independent implementation that could rescue the formula.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces XMedGPT, an 8B-parameter multimodal medical assistant built on InternVL-2 and trained on roughly 7 million image-text pairs. It is evaluated across five core tasks (single-label and multi-label diagnosis, VQA, multiple-choice reasoning, and captioning), region-level grounding, uncertainty quantification, audio-input interaction, and survival/recurrence prediction. The paper claims state-of-the-art results over prior GMAI models, an anatomical-grounding IoU of 0.703, uncertainty-estimation AUCs of 0.862 (VQA) and 0.764 (report generation), and a 26.9% improvement over prior leading models in prognosis. A reliability-index mechanism is proposed that computes semantic entropy over consistency-based question-answer checks, and the paper also reports external validation on in-house clinical data and a senior-clinician human evaluation.","tokens_in":42886,"tokens_out":7245,"duration_ms":72328,"significance":"If the contributions were as claimed, XMedGPT would be a substantial advance: an 8B-parameter generalist medical model with visual grounding, sentence-level uncertainty, audio and region-based interaction, and prognostic modeling, validated on public and external clinical data. The paper's strengths are its breadth (40 modalities, 141 anatomical regions, 18 clinical tasks), the explicit guardrail evaluation, and the inclusion of a senior-clinician human evaluation in the collaboration study. However, the central uncertainty estimator is built on an algebraically false identity, and the reported results contain unresolved numerical contradictions across the abstract, the main text, and the tables. As a result, the headline uncertainty, grounding, and prognostic claims are not currently supported. The absence of released code or checkpoints at submission further limits verification of the benchmark numbers.","major_comments":[{"comment":"Equation (5) is algebraically false. With P(c|x,q) defined in Eq. (2) as the sum over s in c of P(s|x,q), the left side is -P(c)logP(c), while the right side is the within-cluster entropy -sum_{s in c} [P(s)/P(c)] log(P(s)/P(c)). These two quantities are equal only in degenerate cases, e.g., when a single sequence carries all cluster mass. Consequently, Eq. (6) is not a valid semantic-entropy estimator: if H_c is read literally as -P(c)logP(c), then Eq. (6) evaluates -sum_i P(C_i)^2 logP(C_i), which is not semantic entropy; if H_c is read as the within-cluster entropy on the right side of Eq. (5), then Eq. (6) becomes a weighted average of within-cluster entropies and discards the between-cluster uncertainty term entirely. Since the reliability index is defined through this entropy, the reported AUC values of 0.862 on VQA and 0.764 on report generation are not measurements of the proposed estimator. Equation (7) inherits the same error.","section":"Methodology, 'Formulation for robust uncertainty estimation', Eq. (5)"},{"comment":"The paper contains several mutually inconsistent headline numbers that must be reconciled. The abstract reports an IoU of 0.703 across 141 anatomical regions, and the Results section repeats the IoU 0.703 for anatomical localization, but Table 6 reports an overall lesion-detection IoU of 0.9019. The Results section states a region-recognition F1-score of 0.774, whereas Table 6 lists the overall region-recognition F1 as 0.6265. The number of evaluation datasets is given as 347 in the abstract, 23 public benchmarks in the Introduction and Methods, and 38 benchmarks in the Discussion, with no definition of which count is being used. Finally, the abstract and conclusion claim a 26.9% average improvement in survival and recurrence prediction, but the per-task AUC improvements reported in the Results are 16.7% for PFS, 18.1% for NSCLC OS, and 2.7% for GBM OS, whose average is approximately 12.3%.","section":"Abstract; Results regional-performance section; Table 6; Introduction"},{"comment":"The evaluation setup creates a potential closed loop that weakens the claimed superiority over prior models. The principal GMAI comparator, MedRegA, is reference [80], which is the authors' own prior work; this self-comparison is not disclosed in the main text. In Table 10, the free-text region-VQA responses are scored by GPT-4o as judge, and the reasoning dataset used for training is also generated with GPT-4o. Without a human-validated LLM judge or an independent evaluation protocol, the 'accuracy', 'consistency', and 'clinical relevance' scores in Table 10 are not established as objective measurements, and the improvement margins over MedRegA and MedDr should be interpreted with caution.","section":"Results, 'XMedGPT is A Clinician-Centric AI Assistant...'; Table 10; Methodology, 'Formulation for medical reasoning'"},{"comment":"The prognostic claims rest on a modeling choice that is not validated. Patient-level survival probabilities are obtained by averaging slice-level predictions across CT slices, but no analysis is provided to justify this aggregation, and the external cohorts are small (n=49, 58, 133, and 163). The Discussion acknowledges the small-cohort limitation, but the manuscript does not report calibration or decision-curve analyses for the survival models. Given that prognostic modeling is one of the four pillars of the paper, the reported improvement over task-specific models should be supported by an evaluation of the aggregation procedure and by calibration measures, not only by AUC and accuracy on small external sets.","section":"Methodology, 'Implementation for prognostic decision making'; Discussion"}],"minor_comments":[{"comment":"The abstract contains 'external validation sans four anatomical systems', which should read 'across four anatomical systems'; a typographical pass is needed for similar issues (e.g., 'Generalists Medical AI' in the Methodology, 'Direct Policy Optimization' for DPO, and 'Bradeley-Terry' for Bradley-Terry).","section":"Abstract and throughout"},{"comment":"The 'Mean' row in Table 10 appears misaligned with the column headers: the listed mean values (7.98, 6.68, 7.89, 3.22, 4.58, 8.61) do not correspond in order to MedDr, MedRegA, LLaVA-Med, MedFlamingo, RadFM, and Ours. The authors should verify that each column's mean is placed under the correct model.","section":"Table 10"},{"comment":"Figure 4e mentions Steps 1, 2, and 4 of the chain-of-thought evaluation, while the text describes a five-step process; the caption should be aligned with the step numbering used in the main text.","section":"Figure 4 caption"},{"comment":"The code is promised 'after the manuscript is published', which is insufficient for a benchmark-heavy paper with many reported state-of-the-art numbers; providing code, checkpoints, or at least a detailed evaluation protocol at submission would substantially improve verifiability.","section":"Code Availability"}],"recommendation":"reject","confidential_remarks":"The central mathematical error in Eq. (5) is not a stylistic or presentational issue: it invalidates the uncertainty-quantification results that are a core contribution of the paper. The numerical inconsistencies across abstract, Results, and tables further undermine the reliability of the reported benchmark claims. While the general idea of a consistency-based reliability index is worth pursuing, the current manuscript would require a re-derivation of the estimator, a full rerun of the uncertainty experiments, and a reconciliation of all reported numbers. I therefore recommend rejection rather than major revision, as the central claims cannot be accepted in their current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely broad systems paper, but Eq. (5) is algebraically false, so the headline reliability-index results are not measuring what the authors claim.\n\nWhat's actually new: XMedGPT is a real systems integration—audio input, grounded region outputs, and prognostic heads on top of InternVL-2, trained on ~7M image-text pairs. The idea of a clinician-facing reliability index and the guardrail evaluation are sensible additions. The scale of the evaluation, spanning public benchmarks and some in-house external data, is respectable.\n\nThe soft spots are serious. Eq. (5) claims -P(c)logP(c) equals the entropy of the normalized within-cluster distribution. That's not true except in the degenerate single-sample case. Read one way, Eq. (6) computes -ΣP(c)^2 logP(c), which is not semantic entropy. Read the other way, it becomes a weighted average of within-cluster entropies and discards between-cluster uncertainty. Either way the AUCs of 0.862 and 0.764 are evaluating a quantity that is not the defined estimator. That's a load-bearing flaw, not a typo.\n\nThere are also transparency issues. The main baseline MedRegA is the authors' own prior model, but that's not disclosed in the comparisons. The reasoning data and the free-text VQA evaluation are generated or judged by GPT-4o, creating a closed loop. Code and data are not available, so there is no independent check. Some reported numbers also conflict across abstract, text, and tables (IoU 0.703 vs 0.9019, region-recognition F1 0.774 vs 0.6265). The prognostic cohorts are small, though the Discussion acknowledges that.\n\nIf I'm honest, the paper has the shape of a useful systems contribution once the estimator is corrected or replaced, the baselines are made fair, and the inconsistencies are cleaned up. As it stands, the central results don't stand. This deserves referee time—not a desk reject—because the scale and integration are real. But my recommendation is reject in current form and require major revision with code, corrected formulas, and disclosure of self-comparisons.","headline":"Ambitious medical MLLM integration, but the central uncertainty formula is wrong and the reported AUCs don't evaluate the proposed reliability index.","tokens_in":43308,"tokens_out":3749,"would_cite":false,"duration_ms":35966,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that XMedGPT, a single 8-billion-parameter medical multimodal model, can simultaneously ground its findings in images, score its own uncertainty, and predict survival, outperforming prior generalist medical AI and GPT-4o…","keywords":["generalist medical AI","multimodal explainability","visual grounding","semantic entropy","uncertainty quantification","prognostic prediction","medical VQA","radiology report generation"],"falsifier":"Recompute both sides of Equation (5) on the paper's own VQA examples: $-\\hat{P}(c)\\log \\hat{P}(c)$ versus $\\sum_{s\\in c} \\hat{p}_s\\log \\hat{p}_s$, with $\\hat{p}_s$ the probability of sequence $s$ normalized within cluster $c$. They differ whenever the sequences in a cluster have unequal probabilities; then re-estimate the uncertainty AUC using $H=-\\sum_c \\hat{P}(c)\\log \\hat{P}(c)$ and compare with the reported 0.862 and 0.764. A material drop would mean the reliability advantage is an artifact of the mislabeled estimator.","tokens_in":42229,"feed_emoji":"🩻","tokens_out":9408,"duration_ms":85094,"temperature":0.7,"pith_summary":"XMedGPT is a generalist medical AI that aims to do more than perceive: the authors build a single 8-billion-parameter multimodal model that writes diagnoses and reports, points to the anatomical regions behind each statement, attaches a reliability score to its own outputs, and predicts survival and recurrence from imaging plus clinical data. The paper reports that this one model outperforms prior generalist medical AI and GPT-4o across diagnosis, visual question answering, captioning, region grounding, and prognosis, with external validation on hospital data covering eleven lesion types. A sympathetic reader would care because the proposal is a concrete route from opaque medical language models toward tools a clinician can audit: every claim carries a location and a confidence, and the model itself flags when it is unsure. If the reported numbers hold, generalist medical AI becomes a decision-support layer rather than a perception-only system.","feed_headline":"Medical AI grounds findings, doubts itself, and predicts survival","feed_subtitle":"Grounding, uncertainty scoring, and prognosis in one 8B model, beating prior medical AI and GPT-4o.","key_machinery":"The load-bearing machinery is the pairing of a grounded report generator with a reliability index. The generator follows a five-step chain of thought: identify anatomy, localize it, describe each region, integrate patient context, then write the report, so every sentence can be traced to detected boxes. The reliability index converts each generated sentence into several yes/no visual question-answer pairs, compares the answers with the sentence, and computes a reformulated semantic entropy over consistency clusters; equations (5)-(7) define that entropy, with equation (5) intended to correct underestimation when all sampled answers fall in one cluster. This index is what turns raw outputs into high, medium, or low reliability flags for the clinician.","core_discovery":"On the paper's own terms, the discovery is that perception, grounding, uncertainty quantification, and prognosis can live in one medical multimodal model. XMedGPT is fine-tuned from a vision-language transformer on over seven million image-text pairs, including 1.6 million with pixel-level annotations, and then post-trained for chain-of-thought reasoning, audio input, and region-level interaction. The authors claim it achieves 0.703 IoU for lesion localization across 141 anatomical regions, an uncertainty estimation AUC of 0.862 on visual question answering and 0.764 on sentence-level report generation, and survival and recurrence AUCs that beat the prior task-specific models by 26.9% on average and GPT-4o by 25.0%. These results are presented as evidence that explainability, reliability scoring, and prognostic modeling can be unified rather than bolted on.","pith_inferences":["The paper's uncertainty numbers depend on Eq. (5), which identifies the entropy contribution of a cluster with the entropy of the sequence probabilities inside that cluster; the two expressions are not generally equal, so the reported AUCs likely evaluate a related but different estimator.","The prognostic gains come from aggregating slice-level probabilities on external cohorts of 49 to 163 patients; a patient-level, calibration-focused validation on larger cohorts would be needed to know whether the margin over GPT-4o and task-specific models transfers.","The observed improvement in uncertainty AUC as the number of generated questions per sentence rises (0.676 to 0.764) suggests that most of the gain comes from spending more test-time computation, leaving the choice of question budget an open design variable.","The Kendall tau-b of 0.479 between reasoning-path quality and final report quality shows the two co-vary, but it does not establish that the visible reasoning is what produced the correct report; an intervention study would be needed to test causation."],"forward_implications":["If the claims hold, a single 8B model can replace separate tools for diagnosis, VQA, captioning, grounding, audio interaction, and prognosis in one clinical workflow.","Grounded reporting with 0.703 IoU across 141 regions means each textual finding carries spatial evidence a clinician can check against the image.","A reliability index with 0.862 VQA AUC and 0.764 report AUC would let clinicians triage which AI statements need scrutiny and which can be acted on.","Survival and recurrence predictions that beat prior task-specific models by 26.9% would extend generalist medical AI from reading images to supporting treatment decisions.","Detecting 83% of unanswerable image-query mismatches would give the system a practical guardrail against confident hallucinations."],"supporting_citations":[{"why":"Supplies the semantic entropy and bidirectional entailment method that the reliability index extends and reformulates.","marker":"[20]"},{"why":"MedRegA is the prior generalist medical AI used as the main in-domain and region-grounding baseline.","marker":"[80]"},{"why":"GPT-4o system card is the general-purpose baseline across benchmark tasks and prognostic evaluations.","marker":"[35]"},{"why":"Task-specific model for TKI survival-benefit prediction; the baseline for progression-free survival comparisons.","marker":"[18]"},{"why":"BiRD and the Med-GRIT benchmark define the region-VQA tasks and evaluation metrics for grounding.","marker":"[34]"},{"why":"Chest-ImaGenome provides the sentence-region alignments used to build the chain-of-thought reasoning dataset.","marker":"[85]"},{"why":"GMAI-MMbench is the 284-dataset benchmark that supports the broad generalizability claims.","marker":"[90]"},{"why":"OmniMedVQA is the 42-dataset multiple-choice benchmark used for cross-modality evaluation.","marker":"[33]"},{"why":"RadFM supplies training-split conventions and a foundation-model baseline for radiology benchmarks.","marker":"[84]"}],"fun_headline_variants":["XMedGPT unifies explainability, uncertainty scoring, and prognosis","One medical AI for grounding, uncertainty, and survival prediction","Multi-modal AI explains findings, quantifies doubt, beats GPT-4o","Grounding, uncertainty, and prognosis in a single 8B model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The uncertainty-scoring scheme rests on the claim in Equation (5) that the entropy contributed by a cluster of consistent answers equals the entropy of the individual answer probabilities inside that cluster; that identity is not true, so if it gives way the reported reliability scores are not scores of the proposed reliability index.","fun_headline_variants_meta":{"raw":{"variants":["XMedGPT unifies explainability, uncertainty scoring, and prognosis","One medical AI for grounding, uncertainty, and survival prediction","Multi-modal AI explains findings, quantifies doubt, beats GPT-4o","Grounding, uncertainty, and prognosis in a single 8B model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1640,"prompt_tokens":1039,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":655,"tokens_out":601,"duration_ms":5353,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:29:36.701301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute both sides of Equation (5) on the paper's own VQA examples: $-\\hat{P}(c)\\log \\hat{P}(c)$ versus $\\sum_{s\\in c} \\hat{p}_s\\log \\hat{p}_s$, with $\\hat{p}_s$ the probability of sequence $s$ normalized within cluster $c$. They differ whenever the sequences in a cluster have unequal probabilities; then re-estimate the uncertainty AUC using $H=-\\sum_c \\hat{P}(c)\\log \\hat{P}(c)$ and compare with the reported 0.862 and 0.764. A material drop would mean the reliability advantage is an artifact of the mislabeled estimator.","supporting_citations":[{"cited_title":"A refer-and-ground multimodal large language model for biomedicine","cited_arxiv_id":null,"evidence_quote":"BiRD and the Med-GRIT benchmark define the region-VQA tasks and evaluation metrics for grounding."},{"cited_title":"Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai.Advances in Neural Information Processing Systems, 37:94327–94427, 2024","cited_arxiv_id":null,"evidence_quote":"GMAI-MMbench is the 284-dataset benchmark that supports the broad generalizability claims."}],"review_version":1}