{"id":"e08dad60-a7a2-4bba-b607-c010fcacedfc","arxiv_id":"2505.18915","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DeepTumorVQA, a 9,262-volume 3D medical VQA benchmark, shows that current vision-language models handle measurement but remain far from clinical-grade lesion recognition and reasoning.","lead":"This paper introduces DeepTumorVQA, a benchmark of over 9,000 CT scans and 395,000 expert-style questions that tests whether vision-language AI models can recognize tumors, measure organs, and reason like radiologists. It finds current models can handle basic measurements but fail at lesion recognition and clinical reasoning, so they are not yet ready for real diagnosis.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Medical Reasoning labels are unvalidated rule-based arithmetic proxies (Appendix B Table 5, Appendix A.1), so the 'not ready for clinical diagnosis' conclusion hinges on an unproven equivalence between these rules and expert clinical judgment.","rationale":"I agree with the reader that the single most load-bearing assumption is the clinical validity of the rule-based Medical Reasoning labels. The central contribution—a benchmark whose headline conclusion is that VLMs are not ready for clinical diagnosis—stands or falls on whether the 'Medical Reasoning' questions measure genuine clinical reasoning. The paper's own construction, especially Appendix B Table 5 and Appendix A.1, shows that many of these labels are deterministic thresholds or approximations applied to segmentation-derived metadata. If those thresholds do not match expert judgment, the benchmark may be a valid test of precise perception and numerical comparison but not of diagnostic reasoning, and the title's claim would be overstated. The proposed expert-agreement check is decisive and feasible because the data and radiologist annotations are described as available. I did not choose data contamination as the primary concern because leakage would tend to inflate RadFM's performance, making the 'not ready' conclusion more conservative rather than undermining it. Statistical rigor and ablation controls are real but secondary; they are addressable without changing the central claim. Therefore the reader's CONDITIONAL verdict remains appropriate: conditional on validating the medical-reasoning labels.","tokens_in":17828,"tokens_out":5017,"duration_ms":50115,"concrete_test":"Randomly sample 200 Medical Reasoning QA pairs from the DeepTumorVQA test split, stratified by the six subtypes. Have 3–5 board-certified radiologists answer each question from the full CT volume using standard clinical criteria, blinded to the rule-based labels and to each other; compute per-subtype and overall agreement (Cohen's kappa or Fleiss' kappa) between rule labels and expert labels. If agreement for subtypes such as pancreatic tumor staging, pancreatic cyst resectability, and lesion type classification is below roughly 0.7, or accuracy against the expert majority is not high, then the Medical Reasoning category does not validly measure clinical reasoning and the headline conclusion should be restated as a measurement/arithmetic limitation rather than a broad 'not ready for clinical diagnosis' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that current VLMs are not meeting clinical needs because they struggle with reasoning—depends on DeepTumorVQA's Medical Reasoning questions being valid instances of clinical reasoning. That premise is not established. Section 3.1 states that reasoning questions are 'systematically constructed by composing functions over outputs from the recognition and measurement stages'; Appendix B (Table 5) confirms that the medical-reasoning 'logic' is largely arithmetic thresholding on segmentation-derived metadata: fatty liver from liver/spleen HU ratio, pancreatic steatosis from pancreas/spleen HU ratio <0.7, pancreatic cyst resectability from cyst volume >3.0 cm³, and kidney lesion type from an HU threshold. Appendix A.1 further says pancreatic T-stage is 'approximate[d]... based on existing staging protocols' rather than expert-assigned. These rules may reflect screening cut-offs from the literature, but they are not validated as ground truth for diagnosis: real resectability depends on vascular involvement, symptoms, and patient factors; T-stage depends on extra-pancreatic extension and vessel involvement; lesion type depends on enhancement pattern, not a single HU threshold. If these labels are wrong or clinically reductive, then low model accuracy on Medical Reasoning does not demonstrate lack of reasoning capability—it may only demonstrate failure at precise HU/volume estimation followed by arithmetic comparison. The conclusion 'still not meeting clinical needs' would then be supported for measurement/recognition, but not for the 'clinical reasoning' component emphasized in the title. The paper's Limitations (Section 5) acknowledges segmentation noise but never tests whether rule-generated labels agree with independent expert judgment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DeepTumorVQA, a large-scale 3D medical VQA benchmark for abdominal tumors in CT. The dataset comprises 9,262 volumes from 17 public datasets, 395K question-answer pairs generated from radiologist-verified organ/lesion segmentations and structured metadata, organized into measurement, recognition, visual reasoning, and medical reasoning categories. Four VLM families (five model variants) are fine-tuned and evaluated. The authors report that models perform reasonably on measurement, poorly on lesion recognition and reasoning, with RadFM best overall; they also propose an nnUNet-based cropping preprocessing that improves lesion recognition. The manuscript argues current VLMs are not ready for clinical diagnosis and releases benchmark code/data.","tokens_in":18164,"tokens_out":5255,"duration_ms":46389,"significance":"If validated, DeepTumorVQA would be a valuable community resource: it is large-scale, multi-center, publicly released, and organized along a clinically motivated task hierarchy with radiologist involvement. The benchmark's finding that modern medical VLMs collapse to majority-class responses on lesion recognition (Table 4) is an important, sobering result for the field. The paper also ships open-source code and data and includes a concrete preprocessing intervention (nnM3D) with measurable improvements, which is a strength. However, the central clinical-readiness conclusion rests on medical-reasoning labels that are currently unvalidated rule-based proxies, and the model rankings lack statistical grounding; both issues must be addressed before the benchmark can serve as a rigorous diagnostic yardstick.","major_comments":[{"comment":"The Medical Reasoning ground truth is generated by deterministic arithmetic rules over segmentation-derived metadata, e.g., fatty liver from liver/spleen HU ratio, pancreatic steatosis from pancreas/spleen HU ratio < 0.7, pancreatic cyst resectability from cyst volume > 3.0 cm^3, and kidney lesion type from an HU threshold. Section 3.1 explicitly states that reasoning questions are 'systematically constructed by composing functions over outputs from the recognition and measurement stages.' The paper provides no validation that these rules reproduce expert clinical judgment; real resectability, staging, and lesion typing involve vascular involvement, enhancement patterns, and patient factors that these thresholds do not capture. Because the abstract's central conclusion ('still not meeting clinical needs') is largely driven by the Medical Reasoning results, the authors must either add a radiologist-agreement study validating a sample of these labels, or reposition the reasoning category as metadata-composition arithmetic and adjust the clinical-readiness claim accordingly.","section":"Section 3.1, Appendix B Table 5"},{"comment":"All benchmark results are reported as single-run point estimates with no error bars, confidence intervals, or significance tests. The differences used to support the ranking claims are small: for multi-choice total average, RadFM 0.662 vs M3D-P3 0.632 vs CT-CHAT 0.628, and several subtype differences are under 0.02. Without variance estimates or paired significance testing, the conclusions that RadFM 'stands out' and that full fine-tuning versus LoRA matters are not statistically supported. The authors should report multiple seeds or bootstrap confidence intervals on the test set.","section":"Table 3, Section 4.2"},{"comment":"Table 4 reveals that several models (M3D-LLaMA2, M3D-Phi3, CT-CHAT) collapse to majority-class predictions on recognition tasks, yielding degenerately high sensitivity or specificity. Table 3 reports accuracy for recognition subtypes without sensitivity/specificity, and several free-text accuracies are identical to multi-choice values (e.g., colon lesion existence 0.859), which is consistent with such collapse. Reporting accuracy alone therefore overstates or obscures recognition capability. The paper should report sensitivity, specificity, or balanced accuracy for all recognition subtypes, not only the three in Table 4.","section":"Table 4, Section 4.5"},{"comment":"The benchmark is constructed entirely from public datasets (Table 1) that are standard in medical imaging research and are plausibly included in the pretraining corpora of the evaluated VLMs, especially RadFM with 16M 2D+3D pairs. The manuscript does not analyze potential overlap between benchmark volumes and pretraining data. Without a contamination analysis or evaluation on a non-public held-out cohort, the observed RadFM advantage may partially reflect data leakage rather than model capability. Please provide an overlap analysis or qualify the ranking accordingly.","section":"Table 1, Section 4.1"},{"comment":"The ablation in Section 4.3 trains only RadFM with and without measurement/recognition tasks and describes a 'relatively small performance gap' without reporting the numerical accuracies or any uncertainty. This is insufficient evidence for the claim that basic tasks have limited impact on reasoning. The authors should include the full result table with per-subtype numbers and standard errors, or remove this claim.","section":"Section 4.3, Figure 4"}],"minor_comments":[{"comment":"The definition of MRA (mean relative accuracy) is cited to reference [49] but not defined in the manuscript; please provide the formula.","section":"Section 3.1"},{"comment":"The statement defining 'clinical diagnosis' appears in the Question Generation paragraph; consider moving it to Section 3.1 where the diagnostic task hierarchy is introduced.","section":"Section 3.2"},{"comment":"The free-text and multi-choice columns are not clearly separated visually; consider using column groups or color coding to improve readability.","section":"Table 3"},{"comment":"Several thresholds (e.g., the kidney lesion HU threshold and the fatty liver HU ratio criteria) are not stated in the table; please specify the exact values used and cite the clinical sources.","section":"Appendix B Table 5"},{"comment":"The terms nnVLM and nnM3D are used before the naming convention is fully introduced; clarify that 'nn' refers to nnUNet-based preprocessing.","section":"Section 4.5"},{"comment":"The figure is extremely dense; increasing font size or splitting it into two panels would improve legibility.","section":"Figure 1"},{"comment":"Some dataset URLs in Table 1 are likely to become broken over time; consider adding DOIs or stable dataset identifiers.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a potentially valuable asset for the community, and the paper's empirical results on recognition collapse are important. However, the clinical-readiness claim is currently overstated given the unvalidated reasoning labels and the lack of statistical rigor. I would encourage the editor to invite a major revision rather than reject, because the core resource is sound and the concerns are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: DeepTumorVQA, a large 3D abdominal CT VQA benchmark with 9,262 volumes, 395K questions, lesion annotations from 23 radiologists, and four VLMs benchmarked. The dataset is real, released, and the main empirical finding — current VLMs are decent at organ-level measurement but poor at lesion recognition and reasoning — is credible and important for the field. This is the kind of large-scale, reusable testbed medical imaging AI needs; it deserves serious referee time.\n\nWhat's actually new: first tumor-centric 3D VQA benchmark at this scale, with a four-level question hierarchy (measurement, recognition, visual reasoning, medical reasoning), expert 3D lesion annotations, and open code/data. The benchmarking is useful: RadFM's full fine-tuning and larger pretraining clearly beat LoRA-based models, and the nnUNet-cropped variants show explicit localization goes a long way. Those are reproducible findings from shipped code.\n\nSoft spots, in proportion. The biggest is the medical reasoning ground truth. As the appendix shows, several of these questions are threshold checks on segmentation-derived metadata: fatty liver from liver/spleen HU ratio, pancreatic steatosis from pancreas/spleen HU ratio <0.7, resectability from cyst volume >3 cm³, lesion type from an HU cutoff. These are reasonable screening proxies, but not validated against expert clinical judgment. So when a model scores low on 'medical reasoning,' it may be failing at precise HU/volume estimation plus arithmetic, not at clinical reasoning. That weakens the headline claim 'not meeting clinical needs' for the reasoning component specifically; it is solid for recognition, where class collapse is obvious.\n\nSecond, no error bars or significance tests — the paper reports single-run accuracies, and some differences are small. A benchmark paper should report variance or multiple seeds. Third, no contamination analysis: their 17 public datasets likely overlap with pretraining corpora; a per-dataset breakdown would help. Fourth, the rule-based reasoning labels are never checked against a second radiologist's judgment, which is the natural validation step given they already had 23 radiologists.\n\nThese are fixable, not fatal. The central empirical finding about recognition failures stands on its own. The paper is honest about limitations — it explicitly says conclusions would benefit from more controlled ablations.\n\nWho it's for: anyone working on medical VLMs, 3D radiology AI, or VQA benchmarking. A serious referee should engage. My recommendation: accept a revised version after the authors add variance or error bars, a contamination check, and at least a small validation study of the medical-reasoning rules against a radiologist's gold standard. Even without those, the benchmark deserves to exist in the literature.","headline":"A genuinely useful large-scale 3D medical VQA benchmark with credible negative results on VLMs, though the 'medical reasoning' labels are unvalidated arithmetic proxies and the headline conclusion overreaches slightly.","tokens_in":18741,"tokens_out":1897,"would_cite":true,"duration_ms":12221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current 3D medical vision-language models handle measurement tasks but fail at lesion recognition and clinical reasoning, and are therefore not ready for clinical diagnosis.","keywords":["3D medical VQA","visual question answering","abdominal CT","tumor diagnosis","vision-language models","clinical reasoning","benchmark","lesion recognition"],"falsifier":"A concrete check would be to take a sample of DeepTumorVQA cases, have independent board-certified radiologists give their own clinical decisions on the medical-reasoning questions, and compare those decisions with the benchmark's rule-generated ground truth; a disagreement rate comparable to the VLM error rate would falsify the claim that the benchmark measures clinical reasoning. A sharper version would be to swap the rule thresholds, for example resectability at 2.0 cm³ instead of 3.0 cm³, and retest the models: if accuracy follows the threshold rather than the underlying pathology, the models are applying rules rather than diagnosing.","tokens_in":17652,"feed_emoji":"🩻","tokens_out":5692,"duration_ms":50401,"temperature":0.7,"pith_summary":"DeepTumorVQA is a 3D visual-question-answering benchmark for abdominal tumor diagnosis built from 9,262 CT volumes and 395K expert-annotated questions spanning recognition, measurement, visual reasoning, and medical reasoning. Benchmarking RadFM, M3D, Merlin, and CT-CHAT, the paper claims that current models can perform adequately on measurement tasks and coarse quantification, but consistently struggle with small-lesion recognition and with reasoning tasks that build on it, especially medical reasoning. The authors conclude that no existing 3D medical VLM is clinically reliable for tumor diagnosis, and they argue that the benchmark's compositional structure lets future models be scored by diagnostic sub-skills rather than by reading comprehension. A sympathetic reader would care because the benchmark turns the vague question of whether medical VLMs are useful into measurable abilities tied to the radiological workflow.","feed_headline":"Medical VLMs still not ready for tumor diagnosis, benchmark finds","feed_subtitle":"RadFM, M3D, Merlin, and CT-CHAT handle measurements but stumble on lesion recognition and medical reasoning.","key_machinery":"The central object is the DeepTumorVQA question-generation pipeline, a CLEVR-style functional program that converts organ and lesion segmentation masks into structured metadata such as volume, HU values, adjacency, and lesion counts, then composes deterministic functions over that metadata to produce templated questions in four difficulty tiers: Measurement, Recognition, Visual Reasoning, and Medical Reasoning. This machinery matters because it gives the benchmark a dependency structure: reasoning questions are literal compositions of recognition and measurement outputs, so a model's reasoning score can be traced back to its perception score. The evaluation machinery consists of three metrics, accuracy for multiple-choice questions, exact match for categorical free-text answers, and mean relative accuracy for numerical answers, chosen to avoid the weaknesses of BLEU and ROUGE for short factual medical answers.","core_discovery":"The paper's central claim is that state-of-the-art 3D medical vision-language models are not yet clinically reliable for tumor diagnosis: they perform adequately on measurement tasks such as organ volume and HU value estimation, and can approach segmentation-based methods on some recognition tasks, but their accuracy collapses on small or low-contrast lesion recognition and on visual and medical reasoning. The benchmark also exposes two drivers of that performance: large-scale multimodal pretraining, as in RadFM, matters more than parameter count alone, and the design of the vision module plus image preprocessing matters substantially, with tokenized 3D encoders and organ-cropping preprocessing improving lesion recognition. The intended consequence is that DeepTumorVQA serves as a standard testbed for tracking whether future 3D medical VLMs actually gain diagnostic competence.","pith_inferences":["Editorial extension: because the medical-reasoning labels are rule-based proxies, such as resectability decided by cyst volume exceeding 3.0 cm³, the benchmark may measure the ability to apply fixed clinical thresholds rather than full radiological judgment, and a model could pass those items without understanding anatomy or clinical context.","Editorial extension: a direct test of whether the reasoning bottleneck is visual or logical would be to feed the benchmark's metadata tables as text only to the language models; if they still fail those reasoning questions, the deficit is in reasoning rather than perception.","Editorial extension: the authors' ablation that trains RadFM without measurement and recognition tasks suggests the benchmark can also quantify how much explicit perceptual supervision contributes to reasoning, and that large pretraining already encodes much of that knowledge implicitly."],"forward_implications":["None of the four benchmarked VLMs reaches clinically acceptable performance on the full benchmark, and medical reasoning is the weakest category, so current 3D medical VLMs should not be used for autonomous tumor diagnosis.","Measurement questions are the most tractable, meaning existing models can already support quantitative readouts such as organ volume and HU values.","Large-scale multimodal pretraining and full fine-tuning, as in RadFM, yield more consistent gains than larger LLM backbones or LoRA tuning alone.","Organ-cropping preprocessing can dramatically improve lesion recognition, with one model's kidney tumor sensitivity rising from 0% to 80.9%, suggesting that anatomical localization priors can substitute for full voxel-level supervision.","Because reasoning questions are composed from recognition and measurement outputs, improvements in perception should propagate to reasoning scores, allowing the benchmark to track whether VLM gains reflect real diagnostic progress."],"supporting_citations":[{"why":"Supplies the CLEVR-style functional program generation method used to compose questions from structured metadata.","marker":"[28]"},{"why":"RadFM is the best-performing benchmarked VLM and the main evidence that large-scale multimodal pretraining drives performance.","marker":"[48]"},{"why":"M3D is a benchmarked LoRA-tuned VLM whose failure modes support the paper's conclusions about recognition and reasoning limits.","marker":"[7]"},{"why":"Merlin is the benchmarked single-token ResNet-based VLM used to show that vision module design strongly affects 3D perception.","marker":"[11]"},{"why":"CT-CHAT is the CT-specific ViT-based VLM whose results are compared to show that input resolution and spacing alone do not guarantee diagnostic performance.","marker":"[16]"},{"why":"Supplies the segmentation-based organ localization method used in the preprocessing variants that substantially improve lesion recognition.","marker":"[25]"},{"why":"Provides the radiology-style structured report generation approach used to derive metadata and reasoning question context.","marker":"[8]"}],"fun_headline_variants":["Medical VLMs fail 3D tumor reasoning, new benchmark shows","DeepTumorVQA: VLMs not ready for clinical diagnosis","VLMs ace measurements, flub tumor recognition in 3D","3D tumor VQA benchmark exposes VLM clinical gaps","RadFM leads but all VLMs miss clinical bar on 3D tumors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the rule-based labels generated from segmentation metadata, such as resectability decided by cyst volume above 3.0 cm³ and steatosis by a pancreas-to-spleen HU ratio below 0.7, faithfully capture genuine clinical reasoning; if they do not, the conclusion that VLMs are unready for clinical diagnosis may only show that they fail at threshold arithmetic on noisy measurements.","fun_headline_variants_meta":{"raw":{"variants":["Medical VLMs fail 3D tumor reasoning, new benchmark shows","DeepTumorVQA: VLMs not ready for clinical diagnosis","VLMs ace measurements, flub tumor recognition in 3D","3D tumor VQA benchmark exposes VLM clinical gaps","RadFM leads but all VLMs miss clinical bar on 3D tumors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1304,"prompt_tokens":969,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":585,"tokens_out":335,"duration_ms":2660,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:23:14.239152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to take a sample of DeepTumorVQA cases, have independent board-certified radiologists give their own clinical decisions on the medical-reasoning questions, and compare those decisions with the benchmark's rule-generated ground truth; a disagreement rate comparable to the VLM error rate would falsify the claim that the benchmark measures clinical reasoning. A sharper version would be to swap the rule thresholds, for example resectability at 2.0 cm³ instead of 3.0 cm³, and retest the models: if accuracy follows the threshold rather than the underlying pathology, the models are applying rules rather than diagnosing.","supporting_citations":[],"review_version":1}