REVIEW 2 major objections 6 minor 1 cited by
How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that medical vision-language models are strong at visual understanding but consistently weaker at reasoning, making reasoning the critical barrier to clinical deployment.
desk verdict A useful raw benchmark matrix buried under a central claim that is the opposite of the paper's own tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a binary question classifier: joint image-text embeddings from gme-Qwen2-VL-2B-Instruct are used to train a classifier on MedXpert's explicit understanding/reasoning labels, and the trained classifier is then applied to label every question in OmniMedVQA, PMC-VQA, PathVQA, MMMU, SLAKE, and VQA-RAD. This classifier produces the understanding-versus-reasoning split that drives the paper's headline conclusions; results are computed by prompting models with a step-by-step instruction and comparing the boxed answer against ground truth.
What would settle it
A reader can check the paper's Tables 2 and 3 directly: for most model-benchmark pairs, the reported reasoning accuracy is higher than the corresponding understanding accuracy (for example, Qwen2.5-VL-3B on MedXpert scores 0.2151 on reasoning and 0.1802 on understanding), which would contradict the paper's claim that reasoning is consistently lower.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is a systematic decomposition of medical VLM performance into understanding and reasoning components. Using a binary classifier trained on MedXpert labels and applied to six other datasets, the paper reports that all evaluated models, regardless of size or medical specialization, score higher on understanding tasks than on reasoning tasks. From this it concludes that reasoning, not visual recognition, is the current bottleneck for clinical decision support, and that no open-source model is yet reliable enough for deployment.
Load-bearing premise
The entire understanding-versus-reasoning comparison rests on assuming that a classifier trained on MedXpert can correctly label questions in six other medical datasets as understanding or reasoning without any validation on those datasets.
Editorial extensions
If this is right
- If the reasoning bottleneck is real, reasoning-aware training objectives and modular architectures become the most direct levers for improving clinical utility.
- Benchmark design should report understanding and reasoning accuracy separately rather than relying on aggregate scores that mask the split.
- Hybrid strategies that combine the broad generalization of large general-purpose models with medical post-training are a promising path forward.
- Scaling model size improves performance but plateaus, so further gains should come from alignment and domain adaptation rather than parameters alone.
- No current open-source medical VLM should be used for clinical decision support without human oversight.
Reading between the lines
- A natural next study would validate the MedXpert-trained classifier on human-annotated subsets of each target dataset, since the cross-dataset transfer is currently unverified and could drive the reported reasoning gap.
- The pattern in the paper's own Tables 2 and 3, where reasoning accuracy is often numerically higher than understanding accuracy, suggests that the classifier's labels, not model capabilities, may be shaping the split; a label audit would settle this.
- If the classifier-based decomposition were replaced by human reasoning labels, model rankings could shift substantially, making the current comparison a test of the labeling method as much as of the models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Liu et al. evaluate ten open-source vision-language models (Qwen2.5-VL 3B/7B/32B/72B, MiMo-VL-7B SFT/RL, Lingshu-7B/32B, Huatuo-GPT-Vision-7B/32B) on seven medical VQA benchmarks. Using a binary classifier trained on MedXpert and transferred to all other datasets, they split every question into 'understanding' or 'reasoning' and report per-split accuracies. The paper claims three findings: general-purpose models match medical-specific models, reasoning performance is consistently lower than understanding, and performance varies widely across benchmarks. The central finding about reasoning is contradicted by the paper's own tables.
Significance. The study is a useful reference point for the medical VLM community: it evaluates a broad model set under a uniform prompt and answer-extraction protocol, and it is commendable that the authors attempt to separate perceptual from reasoning demands. However, the headline conclusion (abstract, Section 3.2, and Conclusion) that 'reasoning performance is consistently lower than understanding' is not supported by the reported numbers. In Tables 2 and 3, reasoning accuracy exceeds understanding accuracy in 66 of 70 model-benchmark cells; in only four cells is understanding higher. The decomposition classifier in Section 2.2 is also transferred to new datasets without any validation of its labels. Because the central claim is internally contradicted and the classification step is unvalidated, the paper cannot be accepted in its current form.
major comments (2)
- [Abstract; §3.2; Tables 2–3] The central claim is falsified by the paper's own data. In Tables 2 and 3, comparing reasoning accuracy (Table 2) with understanding accuracy (Table 3) across the 10 models and 7 benchmarks, reasoning accuracy is higher in 66 of 70 cells and lower in only 4 cells (Lingshu-7B on OmniMedVQA; Huatuo-GPT-Vision-7B on PathVQA and VQA-RAD; Huatuo-GPT-Vision-32B on PathVQA). For example, Qwen2.5-VL-72B scores 0.7863 on reasoning vs 0.7013 on understanding for SLAKE, and 0.6805 vs 0.6395 for OmniMedVQA. Section 3.2's statement that 'most models perform better on understanding tasks' is therefore the opposite of what the tables show. This is an internal numerical contradiction that invalidates the second abstract finding and the bottleneck claim in the Conclusion.
- [§2.2] The understanding/reasoning decomposition is load-bearing but not validated for cross-dataset transfer. A binary classifier is trained on MedXpert labels using gme-Qwen2-VL-2B-Instruct embeddings and then applied to OmniMedVQA, PMC-VQA, PathVQA, MMMU, SLAKE, and VQA-RAD without any manual spot-checking, agreement statistics, or per-dataset accuracy of the classifier's labels. If the distinction does not transfer across question formats, modalities, and annotation styles—which the paper does not establish—then every understanding-vs-reasoning comparison in Tables 2 and 3 is an artifact of the classifier. The authors should either validate the transferred labels on held-out human annotations or provide dataset-specific evidence that the split is meaningful.
minor comments (6)
- [Abstract] The abstract says 'eight benchmarks' but enumerates seven, and Section 2.1 says 'seven widely used benchmarks'; the count should be corrected.
- [§2.2] The classifier description omits the training set size, label balance, classifier architecture, and hyperparameters, which prevents reproduction.
- [§2.2] MedXpert is both the classifier training source and one of the evaluation benchmarks; the paper should clarify whether the classifier was trained on the same split used for evaluation and whether there is any leakage.
- [Tables 1–3] The captions refer to color shading ('Darker orange indicates higher per-column performance'); in grayscale the ordering is not visible, so consider sorting rows or adding numeric markers.
- [§3.1] Claims such as 'this benefit begins to plateau at larger scales' are based on qualitative inspection of Table 1 without confidence intervals or pairwise significance tests.
- [General] The paper does not state whether code, evaluation logs, or the classifier's predicted labels will be released; for a benchmarking study, releasing the exact prompts, answer-extraction script, and split labels would substantially improve reproducibility.
Circularity Check
No circular derivation: benchmark accuracies are external measurements and the only internal classifier is an auxiliary tool whose outputs are not definitionally tied to the paper's conclusions.
full rationale
The manuscript is a benchmarking study rather than a derivation chain: the central numbers in Tables 1-3 are model accuracies computed from external ground-truth answers using official inference code, so the headline results are not produced by fitting any parameter of this paper. The only internal fitted component is the binary understanding/reasoning classifier of Section 2.2, trained on MedXpert labels and then transferred to six other benchmarks. That classifier is an auxiliary measurement instrument; the paper's conclusion that reasoning is a bottleneck is not entailed by the classifier's training objective, and in fact the published tables contradict that conclusion, with reasoning accuracy exceeding understanding accuracy in 66 of 70 cells. A numerical contradiction is a correctness problem, not a circularity: the claim would be circular only if the classifier were trained to produce lower reasoning scores or if the conclusion were definitionally identical to the training labels, neither of which is the case. The self-citations to MedVLM-R1 and Lingshu are to evaluated models and related work, not to a uniqueness theorem or to the sole justification of the main claim. No equation in the paper reduces a prediction to its own inputs by construction, and the lack of external validation for the classifier transfer is a validity concern rather than circularity. Therefore no significant circularity is found.
Assumptions & free parameters
free parameters (1)
- Understanding/reasoning classifier weights and hyperparameters =
Unreported (trained on MedXpert labels)
assumptions (3)
- domain assumption MedXpert understanding/reasoning labels generalize to all other benchmarks through the trained classifier.
- domain assumption A single prompt with boxed-answer extraction yields fair and comparable accuracy across all models.
- domain assumption The selected seven benchmarks are a representative sample of medical VLM capability.
Cite this review
Pith. "Pith review of How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study." pith.science (2026). https://pith.science/paper/5TUPF2YV
@misc{pith2026250711200,
author = {Pith},
title = {Pith review of: How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/5TUPF2YV}},
note = {Machine review of arXiv:2507.11200}
}
read the original abstract
Vision-Language Models (VLMs) trained on web-scale corpora excel at natural image tasks and are increasingly repurposed for healthcare; however, their competence in medical tasks remains underexplored. We present a comprehensive evaluation of open-source general-purpose and medically specialised VLMs, ranging from 3B to 72B parameters, across eight benchmarks: MedXpert, OmniMedVQA, PMC-VQA, PathVQA, MMMU, SLAKE, and VQA-RAD. To observe model performance across different aspects, we first separate it into understanding and reasoning components. Three salient findings emerge. First, large general-purpose models already match or surpass medical-specific counterparts on several benchmarks, demonstrating strong zero-shot transfer from natural to medical images. Second, reasoning performance is consistently lower than understanding, highlighting a critical barrier to safe decision support. Third, performance varies widely across benchmarks, reflecting differences in task design, annotation quality, and knowledge demands. No model yet reaches the reliability threshold for clinical deployment, underscoring the need for stronger multimodal alignment and more rigorous, fine-grained evaluation protocols.
Figures
Forward citations
Cited by 1 Pith paper
-
Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images
On 240 clinician-graded decisions from 20 wound cases, ChatGPT scored 72.5% while the best medical VLM (HuluMed) scored 40%.
Reference graph
Works this paper leans on
-
[5]
arXiv preprint arXiv:2502.19634 (2025)
Pan, J., Liu, C., Wu, J., Liu, F., Zhu, J., Li, H.B., Chen, C., Ouyang, C., Rueckert, D.: Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. arXiv preprint arXiv:2502.19634 (2025)
arXiv 2025
-
[1]
, " * write output.state after.block = add.period write
ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 'after.sentence := #3 '...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize ":" * " " *...
-
[3]
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-vl technical report (2025), https://arxiv.org/abs/2502.13923
arXiv 2025
-
[4]
arXiv preprint arXiv:2406.19280 (2024)
Chen, J., Gui, C., Ouyang, R., Gao, A., Chen, S., Chen, G.H., Wang, X., Zhang, R., Cai, Z., Ji, K., et al.: Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280 (2024)
arXiv 2024
-
[6]
Team, X.L.C.: Mimo-vl technical report (2025), https://arxiv.org/abs/2506.03569
arXiv 2025
-
[7]
arXiv preprint arXiv:2505.11462 (2025)
Thapa, R., Wu, Q., Wu, K., Zhang, H., Zhang, A., Wu, E., Ye, H., Bedi, S., Aresh, N., Boen, J., et al.: Disentangling reasoning and knowledge in medical large language models. arXiv preprint arXiv:2505.11462 (2025)
arXiv 2025
-
[8]
arXiv preprint arXiv:2506.07044 (2025)
Xu, W., Chan, H.P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu, C., Li, Z., et al.: Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044 (2025)
arXiv 2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.