Pith. sign in

REVIEW 4 major objections 7 minor 24 references

Semantic sampling at inference time raises a general video VLM to 66.5% accuracy on a new 3D medical spatial-grounding benchmark, showing measurable anatomy localization where earlier medical VLM work reported near-random failure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 21:04 UTC pith:PWB6E65P

load-bearing objection Usable multi-axis 3D medical spatial-grounding benchmark plus a cheap decoding tweak that lifts one strong general VLM; the 13% gain needs a format-compliance check. the 4 major comments →

arxiv 2603.14579 v3 pith:PWB6E65P submitted 2026-03-15 cs.CV cs.LG

Medical Image Spatial Grounding with Semantic Sampling

Andrew Seohwan Yu , Mohsen Hariri , Kunio Nakamura , Mingrui Yang , Xiaojuan Li , Vipin Chaudhary This is my paper
classification cs.CV cs.LG
keywords vision language modelsspatial grounding3D medical imagingvisual groundingsemantic samplingmedical VQAinference-time decodinganatomical terminology
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that spatial grounding of anatomy in 3D medical volumes is a distinct, testable capability of vision-language models, not just report-style language fluency. It builds MIS-Ground, a public benchmark that systematically varies modality, slice direction, coordinate orientation, anatomical versus everyday direction words, visual prompts (points, boxes, masks), and text-only or image-free ablations. On that benchmark, general video VLMs—especially larger Qwen models—show real 3D and cross-slice spatial skill, while several specialized medical VLMs struggle mainly to follow the answer format. The authors then introduce MIS-SemSam: an inference-time decoder that rescores next tokens by probability mass in their semantic neighborhood in embedding space, stabilizing directional and anatomical word choice without retraining. With that change, Qwen3-VL-32B gains 13.06% absolute accuracy and reaches 66.5% overall, supporting the claim that much of the remaining error is language-side brittleness rather than total visual blindness.

Core claim

Video-native VLMs already possess nontrivial 3D medical spatial grounding—across slices, orientations, and prompt styles—when measured on a balanced, multi-factor benchmark, and that ability can be improved at negligible cost by decoding toward semantically coherent token neighborhoods rather than single-token logits alone. The authors report that MIS-SemSam raises Qwen3-VL-32B from its baseline to 66.5% overall accuracy on MIS-Ground (+13.06%), with competitive in-plane versus across-slice performance and clear sensitivity to anatomical priors and viewing orientation.

What carries the argument

MIS-SemSam (Medical Image Spatial Semantic Sampling): at each decode step, candidate tokens are rescored by summing (similarity-weighted) probability mass over precomputed nearest-neighbor content tokens in the language embedding space, so directional and anatomical synonyms reinforce one another instead of flipping under small logit noise.

Load-bearing premise

The benchmark’s accuracy numbers fairly measure true spatial grounding, rather than format compliance, answer-tag parsing, or exploitation of language-only anatomy priors—especially when invalid or untagged answers are dropped and specialized medical models fail mainly by not following the response format.

What would settle it

Re-run the full MIS-Ground suite on the same models with forced free-form answers (no special end tags), human adjudication of spatial correctness, and matched format-repair postprocessing for medical VLMs; if the 13% SemSam gain and the 3D/in-plane gap disappear while only tag-compliance rates change, the central performance claims fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces MIS-Ground, a multi-factor benchmark for spatial grounding of anatomical structures in 3D medical volumes (knee MRI from OAI/3DReasonKnee-Bench and torso CT from TotalSegmentator), spanning modalities, slice directions, RAS storage vs viewing orientations, anatomical vs colloquial direction terms, visual prompts (points/boxes/masks), question formats, and text-only / blank-background ablations. It also proposes MIS-SemSam, an inference-time, model-agnostic decoding method that rescores next-token candidates by aggregating probability mass over cosine neighborhoods in content-token embedding space (Eqs. 1–4). Across open and closed VLMs, Qwen3-VL-32B with MIS-SemSam is reported as the strongest configuration, with a 13.06% absolute accuracy gain to 66.5% overall, and the authors use that configuration to answer RQ1–RQ3 and AB1–AB2 on 3D understanding, terminology preference, visual-prompt effects, anatomy priors, and abstract spatial reasoning.

Significance. If the evaluation is fair, the work is useful to medical multimodal research: prior reports of near-random medical spatial grounding are challenged by a carefully factorized benchmark, and an inexpensive decoding-time intervention is shown to help a strong general video VLM without fine-tuning. Strengths include public release of MIS-Ground, systematic ablations (text-only and blank-background), explicit treatment of coordinate systems and terminology, and a fully specified, training-free rescoring procedure with negligible extra cost. These are concrete, reproducible contributions even if the absolute accuracies remain moderate.

major comments (4)
  1. §2.3 and §3: Responses missing the required answer tags after reasoning are omitted from accuracy. The headline claim (abstract / §3) that MIS-SemSam improves Qwen3-VL-32B by 13.06% to 66.5% is therefore not yet isolated from format/tag compliance. Because MIS-SemSam rescores content-token neighborhoods (Eqs. 1–4), it can preferentially stabilize morphological variants and tag-like tokens needed for a response to be counted. Please report, for the same questions, (i) the fraction of omitted/invalid responses with vs. without MIS-SemSam, (ii) accuracy restricted to the intersection of parseable responses, and (iii) a tag-agnostic or fixed-format scoring ablation. Without these, the spatial-grounding interpretation of the gain is overstated relative to the evidence.
  2. §2.4 and §3 / Fig. 2: MIS-SemSam is described as model-agnostic, but the detailed RQ/AB analysis and the 13.06% gain are effectively concentrated on Qwen3-VL-32B (and the “MIS-SemSam family” derived from Qwen3-VL). To support the model-agnostic claim and the method’s contribution beyond one decoder family, report MIS-SemSam on at least one other open family that already produces valid tags (e.g., Molmo2 8B and/or Qwen2.5-VL-72B) under identical truncation and temperature settings, including invalid-response rates.
  3. §2.3 and §3: Specialized 3D medical VLMs (M3D, Med3DVLM, MedGemma) are reported as near-failures, but inputs are forcibly resized to [32,256,256] / [128,256,256] and the same long-form reasoning + special-tag protocol is applied. This confounds “poor spatial grounding” with format compliance and resolution mismatch. Either (a) evaluate those models under their native prompt/output conventions and report a separate compliance metric, or (b) clearly demote them from the main accuracy ranking and avoid implying a fair head-to-head on spatial grounding.
  4. §3 (RQ2–RQ3, AB1): Several key findings rest on language priors and orientation–terminology interactions (e.g., anatomical terms 69.4% vs colloquial 57.8% in standard viewing; visual prompts hurting a 75.3% anatomical baseline in RAS storage). These are interesting, but the paper does not quantify how much of overall accuracy is explained by text-only priors (AB1: 69.3% anatomical without image) versus image-conditioned reasoning on the same item splits. Please report paired image-present vs text-only accuracy by terminology and orientation, and the fraction of items where the image flips the text-only answer. This is load-bearing for the claim that MIS-Ground measures spatial grounding rather than anatomical language priors.
minor comments (7)
  1. Abstract vs §1: the public URL is given as github.com/asy51/mis-ground in the abstract but anonymous.4open.science in the body; unify for the camera-ready version.
  2. §2.1–2.2: State exact train/test or scan-level sampling policy for the 33864 questions (how many unique volumes enter the reported numbers; whether multiple questions per volume are clustered for uncertainty).
  3. §2.3 cites Bayesian credible intervals [9] but §3 reports point percentages without intervals or sample sizes per RQ cell; add n and intervals to Fig. 2 and the RQ bullets.
  4. Eqs. (1)–(4): specify default K, K′, similarity threshold, temperature T, and Filter (top-M / top-p) used for the 13.06% result so the decoding configuration is fully reproducible.
  5. Fig. 1 caption and §2.1: briefly define RAS storage vs standard viewing for non-radiology readers; the distinction is central to RQ2–RQ3.
  6. Typographical cleanup: missing spaces in several compounds (“Visionlanguagemodels”, “modesofMedicalImageSpatialGrounding”, “emon-strated”) and inconsistent hyphenation of MIS-SemSam / MIS-Semsam in Fig. 2.
  7. Related work: situate Semantic Sampling against prior constrained/semantic decoding literature more explicitly so the medical adaptation’s novelty is clear.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark from external annotations and model-agnostic decoding evaluated against fixed ground truth.

full rationale

MIS-Ground is constructed from external annotated sources (OAI/3DReasonKnee-Bench bounding boxes and TotalSegmentator segmentations) with randomly paired structures, fixed visual/text prompts, and answer types; accuracy is scored by matching model outputs (after required tags) to those independent ground-truth spatial relations. MIS-SemSam is an inference-time rescoring of next-token logits by precomputed cosine neighborhoods over content-token embeddings (Eqs. 1–4), requiring no training or fitting to MIS-Ground scores and adding only table lookups. The reported 13.06% gain is a direct before/after comparison of the same model on the same held-out questions under two decoding rules; nothing in the derivation equates the method definition to the accuracy number by construction. The single self-citation (Hariri et al. [9] for Bayesian credible intervals) is used only for reporting and is not load-bearing for the spatial-grounding claims. No uniqueness theorems, fitted parameters renamed as predictions, or ansatzes smuggled via overlapping-author citations appear. The work is therefore self-contained against external data and falsifiable via the released benchmark.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

This is an empirical systems paper. The central claims rest on standard VLM decoding assumptions, curated medical datasets and annotations, and the authors' design choices for questions, validity filtering, and semantic neighborhoods. There are no deep physical axioms; the load-bearing premises are methodological.

free parameters (5)
  • decoding temperature T = 0.5
    Fixed at 0.5 for fairness across models; affects answer stability and SemSam behavior.
  • max new tokens = 8912
    Set to 8912 for long chain-of-thought responses; influences whether final answer tags appear and thus which questions are scored.
  • semantic neighborhood size K / kept neighbors K'
    Controls how much local embedding mass is aggregated in MIS-SemSam; chosen as part of the method configuration rather than derived.
  • candidate truncation filter (top-M / top-p)
    Determines the candidate set rescored by semantic neighborhoods; left as a standard but free decoding choice.
  • CT/MRI preprocessing windows and slice-count filters = CT slices kept in [200,450]; modality-specific windowing
    Percentile/soft-tissue windowing and omitting CT scans outside 200-450 slices shape the visual distribution the benchmark measures.
axioms (5)
  • ad hoc to paper Next-token probabilities from a pretrained VLM, rescored over cosine neighborhoods in token embedding space, are a valid proxy for more stable semantic choices in directional/anatomical language.
    Core MIS-SemSam premise in Section 2.4; not independently proven, only motivated by observed language-side brittleness.
  • domain assumption Ground-truth segmentations/bounding boxes from TotalSegmentator and 3DReasonKnee-Bench correctly define anatomical structure locations for spatial questions.
    Benchmark labels and visual prompts are derived from these annotations in Sections 2.1-2.2.
  • ad hoc to paper Omitting responses without required answer tags yields a fair accuracy comparison across models.
    Stated in Results; can bias against models that reason but fail format compliance.
  • domain assumption Video-native VLMs can be evaluated on multi-slice medical volumes as if slice sequences convey 3D spatial structure.
    Implicit throughout RQ1 and model selection; medical volumes are not natural videos.
  • standard math Bayesian credible intervals under limited trials adequately summarize performance robustness.
    Cited evaluation practice in Section 2.3; standard statistical tooling rather than a new claim.
invented entities (2)
  • MIS-Ground benchmark independent evidence
    purpose: Provide a multi-factor test suite for medical image spatial grounding across vision and language axes.
    Newly curated question set and evaluation protocol; independent evidence is the public release and dataset construction, not external prior existence.
  • MIS-SemSam decoding procedure no independent evidence
    purpose: Stabilize anatomical/directional token choice at inference by aggregating probability mass over semantic token neighborhoods.
    Adaptation of semantic sampling to medical VLM spatial grounding; evidence is empirical accuracy gains on MIS-Ground, not an external physical entity.

pith-pipeline@v1.1.0-grok45 · 12823 in / 3446 out tokens · 28299 ms · 2026-07-14T21:04:30.730578+00:00 · methodology

0 comments
read the original abstract

Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos. In medical imaging research, VLMs represent a bridge between object detection and segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce MIS-Ground, a benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of Medical Image Spatial Grounding. We release MIS-Ground to the public at github.com/asy51/mis-ground. In addition, we present MIS-SemSam, a low-cost, inference-time, and model-agnostic optimization of VLMs that improves their spatial grounding ability with the use of Semantic Sampling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06%.

Figures

Figures reproduced from arXiv: 2603.14579 by Andrew Seohwan Yu, Kunio Nakamura, Mingrui Yang, Mohsen Hariri, Vipin Chaudhary, Xiaojuan Li.

Figure 1
Figure 1. Figure 1: Sample question-answer pairs from MIS-Ground. For each 2D or 3D vision input (top), multiple questions are generated (bottom). Video VLMs display the ability to discern anatomical structure relationships across slices (RQ1); they prefer anatomical direction terms like anterior (RQ2); and they are capable of abstract spatial reason￾ing (AB2). Note: the CT image is shown in axial RAS standard viewing orienta… view at source ↗
Figure 2
Figure 2. Figure 2: Overall accuracy on the MIS-Ground benchmark, by model family and size. Note that Med3DVLM (0.0%) and M3D (15.9%) were tested but omitted from this graph. MIS-Semsam family are derived from the Qwen3-VL family. (RQ1) 3D Understanding: MIS-SemSam effectively processed 3D medical images, accurately predicting slice direction in 77.6% of cases. The model suc￾cessfully determined spatial relationships across s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 11 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2404.00578 (2024)

    Bai, F., Du, Y., Huang, T., Meng, M.Q.H., Zhao, B.: M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, S., et al.: Qwen-vl: A versatile vision-language model for understanding, lo- calization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    arXiv preprint arXiv:2502.13923 (2025) Medical Image Spatial Grounding with Semantic Sampling 9

    Bai, S., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) Medical Image Spatial Grounding with Semantic Sampling 9

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:2601.10611 (2026)

    Clark, C., Zhang, J., Ma, Z., Park, J.S., Salehi, M., Tripathi, R., Lee, S., Ren, Z., Kim, C.D., Yang, Y., et al.: Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611 (2026)

    Pith/arXiv arXiv 2026

  6. [6]

    arXiv preprint arXiv:2507.06261 (2025)

    Gemini Team: Gemini 2.5: Pushing the frontier with advanced reasoning. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  7. [7]

    google/models/gemini/(2026)

    Google DeepMind: Gemini 3 flash (model documentation).https://deepmind. google/models/gemini/(2026)

    2026

  8. [8]

    arXiv preprint arXiv:2403.17834 (2024)

    Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Do- gan, I., Durugol, O.F., Hou, B., Shit, S., et al.: Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834 (2024)

    arXiv 2024

  9. [9]

    In: The Fourteenth In- ternational Conference on Learning Representations (2026),https://openreview

    Hariri, M., Samandar, A., Hinczewski, M., Chaudhary, V.: Don’t pass@k: A bayesian framework for large language model evaluation. In: The Fourteenth In- ternational Conference on Learning Representations (2026),https://openreview. net/forum?id=PTXi3Ef4sT

    2026

  10. [10]

    arXiv preprint arXiv:2601.03191 (2026)

    Hashmi, A.U.R., Saeed, N., Lippert, C.: Anatomix, an anatomy-aware grounded multimodal large language model for chest x-ray interpretation. arXiv preprint arXiv:2601.03191 (2026)

    Pith/arXiv arXiv 2026

  11. [11]

    arXiv preprint arXiv:2003.10286 (2020)

    He, X., Zhang, Y., Mou, L., Xing, E., Xie, P.: Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 (2020)

    Pith/arXiv arXiv 2003

  12. [12]

    Scientific data6(1), 317 (2019)

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data6(1), 317 (2019)

    2019

  13. [13]

    Scientific data 5(1), 180251 (2018)

    Lau, J.J., Gayen, S., Ben Abacha, A., Demner-Fushman, D.: A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5(1), 180251 (2018)

    2018

  14. [14]

    In: Proceedings of the 2024 conference on empirical methods in natural language processing

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 5971– 5984 (2024)

    2024

  15. [15]

    Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., Wu, X.M.: Slake: A semantically- labeledknowledge-enhanceddatasetformedicalvisualquestionanswering.In:2021 IEEE 18th international symposium on biomedical imaging (ISBI). pp. 1650–1654. IEEE (2021)

    2021

  16. [16]

    Os- teoarthritis and Cartilage16(12), 1433–1441 (2008).https://doi.org/10.1016/ j.joca.2008.06.016

    Peterfy, C.G., Schneider, E., Nevitt, M.: The osteoarthritis initiative: report on the design rationale for the magnetic resonance imaging protocol for the knee. Os- teoarthritis and Cartilage16(12), 1433–1441 (2008).https://doi.org/10.1016/ j.joca.2008.06.016

    2008

  17. [17]

    arXiv preprint arXiv:2505.09388 (2025)

    Qwen Team: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  18. [18]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  19. [19]

    In: Biocomputing 2026: Proceedings of the Pacific Symposium

    Sambara, S., Kim, S.E., Zhang, X., Luo, L., Johri, S., Baharoon, M., Ro, D.H., Rajpurkar, P.: 3dreasonknee: Advancing grounded reasoning in medical vision lan- guage models. In: Biocomputing 2026: Proceedings of the Pacific Symposium. pp. 99–113. World Scientific (2025) 10 A. Yu et al

    2026

  20. [20]

    arXiv preprint arXiv:2507.05201 (2025)

    Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al.: Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025)

    Pith/arXiv arXiv 2025

  21. [21]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., et al.: Qwen2-vl technical report. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Radiology: Artificial Intelligence 5(5), e230024 (2023)

    Wasserthal, J., Breit, H.C., Meyer, M.T., Pradella, M., Hinck, D., Sauter, A.W., Heye, T., Boll, D.T., Cyriac, J., Yang, S., et al.: Totalsegmentator: robust segmen- tation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence 5(5), e230024 (2023)

    2023

  23. [23]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Wolf, D., Hillenhagen, H., Taskin, B., Bäuerle, A., Beer, M., Götz, M., Ropinski, T.: Your other left! vision-language models fail to identify relative positions in medical images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 691–701. Springer (2025)

    2025

  24. [24]

    IEEE Journal of Biomedical and Health Informatics (2025)

    Xin, Y., Ates, G.C., Gong, K., Shao, W.: Med3dvlm: An efficient vision-language model for 3d medical image analysis. IEEE Journal of Biomedical and Health Informatics (2025)

    2025