Pith. sign in

REVIEW 3 major objections 5 minor 49 references

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read A 200K-record unified corpus lifts medical grounding F1 from 10.6 to 85.6 through full fine-tuning of a general vision-language grounding model.

desk verdict A large, carefully built medical grounding dataset and a solid fine-tuning study, but the headline same-set F1 may be inflated by image-level splits on video-derived data; the cross-set result is the safer anchor. read the letter →

arxiv 2608.03322 v1 pith:Y6LVVNBE submitted 2026-08-04 cs.CV

classification cs.CV
keywords medicalvisualgroundingvision-languagemodelfine-tuningmultimodalimagingchain-of-thoughtmixture-of-expertsobjectlocalizationdatasetconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper constructs a unified medical visual grounding corpus of about 200,000 image-query-answer examples spanning CT, ultrasound, X-ray, and optical imaging, and uses it to fully fine-tune a general-purpose vision-language grounding model. It claims that this large-scale domain-specific supervision raises detection F1@IoU 0.50 from 10.64 to 85.59 on a held-out split, showing that fragmented medical localization resources can be harmonized into one open-ended grounding task. The paper also adds a 20,000-sample chain-of-thought subset that verbalizes visual evidence before coordinates, and reports that this improves cross-source generalization. A sympathetic reader would care because it suggests a practical path from closed-set medical detectors to models that localize targets described in free-form clinical language.

What carries the argument

The load-bearing machinery is the LocAnyMed-200K corpus: a harmonized set of 209,910 image-query-answer records that serialize box coordinates, point coordinates, and an explicit <box>none</box> token into one generative output language. The second piece is full-parameter fine-tuning of the pretrained 3-billion-parameter grounding model on that corpus, with optional deterministic routing of visual features through shared and modality-specific expert branches. The corpus supplies the supervision; the fine-tuning transfers the general grounding skill to medical images.

What would settle it

Recompute Same-set Det. Val. after re-splitting all sources at the patient or case level (using metadata where available and perceptual hashes otherwise), and check whether F1@IoU 0.50 stays near 85.59. If it collapses toward the 10.64 baseline, the central claim is refuted. A second check: run the trained model on a never-before-seen medical imaging modality and see if any localization transfer remains.

Watch

Extended reading notes

Core claim

The central discovery is that a general grounding model, fine-tuned from head to toe on a unified corpus of medical grounding records, can become a strong multimodal medical localizer. The corpus converts masks, boxes, points, and negative queries into a single serialized format, and the model learns box, point, and no-target outputs from the same objective. On the held-out split, F1@IoU 0.50 rises from 10.64 to 85.59 and F1 Mean from 5.72 to 69.15, exceeding both closed-set detectors trained on the same records and zero-shot general VLMs. The paper further claims that routing visual projections through shared plus modality-specific experts gives a small cross-source robustness gain, and tha

Load-bearing premise

The held-out evaluation split is free of source leakage, so that related images from the same patient, case, or scan do not appear in both training and validation; if they do, the reported F1 gain could come from memorization rather than generalization.

Editorial extensions

If this is right

  • Heterogeneous medical localization datasets can be merged into a single open-ended grounding task without task-specific heads or losses.
  • A generative grounding model can jointly handle boxes, points, and absent-target rejection, with negative-query accuracy reaching 92.9% after adaptation.
  • Visual-evidence rationales can be distilled into the model and improve cross-source robustness without degrading same-set accuracy.
  • Closed-set detectors trained on the same records still excel at strict IoU in some modalities, so generative grounding does not fully replace them yet.
  • Cross-set generalization remains much weaker than same-set performance, pointing to source shift as the next bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline 85.59 gain hinges on the held-out split being free of patient or case leakage; the paper uses deterministic image-level splits for some datasets without patient identifiers, so a stricter case-level split could lower the number. This is not a claim the paper makes, but a testable condition for its central result.
  • The modality-routed expert design suggests that sharing a large visual backbone across imaging modalities benefits from a small deterministic amount of modality-specific capacity; the same recipe may transfer to other multi-domain medical tasks.
  • The CoT rationales are generated by a general-purpose teacher and validated for format, not clinical truthfulness; a clinician rating study would be a natural next test.
  • The dataset's unified format could serve as a common benchmark for medical visual grounding, but the paper does not itself propose it as such.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper constructs LocAnyMed-200K, a unified medical visual grounding dataset of 209,910 image-query-answer records spanning CT, ultrasound, X-ray, and optical imaging, together with a 20,000-record CoT rationale subset. It fine-tunes LocateAnything-3B, introduces modality-routed shared/expert visual FFNs, and reports F1@IoU 0.50 of 85.59 on same-set held-out validation versus 10.64 for the base model, and 33.63 on a cross-set test. The central claim is that large-scale unified medical grounding supervision, optionally combined with visual-evidence rationale supervision, equips a general grounding model with effective medical localization and improves cross-source generalization.

Significance. If the results hold, the dataset and training recipe are a useful contribution: they unify heterogeneous detection annotations into a common instruction format, support negative queries and point targets, and show that full-parameter domain adaptation of a general grounding model transfers to medical imaging. Strengths include careful metric definition (dataset-level matching, explicit absent-target handling), checkpoint curves, modality-wise and macro-F1 diagnostics, and the public release of code. The cross-set evaluation, built from disjoint source datasets and checked by source names, basenames, and perceptual hashes, is a particularly honest estimate of generalization. However, the headline same-set number is vulnerable to split leakage, and some model-selection choices are made on the test split, so the quantitative claims need reinforcement.

major comments (3)
  1. [S1.2, Table 1] S1.2 states that datasets without patient identifiers use a deterministic image-level split, and that Kvasir-SEG is split at image level. Kvasir-SEG is a colonoscopy video dataset; adjacent frames of the same polyp can be near-duplicates while having different file hashes. Because the abstract's headline F1 (10.64→85.59) and the checkpoint selection in Table S5 both use Same-set Det. Val., this split may measure memorization of specific polyps rather than generalization. I request a near-duplicate analysis between train and Same-set Det. Val. (perceptual hashes or frame distances), especially for Kvasir-SEG and any ultrasound/X-ray sources split at image level, plus F1 after removing leaked records. The Cross-set result (33.63) is not subject to this same-source leakage, but it cannot validate the 85.59 headline.
  2. [§4.3, Table 2] The 2:2 MoE allocation is chosen because it "performs best" on Cross-set Det. Test, and the rationale-augmented model is then evaluated on the same split. This is test-set selection: the reported cross-set gain from rationale supervision (+1.59 F1@0.50, +1.83 F1 Mean) is measured on the set used to pick the allocation/checkpoint. Differences among allocations are small (Cross F1 Mean 19.51 vs 18.39/18.39), and no variance estimates are given. Select the allocation on Same-set validation or a nested split, or report the full matrix of MoE allocations with rationale; otherwise the cross-source advantage of 2:2 and of rationale supervision is not established.
  3. [§4.2, Table 2, S4.4] No repeated runs, confidence intervals, or significance tests are reported. Several decisive differences are small: e.g., Same-set F1 Mean among MoE 3:1/2:2/7:1 is 69.17/68.87/69.02, and Cross F1 Mean is 19.51/18.39/18.39. The paper itself concedes in S4.4 that the results "should be treated as comparative trends rather than definitive rankings without repeated-run variance estimates." Since the abstract's central claim is a single number, the authors should provide at least 3 seeds for the main LocAnyMed and the rationale variant and report means and standard deviations. This is needed to support the ranking claims and the claimed improvements.
minor comments (5)
  1. [§3.3] Two references to "Figure 3" in this section describe the architecture and the modality-routed FFN, but Figure 3 in the main text is the dataset-composition figure; the architecture is Figure 2. Please correct the cross-references.
  2. [Tables 1, S4] Several numeric entries are missing spaces due to formatting, e.g., "27.499.2816.02", "33.631.5819.12", and "73.9253.47". Please fix the table formatting.
  3. [§3.2] The abstract says "approximately 200K" while the corpus contains exactly 209,910 records. State both numbers for precision.
  4. [S1.2] The split protocol for ultrasound and X-ray sources without patient identifiers is described only generically. For reproducibility, specify which sources have official/patient-level splits and which use the deterministic image-level split.
  5. [Table S4] Cross-set CT F1 Mean decreases from 6.60 (LocateAnything-3B) to 6.02 (LocAnyMed), even though F1@0.50 improves. This modality-specific caveat should be mentioned when summarizing cross-source generalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical fine-tuning results measured against held-out data, not derivations that reduce to their inputs.

full rationale

The paper's main claims are empirical: fine-tuning LocateAnything-3B on LocAnyMed-200K improves F1@IoU 0.50 from 10.64 to 85.59 on Same-set Det. Val., and to 33.63 on Cross-set Det. Test. These are measurements on evaluation splits, not quantities derived by definition from the training data. No equation in the paper defines the reported F1 in terms of a fitted parameter or a self-citation. The CoT-20K subset is generated by GPT-5.5 as a teacher, but the original ground-truth coordinates are appended byte-for-byte and the grounding evaluation is independent of the teacher's rationales. The modality-routed experts are initialized from the original dense projection, so the architecture preserves the pretrained output at initialization; this is a construction fact, not a circular prediction. The paper's references to LocateAnything (Wang et al., 2026) are to a third-party base model, not to the authors' own prior work, so there is no load-bearing self-citation chain. One caveat, which is a generalization-validity concern rather than circularity, is that S4.2 selects the final checkpoint using Same-set Det. Val. F1 Mean, meaning the same-set numbers are somewhat optimistic as a held-out estimate; the cross-set evaluation provides a more independent anchor. This does not make the derivation circular, because the reported numbers still come from running the model on unseen records, not from re-stating the training objective or a fitted parameter. Overall, the derivation chain is self-contained and empirically grounded.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the held-out split being leak-free, the source annotations being trustworthy, and the known modality assumption for the MoE variant. The free parameters are model-selection choices rather than fitted physical constants.

free parameters (5)
  • shared-to-expert channel allocation = 2:2 (ds=576, de=576)
    Selected from {7:1, 3:1, 2:2} based on best Cross-set Det. Test F1 Mean; the paper acknowledges differences are small and should be treated as trends (Section 4.3, S4.4).
  • training checkpoint = step 1500 for dense SFT and MoE
    Selected by Same-set Det. Val. F1 Mean peak (Table S5); cross-set at this step is not the maximum.
  • target-present to absent-target ratio = approximately 3:1
    Chosen when constructing negative queries for detection records (S1.2); affects class balance and negative accuracy.
  • CoT subset size = 20,000 records
    Chosen as a rationale-augmented subset, with modality distribution fixed; no sensitivity analysis.
  • CoT rationale length = 20-45 words
    Prompt constraint for GPT-5.5 teacher (Table S11); not varied.
assumptions (5)
  • domain assumption Training and validation splits are independent and identically distributed; no same-patient leakage.
    Required for the held-out F1 to measure generalization; the split protocol in S1.2 uses image-level splits when patient identifiers are unavailable.
  • domain assumption Source dataset annotations are complete enough that unannotated images can be treated as true negatives.
    Negative queries are constructed from images in datasets without the target annotation; if a target is present but unannotated, the 'absent' label is wrong. Invoked in S1.2.
  • domain assumption The imaging modality is known and reliable at inference.
    Modality-routed experts require a valid modality label for each image (S2.1); a missing or wrong modality label raises an error.
  • domain assumption Normalized coordinates in [0,1000] remain aligned because the model resizes without cropping.
    Coordinate conversion (Eq. 1) and the model's processor behavior are assumed to preserve alignment; stated in S1.3.
  • domain assumption GPT-5.5-generated rationales are a reliable supervision source.
    CoT-20K is distilled from GPT-5.5 with automated validation but only a 100-record manual pilot; questionable rationales could bias the model (S5.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LocAnyMed: Vision-Language Grounding for Multimodal Medical Images." pith.science (2026). https://pith.science/paper/Y6LVVNBE

@misc{pith2026260803322,
  author       = {Pith},
  title        = {Pith review of: LocAnyMed: Vision-Language Grounding for Multimodal Medical Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y6LVVNBE}},
  note         = {Machine review of arXiv:2608.03322}
}
read the original abstract

Medical visual grounding connects free-form clinical queries to spatial evidence in medical images and is an important component of interpretable medical artificial intelligence. However, general-purpose grounding models are predominantly trained on natural images, while existing medical localization resources remain fragmented across imaging modalities, datasets, and task formulations. To address this gap, we construct LocAnyMed-200K, a multimodal medical visual grounding dataset containing approximately 200K image-query-answer examples across computed tomography, optical medical imaging, ultrasound, and X-ray. We harmonize heterogeneous detection and localization resources into a unified free-form instruction format that supports one or multiple bounding boxes, point coordinates, and no-target outputs for negative queries. Full-parameter fine-tuning of LocateAnything-3B on LocAnyMed-200K improves F1@IoU 0.50 from 10.64 to 85.59 on a held-out evaluation split, demonstrating that large-scale domain-specific supervision can equip a general grounding model with effective medical localization capabilities. Beyond spatial coordinates, a clinically interpretable grounding system should also communicate the evidence supporting its prediction. We therefore derive LocAnyMed-CoT-20K, a rationale-augmented subset that connects anatomical context, visual observations, and spatial conclusions through structured reasoning and further improves cross-source generalization through fine-tuning. Together, these resources provide a unified foundation for studying both localization accuracy and rationale quality across heterogeneous medical imaging modalities. The code is publicly available at https://github.com/MiliLab/LocAnyMed.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 25 canonical work pages

  1. [1]

    International conference on medical image computing and computer-assisted intervention , pages=

    nnDetection: a self-configuring method for medical object detection , author=. International conference on medical image computing and computer-assisted intervention , pages=. 2021 , organization=

  2. [2]

    International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=

    Medical phrase grounding with region-phrase context contrastive alignment , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2023 , organization=

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

    Uncertainty-aware medical diagnostic phrase identification and grounding , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

  4. [4]

    arXiv preprint arXiv:2605.27365 , year=

    LocateAnything: Fast and high-quality vision-language grounding with parallel box decoding , author=. arXiv preprint arXiv:2605.27365 , year=

  5. [5]

    Nature communications , volume=

    Segment anything in medical images , author=. Nature communications , volume=. 2024 , publisher=

  6. [6]

    Nature methods , volume=

    A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities , author=. Nature methods , volume=. 2025 , publisher=

  7. [7]

    international conference on medical image computing and computer-assisted intervention , pages=

    M4oe: A foundation model for medical multimodal image segmentation with mixture of experts , author=. international conference on medical image computing and computer-assisted intervention , pages=. 2024 , organization=

  8. [8]

    arXiv preprint arXiv:2506.08356 , year=

    Medmoe: modality-specialized mixture of experts for medical vision-language understanding , author=. arXiv preprint arXiv:2506.08356 , year=

Show all 49 references
  1. [9]

    arXiv preprint arXiv:2506.04034 , year=

    Rex-thinker: Grounded object referring via chain-of-thought reasoning , author=. arXiv preprint arXiv:2506.04034 , year=

  2. [10]

    arXiv preprint arXiv:2510.04477 , year=

    MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models , author=. arXiv preprint arXiv:2510.04477 , year=

  3. [11]

    Journal of medical imaging , volume=

    DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning , author=. Journal of medical imaging , volume=. 2018 , publisher=

  4. [12]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Mdetr-modulated detection for end-to-end multi-modal understanding , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  5. [13]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Grounded language-image pre-training , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  6. [14]

    European conference on computer vision , pages=

    Simple open-vocabulary object detection , author=. European conference on computer vision , pages=. 2022 , organization=

  7. [15]

    European conference on computer vision , pages=

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection , author=. European conference on computer vision , pages=. 2024 , organization=

  8. [16]

    International Conference on Learning Representations , volume=

    Grounding multimodal large language models to the world , author=. International Conference on Learning Representations , volume=

  9. [17]

    arXiv preprint arXiv:2510.12798 , year=

    Detect anything via next point prediction , author=. arXiv preprint arXiv:2510.12798 , year=

  10. [18]

    2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages=

    MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities , author=. 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages=. 2026 , organization=

  11. [19]

    arXiv preprint arXiv:2410.12694 , year=

    Vividmed: Vision language model with versatile visual grounding for medicine , author=. arXiv preprint arXiv:2410.12694 , year=

  12. [20]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  13. [21]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Towards a multimodal large language model with pixel-level insight for biomedicine , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  14. [22]

    Nature Communications , year=

    A universal foundation model for grounded biomedical image interpretation , author=. Nature Communications , year=

  15. [23]

    European conference on computer vision , pages=

    Making the most of text semantics to improve biomedical vision--language processing , author=. European conference on computer vision , pages=. 2022 , organization=

  16. [24]

    arXiv preprint arXiv:2601.06847 , year=

    MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data , author=. arXiv preprint arXiv:2601.06847 , year=

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    Uni-med: a unified medical generalist foundation model for multi-task learning via connector-MoE , author=. Advances in Neural Information Processing Systems , volume=

  18. [26]

    arXiv preprint arXiv:2505.14231 , year=

    Univg-r1: Reasoning guided universal visual grounding with reinforcement learning , author=. arXiv preprint arXiv:2505.14231 , year=

  19. [27]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Gemex: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  20. [28]

    arXiv preprint arXiv:2603.06665 , year=

    Better eyes, better thoughts: Why vision chain-of-thought fails in medicine , author=. arXiv preprint arXiv:2603.06665 , year=

  21. [29]

    Nature communications , volume=

    The medical segmentation decathlon , author=. Nature communications , volume=. 2022 , publisher=

  22. [30]

    Radiology: Artificial Intelligence , volume=

    TotalSegmentator: robust segmentation of 104 anatomic structures in CT images , author=. Radiology: Artificial Intelligence , volume=. 2023 , publisher=

  23. [31]

    Medical image analysis , volume=

    Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge , author=. Medical image analysis , volume=. 2017 , publisher=

  24. [32]

    International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=

    Beyond adapting SAM: Towards end-to-end ultrasound image segmentation via auto prompting , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2024 , organization=

  25. [33]

    Medical physics , volume=

    BUS-BRA: A breast ultrasound dataset for assessing computer-aided diagnosis systems , author=. Medical physics , volume=. 2024 , publisher=

  26. [34]

    Scientific data , volume=

    A pediatric wrist trauma X-ray dataset (GRAZPEDWRI-DX) for machine learning , author=. Scientific data , volume=. 2022 , publisher=

  27. [35]

    Scientific data , volume=

    Fracatlas: A dataset for fracture classification, localization and segmentation of musculoskeletal radiographs , author=. Scientific data , volume=. 2023 , publisher=

  28. [36]

    arXiv preprint arXiv:2305.19112 , year=

    Dentex: An abnormal tooth detection with dental enumeration and diagnosis benchmark for panoramic x-rays , author=. arXiv preprint arXiv:2305.19112 , year=

  29. [37]

    International conference on multimedia modeling , pages=

    Kvasir-seg: A segmented polyp dataset , author=. International conference on multimedia modeling , pages=. 2019 , organization=

  30. [38]

    saliency maps from physicians , author=

    WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians , author=. Computerized medical imaging and graphics , volume=. 2015 , publisher=

  31. [39]

    Information Sciences , volume=

    Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening , author=. Information Sciences , volume=. 2019 , publisher=

  32. [40]

    IEEE transactions on medical imaging , volume=

    Ridge-based vessel segmentation in color images of the retina , author=. IEEE transactions on medical imaging , volume=. 2004 , publisher=

  33. [41]

    IEEE Transactions on Medical imaging , volume=

    Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response , author=. IEEE Transactions on Medical imaging , volume=. 2000 , publisher=

  34. [42]

    European conference on computer vision , pages=

    End-to-end object detection with transformers , author=. European conference on computer vision , pages=. 2020 , organization=

  35. [43]

    arXiv preprint arXiv:2606.03748 , year=

    Ultralytics YOLO26: unified real-time end-to-end vision models , author=. arXiv preprint arXiv:2606.03748 , year=

  36. [44]

    arXiv preprint arXiv:2406.03459 , year=

    Lw-detr: A transformer replacement to yolo for real-time detection , author=. arXiv preprint arXiv:2406.03459 , year=

  37. [45]

    arXiv preprint arXiv:2603.18739 , year=

    EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation , author=. arXiv preprint arXiv:2603.18739 , year=

  38. [46]

    arXiv preprint arXiv:2412.10302 , year=

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding , author=. arXiv preprint arXiv:2412.10302 , year=

  39. [47]

    arXiv preprint arXiv:2512.17436 , year=

    Xiaomi MiMo-VL-Miloco Technical Report , author=. arXiv preprint arXiv:2512.17436 , year=

  40. [48]

    5 technical report , author=

    Ovis2. 5 technical report , author=. arXiv preprint arXiv:2508.11737 , year=

  41. [49]

    arXiv preprint arXiv:2511.21631 , year=

    Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.