Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

AnatomiX argues that an X-ray model that first detects thoracic anatomy, then lets a language model reason over those detections, beats current grounding models by more than 25%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:21 UTC pith:ATX2VE7W

load-bearing objection Interesting architecture, but the descriptive grounding gains are suspect until the retrieval leak from the validation split is dealt with. the 4 major comments →

arxiv 2601.03191 v3 pith:ATX2VE7W submitted 2026-01-06 cs.CV cs.AIcs.LG

AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation

Anees Ur Rehman Hashmi , Numan Saeed , Christoph Lippert This is my paper
classification cs.CV cs.AIcs.LG
keywords multimodal large language modelchest X-ray interpretationanatomy groundingphrase groundinggrounded captioninggrounded diagnosiscontrastive retrievalmedical image understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the bottleneck in chest X-ray multimodal language models is anatomical understanding, not text generation. The proposed model, AnatomiX, splits the task into two stages: a perception module first identifies 36 thoracic structures and extracts a feature token and bounding box for each, and only then does a language model generate answers grounded in those anatomy-specific features. On four grounding tasks — localizing anatomical structures, localizing a described finding, and diagnosing or captioning a user-specified region — the authors report more than 25% gains over existing approaches, while staying competitive on report generation, visual question answering, and image classification. The design also keeps working when X-rays are horizontally flipped, which the authors attribute to genuine anatomy recognition rather than orientation shortcuts. If these results hold, explicit anatomy-first processing, rather than bigger models, is the path to reliable spatial reasoning in medical imaging.

Core claim

The central claim is that separating "what anatomy is visible" from "what the image shows overall" is what makes grounded chest X-ray interpretation work. AnatomiX's Anatomy Perception Module uses a DETR-style decoder with 36 fixed object tokens, one per thoracic structure, to predict bounding boxes and produce localized feature tokens; these features are aligned with sentence embeddings via a soft contrastive loss that tolerates co-occurring findings. At inference, the model retrieves a likely textual description for each anatomy from a compact vector database and feeds image tokens, anatomy tokens, boxes, and retrieved sentences into a language model trained with LoRA. The authors report t

What carries the argument

The load-bearing component is the Anatomy Perception Module (APM): a DETR-style decoder with 36 non-permutation-invariant object tokens, each hardwired to one thoracic anatomy. The decoder outputs a bounding box and a localized feature token per anatomy; a cross-attention feature extractor refines these tokens; a soft contrastive self-similarity loss aligns them with sentence embeddings while permitting overlapping findings; and at inference a lightweight vector database retrieves the most similar sentence per anatomy. These anatomy tokens, boxes, and retrieved sentences are projected into the LLM's embedding space and inserted into a structured prompt, so the language model reasons over 'ob

Load-bearing premise

The load-bearing premise is that the validation-set sentence database used during inference is a neutral source of context — if its sentences overlap with test-set phrasing, the language model can copy them and inflate the reported grounded-captioning and diagnosis scores.

What would settle it

Rerun the grounded captioning, grounded diagnosis, and report-generation evaluations with the vector database rebuilt exclusively from the Chest-ImaGenome training split (or with any test-derived sentences removed). If BERTScore, ROUGE, RadGraph-F1, or CheXbert-14-F1 drops substantially, the reported descriptive gains are partly retrieval leakage rather than anatomy understanding.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the two-stage anatomy-first design is the cause of the gains, existing single-step grounding models can be upgraded by adding an explicit anatomy detection stage rather than by scaling model size.
  • Grounded diagnosis and captioning improve by 25–30% on reported benchmarks, meaning clinicians could interrogate regional findings with coordinate-based questions and receive clinically relevant, localized answers.
  • Horizontal-flip robustness suggests localization depends on image content, not on radiological markers or orientation cues, which matters for real-world X-rays with variable laterality.
  • Ablations show retrieved textual sentences drive descriptive generation while anatomical tokens drive spatial tasks, indicating the two signal types are complementary.
  • Report generation reaches competitive scores with a roughly 1.5× smaller model than the top clinical-F1 systems, suggesting anatomy-focused design can substitute for parameter count.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The vector database is built from the validation split of Chest-ImaGenome (per the supplementary); this is not proven neutral. A test build from training-only sentences, or with test sentences removed, would clarify whether reported gains on grounded captioning and diagnosis come from anatomy awareness or from retrieving phrases that resemble test references.
  • The same two-stage anatomy-first structure could transfer to MRI or CT, where organ-level tokens could be defined by atlas or segmentation priors; the paper itself suggests MRI as future work.
  • The flipped-image experiment, with markers removed, is a practical robustness probe that could be standardized as a benchmark for laterality understanding in medical MLLMs.
  • Because the method feeds retrieved sentences directly into the prompt, an adversarial or unusual image could cause the retrieval step to supply a wrong but confident sentence; measuring retrieval accuracy per anatomy as a standalone diagnostic would make failure modes visible.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. AnatomiX is a two-stage multimodal large language model for chest X-ray interpretation. The Anatomy Perception Module (APM) detects 36 thoracic anatomical objects, produces per-object tokens and bounding boxes, aligns those tokens with text via a soft contrastive loss, and at inference retrieves a 'likely findings' sentence for each object from a vector database. These outputs are injected into a prompt for a LoRA-tuned MedGemma-4b LLM, which is evaluated on four grounding tasks, report generation, VQA, and image understanding. The paper claims over 25% improvement over prior art on anatomy grounding, phrase grounding, grounded diagnosis, and grounded captioning, and attributes this to the anatomy-first design and its robustness to horizontal flipping.

Significance. If the reported gains are reproducible and not driven by retrieval leakage, the anatomy-first two-stage design is a meaningful contribution: explicit anatomical object tokens and boxes give the LLM spatially grounded inputs that single-stage grounding models lack. The paper is also unusually extensive in its ablations and per-anatomy breakdowns (Tables S2–S8), which is a strength. However, the central descriptive-task claims rest on retrieval from a validation-set-derived vector database, and several numerical/formal inconsistencies in the current manuscript make the headline results difficult to verify. The significance is therefore contingent on resolving these points.

major comments (4)
  1. [Supp. S2; Table 1 (GD/GC rows)] The VDB is constructed from the validation split of Chest-ImaGenome (Supp. S2), while grounded diagnosis/captioning are evaluated on MS-CXR and VinDr-Instruct. The ablations in Table 1 show that removing retrieved sentences (AnatomiX-ˆSt) drops GD/GC CheXbert-14-F1 from 0.54/0.78 to 0.28/0.24 and BERTScore from 0.63/0.65 to 0.19/0.25. Because MS-CXR is derived from the same MIMIC-CXR/Chest-ImaGenome family, validation-set sentences can overlap in phraseology with MS-CXR test references, allowing the LLM to copy or closely paraphrase retrieved text and inflate descriptive metrics. The paper does not report disjointness between the VDB source split and these test sets, nor retrieval-overlap statistics. This is load-bearing for the >25% claim on GD/GC; please rebuild VDB from the training split (or an external corpus), re-report the results, and quantify phrase overlap between VDB entries a
  2. [§3.1, Eqs. (7)–(8) vs Supp. S1, Eqs. (S1)–(S4)] The main-text contrastive loss is not the loss implemented. Eq. (7) sets Sself = ˆSE ˆST_E without the l2 normalization and row-wise softmax used in Supp. Eq. (S1)–(S2), and Eq. (8) writes LCL as a sum over i of scalar log-ratio terms, which is dimensionally inconsistent with a matrix KL objective. The supplementary version (with temperature τ=0.01 and averaged bidirectional KL) is the one actually used. Please align the main text with the supplementary or explicitly say the main text is schematic.
  3. [Table 1 vs Table S3] The aggregate phrase-grounding numbers are not reproducible from the per-dataset results. Using test sizes from Table S1 (MS-CXR 528, PadChest-Gr 1121, VinDr-Instruct 6166), the weighted IoU/mAP from Table S3 is approximately 0.28/0.22, and the unweighted mean is 0.40/0.30; neither matches the reported 0.46/0.35. The aggregation rule (micro vs macro averaging, any post-processing, or additional datasets) must be stated so the table can be reproduced.
  4. [Tables S5–S8 (b–c), §5] The flipped-image comparison reports RadVLM IoU/mAP of 0.00 for essentially all left- and right-sided anatomical objects. Uniform zeros are atypical of a model with any localization ability and may reflect an evaluation artifact (e.g., coordinate-space mismatch after flipping, or failure to flip the input for the baseline). Please specify exactly how flipping was applied to inputs and outputs for each model, and show sample boxes from RadVLM on flipped images.
minor comments (4)
  1. [Supp. Fig. S3 caption] Typo: 'anaomical' should be 'anatomical'.
  2. [§4, first sentence after Table S1 reference] Grammar: 'Supp. Table S1 provides the summary' should read 'provides a summary' or 'provides the summaries'.
  3. [Equation numbering, §3.1] The loss symbol is written as LCL in Eq. (8) but L_CL in Eq. (9); make notation consistent.
  4. [Table 1 column headings] The column header 'BERT' should be 'BERTScore' to match the metric name used in the text and elsewhere.

Circularity Check

0 steps flagged

No circularity: AnatomiX's central claims are empirical benchmark results; the validation-split VDB is a leakage/correctness concern, not a circular derivation.

full rationale

Walking the derivation chain: APM (Sec. 3.1) is trained with box losses (Eqs. 2-3) and a soft contrastive loss (Eqs. 7-8); the LLM is LoRA-tuned with cross-entropy (Eq. 10). None of these equations defines a reported prediction in terms of the ground truth it is evaluated against; performance is empirical and measured on held-out splits of public benchmarks (MS-CXR, VinDr-Instruct, MIMIC-CXR, etc.). The anatomy-grounding and phrase-grounding results come from box prediction, not from retrieving target text. The only load-bearing memory is the VDB (Supp. S2), built from 'the validation set of the Chest-ImaGenome dataset,' whose retrieved sentences are inserted into the prompt as 'Likely findings' (Fig. S1); Table 1 ablations show that removing them sharply reduces descriptive-task scores. This is a legitimate evaluation-leakage/attribution risk—if validation phrases overlap with test phrasing, metrics can be inflated—but it is not circular: the VDB sentences are not the test labels, and no fitted parameter is renamed as a prediction. The paper also does not rely on any load-bearing self-citation chain; architectural choices are cited to non-overlapping prior work (DETR, MedGemma, etc.). Thus no reduction of a claimed result to its own inputs can be exhibited, so the circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper's central claim rests on a fixed anatomical ontology, a pretrained text encoder as the semantic ground truth, a newly introduced soft contrastive objective, and a validation-set-derived retrieval database that is used at inference. These are mostly domain assumptions rather than fully derived first-principles choices; each could affect the reported gains if violated.

free parameters (5)
  • N (number of anatomical slots) = 36
    Fixed to the 36 Chest-ImaGenome anatomy classes; the model cannot represent anatomy outside this set, so performance on unseen anatomies is undefined.
  • tau (contrastive temperature) = 0.01
    Set in Supp. eq. S3; no sensitivity analysis provided.
  • lambda1, lambda2 (box loss weights) = 5, 2
    Defaults inherited from DETR; not tuned for medical data.
  • LoRA rank/alpha = not reported
    LoRA config for LLM fine-tuning is not specified, so exact reproduction is impossible.
  • embedding projection dims (d, s, l) = not reported
    Dimensions of image tokens, anatomical tokens, and LLM embedding space are not given; the 'lower-dimensional space' size s is unspecified.
axioms (5)
  • domain assumption The 36 Chest-ImaGenome anatomical objects are a complete ontology for CXR interpretation across all test datasets (MS-CXR, PadChest, VinDr, MIMIC).
    The APM produces exactly N=36 slots; any finding localized to an anatomy outside this set must be mapped to one of these slots, which may bias grounding.
  • domain assumption BiomedBERT sentence embeddings capture clinical similarity such that the self-similarity matrix S_self is a valid target for visual-textual alignment.
    Introduced in Section 3.1 and Supp. S1; the loss uses this matrix as the only supervision beyond bounding boxes.
  • ad hoc to paper The soft contrastive KL loss (Supp. eq. S4) is a suitable objective for aligning overlapping radiological findings.
    This is a new loss formulation; no theoretical guarantee or external validation beyond APM ablations (Table S2) is given.
  • domain assumption The VDB retrieval returns the most relevant sentence for each anatomical object and injecting it into the prompt improves grounding without leaking test information.
    Section 3.1 and Supp. S2; if the database contains near-duplicates of test sentences, reported NLG gains may be inflated.
  • domain assumption Flipped-image evaluation is performed fairly, with consistent coordinate transforms and prompt semantics for baselines.
    The near-zero performance of RadVLM on flipped images is interpreted as a lack of anatomical understanding; an alternative explanation (e.g., a coordinate normalization mismatch) is not ruled out.

pith-pipeline@v1.3.0-alltime-deepseek · 22394 in / 21312 out tokens · 188598 ms · 2026-08-03T12:21:03.232647+00:00 · methodology

0 comments
read the original abstract

Multimodal medical large language models have shown substantial progress in chest X-ray interpretation but continue to face challenges in spatial reasoning and anatomical understanding. Although existing grounding techniques improve overall performance, they often fail to establish a true anatomical correspondence, resulting in incorrect anatomical understanding in the medical domain. To address this gap, we introduce AnatomiX, a multitask multimodal large language model for anatomically grounded chest X-ray interpretation. Inspired by the radiological workflow, AnatomiX adopts a two stage approach: first, it identifies anatomical structures and extracts their features, and then leverages a large language model to perform diverse downstream tasks such as phrase grounding, report generation, visual question answering, and image understanding. Extensive experiments across multiple benchmarks demonstrate that AnatomiX achieves superior anatomical reasoning and delivers over 25% improvement in performance on anatomy grounding, phrase grounding, grounded diagnosis and grounded captioning tasks compared to existing approaches. Code and pretrained model are available at https://aneesurhashmi.github.io/anatomix

Figures

Figures reproduced from arXiv: 2601.03191 by Anees Ur Rehman Hashmi, Christoph Lippert, Numan Saeed.

Figure 1
Figure 1. Figure 1: Comparison between AnatomiX and RadVLM [10] in anatomy understanding. (a) and (b) show both models predicting the disease on the correct side (color scheme: red for model’s output, green for all ground truth locations). (c), (d) and (e) show models’ outputs for the same image flipped on the vertical axis (left ↔ right), where RadVLM completely fails to recognize the correct anatomical object, while Anatomi… view at source ↗
Figure 2
Figure 2. Figure 2: Anatomy Perception Module (APM) architecture (a): The encoder E outputs image embedding Ip, while the decoder D and feature extraction module M output object bounding boxes yˆbox, and anatomical object tokens Oˆ A, respectively. Different colors in O˜ , O and Oˆ A represent specific anatomical objects. (b) shows the contrastive alignment using frozen sentence encoder S and self-similarity loss. (c): The ve… view at source ↗
Figure 3
Figure 3. Figure 3: (a): LM architecture. (b): Datasets and tasks. \mathrm {O}_A = \mathcal {M}(Q, K, V) = Softmax \left (\frac {Q K^T}{\sqrt {d}}\right ) V , \label {eq:anatomy_module} (4) where Q = OWQ is the query matrix, K = IpWK is the key matrix and V = IpWV is the value matrix. The output OA represent localized image features for the anatomical objects at the corresponding positions specified by the predicted bounding … view at source ↗
Figure 4
Figure 4. Figure 4: Sample input–output pairs and comparison with second best models on ground￾ing tasks. AnatomiX is compared with RadVLM [10] for phrase and anatomy ground￾ing, and with CheXagent [8] for grounded diagnosis and captioning. GT: Ground Truth. Box colors: blue: user input, green: ground truth, red: model’s output. CXRs and to describe the features present within those regions. This information allows clinicians… view at source ↗
Figure 5
Figure 5. Figure 5: AnatomiX output for anatomy and phrase grounding on flipped images (left ↔ right) with radiographic markers removed. rately localizes both anatomical regions and pathologies compared to the second best models RadVLM [10] and CheXagent [8]. Detailed anatomy-wise results for anatomy grounding task are given in the supp. Tables S5–S8 (a). To further evaluate anatomical understanding, we conduct phrase and ana… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Medical Image Spatial Grounding with Semantic Sampling

    cs.CV 2026-03 conditional novelty 6.0

    MIS-Ground stress-tests 3D medical spatial grounding in VLMs, and MIS-SemSam raises Qwen3-VL-32B accuracy on it by 13.06% via semantic-neighborhood decoding.

Reference graph

Works this paper leans on

37 extracted references · 13 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Mayo Clinic Proceedings: Digital Health (2024) 2

    Anisuzzaman, D., Malins, J.G., Friedman, P.A., Attia, Z.I.: Fine-tuning llms for specialized use cases. Mayo Clinic Proceedings: Digital Health (2024) 2

    2024

  2. [2]

    Bae, S., Kyung, D., Ryu, J., Cho, E., Lee, G., Kweon, S., Oh, J., Ji, L., Chang, E., Kim, T., et al.: Mimic-ext-mimic-cxr-vqa: A complex, diverse, and large-scale visual question answering dataset for chest x-ray images (2024) 9

    2024

  3. [3]

    In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization

    Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with im- proved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization. pp. 65–72 (2005) 11, 12

    2005

  4. [4]

    arXiv preprint arXiv:2406.04449 (2024) 2, 3, 10, 12

    Bannur, S., Bouzid, K., Castro, D.C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al.: Maira-2: Grounded radiology report generation. arXiv preprint arXiv:2406.04449 (2024) 2, 3, 10, 12

    Pith/arXiv arXiv 2024

  5. [5]

    In: European conference on computer vision

    Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: European conference on computer vision. pp. 1–21. Springer (2022) 9

    2022

  6. [6]

    In: European conference on computer vision

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020) 5

    2020

  7. [7]

    NEJM AI2(7), AIdbp2401120 (2025) 9

    de Castro, D.C., Bustos, A., Bannur, S., Hyland, S.L., Bouzid, K., Wetscherek, M.T., Sánchez-Valverde, M.D., Jaques-Pérez, L., Pérez-Rodríguez, L., Takeda, K., et al.: Padchest-gr: A bilingual chest x-ray dataset for grounded radiology report generation. NEJM AI2(7), AIdbp2401120 (2025) 9

    2025

  8. [8]

    arXiv preprint arXiv:2401.12208 (2024) 2, 3, 10, 11, 12, 13

    Chen, Z., Varma, M., Delbrouck, J.B., Paschali, M., Blankemeier, L., Van Veen, D., Valanarasu, J.M.J., Youssef, A., Cohen, J.P., Reis, E.P., et al.: Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208 (2024) 2, 3, 10, 11, 12, 13

    Pith/arXiv arXiv 2024

  9. [9]

    arXiv preprint arXiv:2210.12186 (2022) 11, 12

    Delbrouck, J.B., Chambon, P., Bluethgen, C., Tsai, E., Almusa, O., Langlotz, C.P.: Improving the factual correctness of radiology report generation with semantic rewards. arXiv preprint arXiv:2210.12186 (2022) 11, 12

    Pith/arXiv arXiv 2022

  10. [10]

    arXiv preprint arXiv:2502.03333 (2025) 2, 3, 10, 11, 12

    Deperrois, N., Matsuo, H., Ruipérez-Campillo, S., Vandenhirtz, M., Laguna, S., Ryser, A., Fujimoto, K., Nishio, M., Sutter, T.M., Vogt, J.E., et al.: Radvlm: A multitask conversational vision-language model for radiology. arXiv preprint arXiv:2502.03333 (2025) 2, 3, 10, 11, 12

    arXiv 2025

  11. [11]

    ACM Transactions on Computing for Healthcare (HEALTH) 3(1), 1–23 (2021) 6

    Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3(1), 1–23 (2021) 6

    2021

  12. [12]

    ICLR1(2), 3 (2022) 8, 9

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022) 8, 9

    2022

  13. [13]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Jiang, D., Ye, M.: Cross-modal implicit relation reasoning and aligning for text-to- image person retrieval. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2787–2797 (2023) 7

    2023

  14. [14]

    Scientific data6(1), 317 (2019) 8, 9 16 A

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data6(1), 317 (2019) 8, 9 16 A. Hashmi et al

    2019

  15. [15]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023) 3

    2023

  16. [16]

    Advances in Neural Information Processing Systems36, 28541–28564 (2023) 3

    Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems36, 28541–28564 (2023) 3

    2023

  17. [17]

    arXiv preprint arXiv:2505.02830 (2025) 3

    Li, Q., Cui, Z., Bae, S., Xu, J., Yuan, R., Zhang, Y., Feng, R., Shen, Q., Zhang, X., He, J., et al.: Aor: Anatomical ontology-guided reasoning for medical large multimodal model in chest x-ray interpretation. arXiv preprint arXiv:2505.02830 (2025) 3

    Pith/arXiv arXiv 2025

  18. [18]

    In: Text sum- marization branches out

    Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text sum- marization branches out. pp. 74–81 (2004) 11, 12

    2004

  19. [19]

    Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., Wu, X.M.: Slake: A semantically- labeledknowledge-enhanceddatasetformedicalvisualquestionanswering.In:2021 IEEE 18th international symposium on biomedical imaging (ISBI). pp. 1650–1654. IEEE (2021) 9

    2021

  20. [20]

    Advances in neural information processing systems36, 34892–34916 (2023) 1, 3

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023) 1, 3

    2023

  21. [21]

    Liu, J., Liu, Z., Cen, Z., Zhou, Y., Zou, Y., Zhang, W., Jiang, H., Ruan, T.: Can multimodal large language models understand spatial relations? arXiv preprint arXiv:2505.19015 (2025) 2

    Pith/arXiv arXiv 2025

  22. [22]

    arXiv preprint arXiv:1711.05101 (2017) 9

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 9

    Pith/arXiv arXiv 2017

  23. [23]

    arXiv preprint arXiv:2410.12694 (2024) 2, 3

    Luo, L., Tang, B., Chen, X., Han, R., Chen, T.: Vividmed: Vision language model with versatile visual grounding for medicine. arXiv preprint arXiv:2410.12694 (2024) 2, 3

    Pith/arXiv arXiv 2024

  24. [24]

    In: European Conference on Computer Vision

    Ma, C., Jiang, Y., Wu, J., Yuan, Z., Qi, X.: Groma: Localized visual tokeniza- tion for grounding multimodal large language models. In: European Conference on Computer Vision. pp. 417–435. Springer (2024) 2

    2024

  25. [25]

    In: European Conference on Computer Vision

    Müller, P., Kaissis, G., Rueckert, D.: Chex: Interactive localization and region description in chest x-rays. In: European Conference on Computer Vision. pp. 92–

  26. [26]

    Scientific Data9(1), 429 (2022) 9

    Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., Nguyen, D.B., Le, D.D., Pham, C.M., Tong, H.T., Dinh, D.H., et al.: Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations. Scientific Data9(1), 429 (2022) 9

    2022

  27. [27]

    Pellegrini, C., Özsoy, E., Busam, B., Navab, N., Keicher, M.: Radialog instruct dataset 3, 9, 12

  28. [28]

    arXiv preprint arXiv:2306.14824 (2023) 2

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F.: Kosmos- 2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023) 2

    Pith/arXiv arXiv 2023

  29. [29]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 7

    2021

  30. [30]

    arXiv preprint arXiv:2507.05201 (2025) 3, 8, 12 AnatomiX MLLM for Chest X-Ray Interpretation 17

    Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al.: Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025) 3, 8, 12 AnatomiX MLLM for Chest X-Ray Interpretation 17

    Pith/arXiv arXiv 2025

  31. [31]

    arXiv preprint arXiv:2004.09167 (2020) 11, 12, 13

    Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A.Y., Lungren, M.P.: Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arXiv preprint arXiv:2004.09167 (2020) 11, 12, 13

    Pith/arXiv arXiv 2004

  32. [32]

    Advances in Neural Information Processing Systems37, 20198–20224 (2024) 2

    Szot, A., Mazoure, B., Agrawal, H., Hjelm, R.D., Kira, Z., Toshev, A.: Ground- ing multimodal large language models in actions. Advances in Neural Information Processing Systems37, 20198–20224 (2024) 2

    2024

  33. [33]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Wolf, D., Hillenhagen, H., Taskin, B., Bäuerle, A., Beer, M., Götz, M., Ropinski, T.: Your other left! vision-language models fail to identify relative positions in medical images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 691–701. Springer (2025) 2, 4

    2025

  34. [34]

    arXiv preprint arXiv:2108.00316 (2021) 8, 9

    Wu, J.T., Agu, N.N., Lourentzou, I., Sharma, A., Paguio, J.A., Yao, J.S., Dee, E.C., Mitchell, W., Kashyap, S., Giovannini, A., et al.: Chest imagenome dataset for clinical reasoning. arXiv preprint arXiv:2108.00316 (2021) 8, 9

    Pith/arXiv arXiv 2021

  35. [35]

    Information Fusion p

    Xiao, H., Zhou, F., Liu, X., Liu, T., Li, Z., Liu, X., Huang, X.: A comprehensive surveyoflargelanguagemodelsandmultimodallargelanguagemodelsinmedicine. Information Fusion p. 102888 (2024) 1

    2024

  36. [36]

    arXiv preprint arXiv:1904.09675 (2019) 11, 12, 13

    Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019) 11, 12, 13

    Pith/arXiv arXiv 1904

  37. [37]

    Right lower lung shows pleural effusion and atelectasis

    Zou, K., Bai, Y., Chen, Z., Zhou, Y., Chen, Y., Ren, K., Wang, M., Yuan, X., Shen, X., Fu, H.: Medrg: Medical report grounding with multi-modal large language model. arXiv preprint arXiv:2404.06798 (2024) 3 18 A. Hashmi et al. Supplementary Material Multimodal Prompt User: You are a professional radiologist. I will provide you with context containing like...

    Pith/arXiv arXiv 2024