Pith. sign in

REVIEW 3 major objections 5 minor 69 references

Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MedTrim replaces pairwise contrastive alignment with meta-entity-guided triplet mining to align chest X-ray images and radiology reports, improving retrieval and zero-shot classification.

desk verdict MedTrim is a solid, well-ablated triplet-mining method for med-VLM alignment, but its fine-grained retrieval gains are entangled with an unvalidated extractor and a couple of internal inconsistencies need cleanup. read the letter →

arxiv 2504.15929 v2 pith:ZTHIKNIO submitted 2025-04-22 cs.CV cs.AI

Saban Ozturk , Melih B. Yilmaz , Muti Kara , M. Talat Yavuz , Aykut Koç , Tolga Çukur This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodalalignmentvision-languagemodeltripletminingchestX-rayradiologyreportsmeta-entityzero-shotclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that aligning chest X-ray images with radiology reports works better when the alignment is guided not just by disease class but by fine-grained pathology attributes, and that this can be done by replacing pairwise contrastive learning with triplet learning. It introduces MedTrim, which first extracts structured meta-entities—disease names plus adjectival and directional descriptors such as 'severe' or 'upper lobe'—from radiology reports using an ontology-based rule module. A new score function measures overlap of these meta-entities between samples, and triplet mining selects for each anchor a highly similar positive and a moderately similar semi-hard negative. A multimodal triplet objective then pulls matching image-text pairs together and pushes mismatched ones apart, in both within-modal and cross-modal directions. On public chest X-ray data, MedTrim reports higher retrieval precision and higher zero-shot classification accuracy, AUC, and F1 than contrastive and knowledge-guided baselines, which is why a sympathetic reader would care: fine-grained attributes are clinically important but usually lost in class-level contrastive alignment.

What carries the argument

The load-bearing object is the meta-entity score function $$\mathrm{score}(m_i,m_j) = \frac{\delta_d(d_i\cap d_j)}{|d_i\cup d_j|} \sum_{d[q]\in d_i\cap d_j} \frac{\gamma_0+\gamma_1 JI_{\mathrm{adj}}(d[q])+\gamma_2 JI_{\mathrm{dir}}(d[q])}{\gamma_0+\gamma_1\delta_{\mathrm{adj}}(d[q])+\gamma_2\delta_{\mathrm{dir}}(d[q])},$$ a normalized weighted Jaccard similarity that respects the hierarchy disease to adjective to direction. The indicator $\delta_d$ zeroes out pairs with no shared disease; the Jaccard indices $JI_{\mathrm{adj}}$ and $JI_{\mathrm{dir}}$ measure descriptor overlap; the denominator keeps each disease term in $[0,1]$. This score selects triplets: the positive maximizes it, and the negative minimizes it subject to $\tau_{\min}\le \mathrm{score}\le\tau_{\max}$. The same entities are then used in the objective terms $f_{\mathrm{tri}}(E_A,E_P,E_N)=\max(0,\cos(E_A,E_P)-\cos(E_A,E_N)+\alpha)$, applied to both within-modal and cross-modal embedding pairs and blended by $\eta$.

What would settle it

Take a held-out set of radiology reports, have radiologists annotate the disease, adjective, and direction attributes, and compare the ontology extractor's output against them; if agreement is low, the triplet selection and the consistency metrics that measure MedTrim's advantage are both driven by the same possibly flawed extractor, and the reported retrieval and classification gains would need to be re-checked with an independent extractor.

Watch

Extended reading notes

Core claim

The central discovery is that explicit triplet learning over structured pathology attributes can replace pairwise contrastive learning for aligning medical images and text, and that the gain is concentrated in fine-grained attributes that class-level contrastive learning flattens. MedTrim extracts from each radiology report a structured list of meta-entities $m=(d,\mathrm{adj}(d),\mathrm{dir}(d))$ via an ontology-based rule module, and defines a score function that returns a weighted Jaccard similarity over shared disease classes and their adjectival and directional descriptors. The mining step then picks for every anchor a positive sample with maximum score and a semi-hard negative sample whose score lies in a middle band, so negatives share at least one disease class but mismatch descriptors. The training objective applies a cosine triplet loss separately to image-to-text, text-to-image, image-to-image, and text-to-text embeddings. On a large public chest X-ray collection, MedTrim reports higher retrieval precision across all four retrieval directions and higher zero-shot accuracy, AUC, and F1 on two external chest X-ray classification benchmarks than contrastive and knowledge-guided baselines.

Load-bearing premise

The whole pipeline assumes that the ontology-based extractor correctly recognizes disease names and their adjective and direction descriptors in radiology reports, but its accuracy is never tested against a human-annotated reference.

Editorial extensions

If this is right

  • Semi-hard negatives selected by entity overlap are the main driver of fine-grained discrimination: increasing their share in the mined triplets raises retrieval precision across all four retrieval tasks.
  • The joint within-modal and cross-modal objective is necessary: dropping either family of triplet terms measurably lowers precision, with the full four-term objective outperforming both within-only and cross-only variants.
  • Guidance from adjective and direction descriptors transfers to retrieval consistency in those attributes: removing adjective guidance drops adjective consistency, and removing direction guidance drops direction consistency in the ablations.
  • Zero-shot classification on external benchmarks improves along with retrieval, indicating that the aligned embeddings generalize beyond the training distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same ontology-based extractor serves as both the triplet selector and the evaluation oracle for descriptor consistency, re-scoring the same retrieval results with independently annotated attributes would isolate how much of MedTrim's advantage is representational rather than inherited from the extractor.
  • The same score-and-triplet machinery should transfer to other imaging modalities or languages, but only after rebuilding the ontology; continuous attributes such as size or severity percentiles would require a variant of the Jaccard weighting rather than exact set overlap.
  • A plausible but untested benefit is better few-shot generalization to rare or unseen disease classes, since semi-hard negatives force the model to separate shared class from differing descriptors; the paper does not run few-shot experiments.
  • The score function is additive over descriptor types, so the framework can absorb additional attribute ontologies such as contour, extent, or temporal change by adding new Jaccard terms and adjusting the $\gamma$ weights.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MedTrim, an alignment method for medical vision-language models on chest X-rays. It first extracts disease, adjective, and direction meta-entities from radiology reports using an ontology-based rule module (OBER), then uses these entities in a score function (Eq. 1) to mine triplets with positive and semi-hard negative samples. A multimodal triplet objective (Eq. 17) combines within-modal (I2I, T2T) and cross-modal (I2T, T2I) losses. Experiments on MIMIC-CXR evaluate P@R retrieval consistency for disease, adjective, and direction, and zero-shot classification on CheXpert and RSNA, reporting gains over ConVIRT, JoImTeRNet, GLoRIA, CheXzero, LIMITR, and MedFILIP. The paper concludes that MedTrim elicits superior retrieval and classification performance on CXR data over state-of-the-art alignment methods.

Significance. MedTrim is a sensible and technically clean proposal: explicit triplet mining with fine-grained entity guidance plus within-modal objectives is a reasonable departure from pairwise contrastive methods, and the paper ships code and thorough ablations that support the design choices (Tables 1, 2, and 4, Figs. 3 and 4). If the fine-grained retrieval results are correct, the method offers a practically useful improvement in CXR image-text alignment. However, the adjective and direction retrieval evidence is connected to OBER in both training and evaluation, and OBER's accuracy is never established; this currently tempers the significance of the central claim. The classification results (Table 6) use independent CheXpert and RSNA labels and are therefore more robust, showing small but consistent gains over the compared baselines.

major comments (3)
  1. [Sec. 3.2, Sec. 4.4, Table 5] OBER is used both to mine triplets (Algorithm 1 and Eq. 1) and to compute the adjective and direction Precision@R for every competing method, but no validation of OBER's extraction accuracy is reported. The paper does not give precision/recall against human annotations, CheXpert labels, or a published CXR named-entity benchmark. A systematic OBER bias (for example, missing adjectives such as 'mild' or misassigning directional phrases) would train MedTrim to imitate that bias and then reward it on the same yardstick, while baselines such as MedFILIP and LIMITR are scored with an extractor they never saw. The Adj. and Dir. columns of Table 5, which carry the fine-grained retrieval claim in Sec. 7, are therefore not trustworthy without an external yardstick. I request: (i) OBER precision/recall on a manually annotated subset of MIMIC-CXR reports or against a published CXR NER benchmark; (ii) a sensitivity check in which adjective and direction P@R are recomputed with an independent extractor or human labels; and (iii) reporting of how often OBER returns empty adj/dir sets and how those cases affect the denominator in Eq. 1.
  2. [Sec. 3.3, Eqs. (5) and (6)] The text states that delta_adj and delta_dir 'indicate whether the union set of adjectival and directional descriptors ... is non-empty', but Eqs. (5) and (6) define them with sgn(|adji(d[q]) intersect adjj(d[q])|) and sgn(|diri(d[q]) intersect dirj(d[q])|), i.e., intersection non-emptiness. If both samples mention the disease but use disjoint adjectives, the intersection is empty and delta becomes 0, so the denominator in Eq. 1 is gamma0 rather than gamma0+gamma1+gamma2. The claimed normalization of the weighted score to [0,1] is then false, and the effective weighting of adjective and direction matches changes with vocabulary overlap. Because Eq. 1 drives all triplet mining, this inconsistency is load-bearing and must be fixed by changing the definitions to union non-emptiness (or by amending the text and re-deriving the normalization claim).
  3. [Sec. 4.2 vs. Sec. 5.1, Fig. 4a] The implementation details state that 1,250,000 unique triplets were mined from the training set, while the ablation study in Fig. 4a concludes that approximately 0.25 million triplets strikes the optimal balance between retrieval precision and training efficiency. It is not clear which configuration produced Tables 5 and 6. If the final model used 1.25 million triplets, the choice contradicts the ablation-based recommendation; if it used 0.25 million, the implementation section should say so. This inconsistency affects both the efficiency claim and the comparability of training budgets across methods, so it should be clarified and justified.
minor comments (5)
  1. [Algorithm 1] Algorithm 1 takes the CXR image I as input but never uses it; remove I from the signature or explain why it is listed.
  2. [Sec. 3.3, Eq. (1)] In the sentence defining the score function, 'di' and 'dy' are inconsistent notation; the second set should be dj to match the equation.
  3. [Sec. 4.4] The captions of Figs. 5 and 6 refer to 'similarity scores to the ground-truth CXR image/report', but this quantity is not defined in Sec. 4.4; please specify how the query-to-ground-truth cosine similarity is computed and displayed.
  4. [Sec. 5.2] The claim of statistical significance (p < 0.05) would benefit from a precise statement of the Wilcoxon test protocol, including which sets of comparisons were pooled and whether any multiple-comparison correction was applied.
  5. [Sec. 6] The discussion lists useful limitations, but it does not address the lack of OBER validation even though OBER is the central knowledge-extraction component; this should be acknowledged as a limitation and connected to the suggested future work.

Circularity Check

1 steps flagged · score 4.0 of 10

Fine-grained Adj./Dir. retrieval is scored with the same OBER extractor that builds MedTrim's triplet supervision, making those gains self-referential; classification results remain externally grounded.

  1. other [Sec. 3.2 (Alg. 1), Sec. 3.3 (Eqs. 1-3), Sec. 4.4]
    "To address this gap, we introduce a new ontology-based recognition module (OBER) for high-fidelity extraction of meta-entities from CXR reports as outlined in Alg. 1. ... MedTrim employs Eq. 1 to compute the similarity scores between A and remaining samples within the mini-batch. ... The level of consistency between the meta-entities of query and retrieved items was measured via Jaccard index."

    The same OBER outputs (Alg. 1) are consumed twice: Eq. 1's JI_adj/JI_dir terms select the training triplets, and Sec. 4.4's Jaccard consistency scores define the Adj./Dir. P@R evaluation. A model optimized to place high-Jaccard pairs close in embedding space will tend to rank OBER-consistent items first, so the resulting P@R measures reproduction of OBER's attribute decisions rather than agreement with an external ground truth. Since OBER is never validated against human annotations or an independent extractor, this part of the retrieval evaluation is self-referential. The CheXpert/RSNA classification metrics use independent labels, so the circularity is partial and confined to the fine-grained attribute claims.

full rationale

The main derivation chain—OBER meta-entity extraction, Jaccard-based score (Eq. 1), triplet mining, and the multimodal triplet objective (Eq. 17)—is internally coherent and does not rename or derive a known result. The zero-shot classification evaluations on CheXpert and RSNA use external disease labels, so the headline classification claim is not circular. Disease-level retrieval comparisons also rest on standard disease classes, giving the retrieval claim independent content. The one genuinely self-referential element is the fine-grained adjective/direction evaluation: OBER supplies both the triplet-supervision signal (through Eq. 1's JI_adj and JI_dir terms) and the Sec. 4.4 consistency measure, and OBER's extraction accuracy is never tested against human annotations or an external extractor. That makes the Adj./Dir. P@R improvements an agreement-with-own-extractor score rather than a validation of true attribute alignment. This is a partial circularity: it does not infect the independently labeled classification results, but it limits the evidentiary value of the fine-grained retrieval claims. Self-citations in the reference list are background citations and are not load-bearing.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several fitted weights, hand-curated vocabulary lists, and an unvalidated entity extractor. The gamma weights, tau bounds, alpha, eta, and the 1.25M triplet count are chosen by cross-validation or ablation. OBER's ontology is curated by the authors without a reported accuracy measurement. The evaluation of adjective and direction consistency uses the same OBER extraction, which is a self-referential component.

free parameters (5)
  • gamma_0, gamma_1, gamma_2 = 0.85, 0.1, 0.05
    Weights for disease, adjective, and direction terms in the entity similarity score (Eq. 1), selected via cross-validation (Sec. 4.2).
  • tau_min, tau_max = 0.25, 0.6
    Score bounds for semi-hard negative selection in triplet mining, selected via cross-validation (Sec. 4.2).
  • alpha = 0.3
    Triplet margin in Eq. 12, selected via cross-validation (Sec. 4.2).
  • eta = 0.5
    Balance between cross-modal and within-modal loss terms in Eq. 17, selected via cross-validation (Sec. 4.2).
  • number of training triplets = 1,250,000
    Final model mines 1.25M triplets (Sec. 4.2) despite the ablation (Sec. 5.1) suggesting 0.25M is an efficient optimum.
assumptions (6)
  • domain assumption Pre-trained ViT and BioClinicalBERT encoders provide suitable initial representations for CXR alignment (Sec. 3.4).
    The method fine-tunes these encoders; their transfer capability is assumed.
  • ad hoc to paper OBER's hand-curated ontology (12 diseases, 98 adjectives, 4 directions, plus splitting and delete words) sufficiently covers radiology report vocabulary (Sec. 3.2).
    No recall or precision evaluation of OBER is reported.
  • ad hoc to paper The score function in Eq. 1, including the delta normalization, meaningfully orders sample similarity for triplet mining (Sec. 3.3).
    The prose and Eq. 5 disagree on whether delta uses union or intersection, so this is an unverified choice.
  • standard math Cosine-similarity triplet loss with margin alpha improves embedding alignment (Eq. 12).
    Standard triplet loss, widely used in representation learning.
  • ad hoc to paper Meta-entity consistency measured with OBER is a valid proxy for fine-grained alignment quality (Sec. 4.4).
    Self-referential; OBER is not compared with human annotations.
  • domain assumption CheXpert and RSNA zero-shot classification is a fair evaluation of the aligned embeddings (Sec. 5.2.2).
    Assumes label vocabularies overlap with MIMIC-CXR training labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models." pith.science (2026). https://pith.science/paper/ZTHIKNIO

@misc{pith2026250415929,
  author       = {Pith},
  title        = {Pith review of: Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZTHIKNIO}},
  note         = {Machine review of arXiv:2504.15929}
}
read the original abstract

Diagnostic imaging relies on interpreting both images and radiology reports, but the growing data volumes place significant pressure on medical experts, yielding increased errors and workflow backlogs. Medical vision-language models (med-VLMs) have emerged as a powerful framework to efficiently process multimodal imaging data, particularly in chest X-ray (CXR) evaluations, albeit their performance hinges on how well image and text representations are aligned. Existing alignment methods, predominantly based on contrastive learning, prioritize separation between disease classes over segregation of fine-grained pathology attributes like location, size or severity, leading to suboptimal representations. Here, we propose MedTrim (Meta-entity-driven Triplet mining), a novel method that enhances image-text alignment through multimodal triplet learning synergistically guided by disease class as well as adjectival and directional pathology descriptors. Unlike common alignment methods that separate broad disease classes, MedTrim leverages structured meta-entity information to preserve subtle but clinically significant intra-class variations. For this purpose, we first introduce an ontology-based entity recognition module that extracts pathology-specific meta-entities from CXR reports, as annotations on pathology attributes are rare in public datasets. For refined sample selection in triplet mining, we then introduce a novel score function that captures an aggregate measure of inter-sample similarity based on disease classes and adjectival/directional descriptors. Lastly, we introduce a multimodal triplet alignment objective for explicit within- and cross-modal alignment between samples sharing detailed pathology characteristics. Our demonstrations indicate that MedTrim improves performance in downstream retrieval and classification tasks compared to state-of-the-art alignment methods.

Figures

Figures reproduced from arXiv: 2504.15929 by the authors.

Figure 1
Figure 1. Multimodal CXR data carry key attributes pertaining to disease class, and adjectival and directional descrip [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. a) Given a batch of multimodal CXR samples, MedTrim first deploys its OBER module to identify disease labels along with adjectival and directional descriptors of corresponding pathology from radiology reports. b) Using these pathology attributes as meta entities, MedTrim performs triplet mining to select positive and semi-hard negative samples via entity-based weighting with respect to the anchor sample. The embeddi… view at source ↗
Figure 3
Figure 3. Distribution of retrieval performance across disease classes. Results shown for (a) I2T, (b) T2I retrieval tasks for disease class. MedTrim was compared against Contrastive and Triplet variants (see legend). were curated. Hyperparameters for triplet mining were selected as γ0 = 0.85, γ1 = 0.1, γ2 = 0.05, τmin = 0.25, τmax = 0.6, α = 0.3, and η = 0.5 via cross-validation. 4.3 Competing Methods MedTrim was comparative… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) Ablation study on the number of mined triplets. P@50 (%; left y-axis) in I2I, I2T, T2I, and T2T tasks are plotted as a function of the number of triplets, along with the training time (min; right y-axis). (b) Ablation study on the percentage of semi-hard negative s…
Figure 5
Figure 5. Figure 5: Representative results for T2I retrieval. Given a query CXR report, the top-ranked images retrieved by competing methods are displayed, along with their similarity scores to the ground-truth CXR image. For each method, meta-entities extracted from the reports of the re…
Figure 6
Figure 6. Figure 6: Representative results for I2T retrieval. Given a query CXR image, the top-ranked reports retrieved by competing methods are displayed, along with their similarity scores to the ground-truth CXR report. For each method, meta-entities in the retrieved reports are highli…
Figure 7
Figure 7. Figure 7: Representative results for zero-shot disease classification. For an input CXR image, Grad-CAM heatmaps (see colorbar) highlighting regions with high contribution to the classification decision are displayed for competing methods, along with confidence scores (i.e., pro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 56 canonical work pages

  1. [1]

    Robust stochastic neural ensemble learning with noisy labels for thoracic disease classification

    Hongyu Wang, Jiang He, Hengfei Cui, Bo Yuan, and Yong Xia. Robust stochastic neural ensemble learning with noisy labels for thoracic disease classification. IEEE Trans Med Imaging, pages 1–1, 2024

    work page 2024

  2. [2]

    Dawidowicz, E

    G. Dawidowicz, E. Hirsch, and A. Tal. Limitr: Leveraging local information for medical image-text representa- tion. In IEEE Int Conf Comput Vis, pages 21108–21116, 2023

    work page 2023

  3. [3]

    Dynamic graph enhanced contrastive learning for chest x-ray report generation

    Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, and Xiaojun Chang. Dynamic graph enhanced contrastive learning for chest x-ray report generation. In IEEE Comput Vis Pattern Recognit , pages 3334–3343, 2023

    work page 2023

  4. [4]

    Miter: Medical image–text joint adaptive pretraining with multi-level contrastive learning

    Chang Shu, Yi Zhu, Xiaochu Tang, Jing Xiao, Youxin Chen, Xiu Li, Qian Zhang, and Zheng Lu. Miter: Medical image–text joint adaptive pretraining with multi-level contrastive learning. Exp Syst App, 238:121526, 2024

    work page 2024

  5. [5]

    Vision-language models for radiology AI

    Akshay Chaudhari. Vision-language models for radiology AI. In Emerging Top Artif Intelli, volume 13118, page 1311806. International Society for Optics and Photonics, SPIE, 2024

    work page 2024

  6. [6]

    Automated radiographic report generation purely on transformer: A multicriteria supervised approach

    Zhanyu Wang, Hongwei Han, Lei Wang, Xiu Li, and Luping Zhou. Automated radiographic report generation purely on transformer: A multicriteria supervised approach. IEEE Trans Med Imaging, 41(10):2803–2813, 2022

    work page 2022

  7. [7]

    Chestxraybert: A pretrained language model for chest radiology report summarization

    Xiaoyan Cai, Sen Liu, Junwei Han, Libin Yang, Zhenguo Liu, and Tianming Liu. Chestxraybert: A pretrained language model for chest radiology report summarization. IEEE Trans Multimed, 25:845–855, 2023

    work page 2023

  8. [8]

    Veasey and Amir A

    Benjamin P. Veasey and Amir A. Amini. Low-rank adaptation of pre-trained large vision models for improved lung nodule malignancy classification. IEEE Open Journal of Engineering in Medicine and Biology, 6:296–304, 2025

    work page 2025

Show all 69 references
  1. [9]

    Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando P ´erez-Garc´ıa, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, and Ozan Okta...

    2023

  2. [10]

    O’Neil, and Sotirios A

    Konstantinos Vilouras, Pedro Sanchez, Alison Q. O’Neil, and Sotirios A. Tsaftaris. Zero-shot medical phrase grounding with off-the-shelf diffusion models. IEEE Journal of Biomedical and Health Informatics, pages 1–10, 2024

    2024

  3. [11]

    Ziruo Yi, Ting Xiao, and Mark V . Albert. A survey on multimodal large language models in radiology for report generation and visual question answering. Information, 16(2), 2025

    2025

  4. [12]

    Lungren, Hoifung Poon, and Akshay S Chaudhari

    Shih-Cheng Huang, Malte Jensen, Serena Yeung-Levy, Matthew P. Lungren, Hoifung Poon, and Akshay S Chaudhari. Multimodal foundation models for medical imaging - a systematic review and implementation guide- lines. medRxiv preprint 2024.10.23.24316003, 2024

    2024

  5. [13]

    Joint learning of localized representations from medical images and reports

    Philip M ¨uller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. Joint learning of localized representations from medical images and reports. In Eur Conf Comput Vis, page 685–701, 2022

    2022

  6. [14]

    Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, and Ozan Oktay

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, and Ozan Oktay. Making the most of text semantics to improve biomedical vision–language...

    2022

  7. [15]

    Mapping medical image-text to a joint space via masked modeling

    Zhihong Chen, Yuhao Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung-Hui Chang. Mapping medical image-text to a joint space via masked modeling. Med Image Anal, 91:103018, 2024

    2024

  8. [16]

    Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques

    Weijian Huang, Cheng Li, Hong-Yu Zhou, Jiarun Liu, Hao Yang, Yong Liang, Guangming Shi, Hairong Zheng, and Shanshan Wang. Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques. arXiv:2401.01583, 2024

    2024 arXiv

  9. [17]

    X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation

    Tom van Sonsbeek and Marcel Worring. X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation. arXiv:2302.11352, 2023

    2023 arXiv

  10. [18]

    Enhancing medical vision-language contrastive learning via inter-matching relation modelling

    Mingjian Li, Mingyuan Meng, Michael Fulham, David Dagan Feng, Lei Bi, and Jinman Kim. Enhancing medical vision-language contrastive learning via inter-matching relation modelling. arXiv:2401.10501, 2024. 15

    2024 arXiv

  11. [19]

    Cross-modal prototype driven network for radiology report generation

    Jun Wang, Abhir Bhalerao, and Yulan He. Cross-modal prototype driven network for radiology report generation. In Eur Conf Comput Vis, pages 563–579, 2022

    2022

  12. [20]

    Semantic extension for cross-modal retrieval of medical image-diagnosis report

    Guohui Ding, Qi Zhang, Shizhan Geng, and Chunlong Fan. Semantic extension for cross-modal retrieval of medical image-diagnosis report. In Nat Lang Process Chin Comput, pages 442–455, 2023

    2023

  13. [21]

    Fine-grained medical vision-language representation learning for radiology report generation

    Siyuan Wang, Bo Peng, Yichao Liu, and Qi Peng. Fine-grained medical vision-language representation learning for radiology report generation. In Empir Methods Nat Lang Process, pages 15949–15956, 2023

    2023

  14. [22]

    Exploring vision language pretraining with knowledge enhancement via large language model

    Chuenyuet Tung, Yi Lin, Jianing Yin, Qiaoyuchen Ye, and Hao Chen. Exploring vision language pretraining with knowledge enhancement via large language model. In Trust Artif Intelli Health, pages 81–91, 2024

    2024

  15. [23]

    Manning, and Curtis P

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive learning of medical visual representations from paired images and text. Proc Mach Learn Res, 182:1–24, 2022

    2022

  16. [24]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Int Conf Mach Lea...

    2021

  17. [25]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Int Conf Mach Learn, volume 139, pages 4904–4916, 2021

    2021

  18. [26]

    Deep visual-semantic alignments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In IEEE Comput Vis Pattern Recognit, pages 3128–3137, 2015

    2015

  19. [27]

    Vse++: Improving visual-semantic embed- dings with hard negatives

    Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embed- dings with hard negatives. arXiv:1707.05612, 2017

    2017 arXiv

  20. [28]

    Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S

    Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Aksh...

    2024 arXiv

  21. [29]

    Harnessing the power of pre-trained vision-language models for efficient medical report generation

    Qi Li. Harnessing the power of pre-trained vision-language models for efficient medical report generation. In ACM Int Conf Inf Know Man, page 1308–1317, 2023

    2023

  22. [30]

    Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays

    Xiaosong Wang, Ziyue Xu, Leo Tam, Dong Yang, and Daguang Xu. Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays. arXiv:2103.16022, 2021

    2021 arXiv

  23. [31]

    Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

    Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning. InAdv Neural Inf Process Syst, pages 1–14, 2022

    2022

  24. [32]

    Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias

    Zhongwei Wan, Che Liu, Mi Zhang, Jie Fu, Benyou Wang, Sibo Cheng, Lei Ma, C ´esar Quilodr´an-Casas, and Rossella Arcucci. Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias. arXiv:2305.19894, 2023

    2023 arXiv

  25. [33]

    Nezhad, Gokberk Elmas, Bilal Kabas, Fuat Arslan, and Tolga C ¸ ukur

    Valiyeh A. Nezhad, Gokberk Elmas, Bilal Kabas, Fuat Arslan, and Tolga C ¸ ukur. Generative autoregressive transformers for model-agnostic federated mri reconstruction. arXiv:2502.04521, 2025

    2025

  26. [34]

    Atli, Bilal Kabas, Fuat Arslan, Arda C

    Omer F. Atli, Bilal Kabas, Fuat Arslan, Arda C. Demirtas, Mahmut Yurt, Onat Dalmaz, and Tolga C ¸ ukur. I2i- mamba: Multi-modal medical image synthesis via selective state space modeling. arXiv:2405.14022, 2024

    2024

  27. [35]

    Cxr-llava: a multimodal large language model for interpreting chest x-ray images

    Seowoo Lee, Jiwon Youn, Hyungjin Kim, Mansu Kim, and Soon Ho Yoon. Cxr-llava: a multimodal large language model for interpreting chest x-ray images. Eur Radiol, pages 1–13, 2025

    2025

  28. [36]

    Collaboration between clinicians and vision– language models in radiology report generation

    Ryutaro Tanno, David GT Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Jo- hannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, et al. Collaboration between clinicians and vision– language models in radiology report generation. Nat Med, pages 1–10, 2024

    2024

  29. [37]

    Jay Kuo, Aichi Chien, and Kai-Wei Chang

    Masoud Monajatipoor, Mozhdeh Rouhsedaghat, Liunian Harold Li, C.-C. Jay Kuo, Aichi Chien, and Kai-Wei Chang. Berthop: An effective vision-and-language model for chest x-ray disease diagnosis. In Med Image Comput Comput Assist Inter, pages 725–734, 2022

    2022

  30. [38]

    Self-supervised multi-modal training from uncurated images and reports enables monitoring ai in radiology

    Sangjoon Park, Eun Sun Lee, Kyung Sook Shin, Jeong Eun Lee, and Jong Chul Ye. Self-supervised multi-modal training from uncurated images and reports enables monitoring ai in radiology. Med Image Anal , 91:103021, 2024

    2024

  31. [39]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In IEEE Int Conf Comput Vis, pages 21315–21326, 2023. 16

    2023

  32. [40]

    Knowledge-enhanced visual-language pre-training on chest radiology images

    Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre-training on chest radiology images. Nat Commun, 14(1):4542, 2023

    2023

  33. [41]

    Improving medical vision-language contrastive pretraining with semantics-aware triage

    Bo Liu, Donghuan Lu, Dong Wei, Xian Wu, Yan Wang, Yu Zhang, and Yefeng Zheng. Improving medical vision-language contrastive pretraining with semantics-aware triage. IEEE Trans Med Imaging , 42(12):3579– 3589, 2023

    2023

  34. [42]

    Lungren, and Serena Yeung

    Shih-Cheng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In IEEE Int Conf Comput Vis, pages 3922–3931, 2021

    2021

  35. [43]

    A contrastive triplet network for automatic chest x-ray reporting

    Yan Yang, Jun Yu, Hanliang Jiang, Weidong Han, Jian Zhang, and Wei Jiang. A contrastive triplet network for automatic chest x-ray reporting. Neurocomput, 502:71–83, 2022

    2022

  36. [44]

    Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images

    Che Liu, Anand Shah, Wenjia Bai, and Rossella Arcucci. Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images. arXiv:2310.07027, 2023

    2023 arXiv

  37. [45]

    MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning

    Jiarun Liu, Hong-Yu Zhou, Cheng Li, Weijian Huang, Hao Yang, Yong Liang, and Shanshan Wang. MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning. arXiv:2401.01591, 2024

    2024 arXiv

  38. [46]

    Visual prior-based cross- modal alignment network for radiology report generation

    Sheng Zhang, Chuan Zhou, Leiting Chen, Zhiheng Li, Yuan Gao, and Yongqi Chen. Visual prior-based cross- modal alignment network for radiology report generation. Comput Biol Med, 166:107522, 2023

    2023

  39. [47]

    Unify, align and refine: Multi-level semantic alignment for radiology report generation

    Yaowei Li, Bang Yang, Xuxin Cheng, Zhihong Zhu, Hongxiang Li, and Yuexian Zou. Unify, align and refine: Multi-level semantic alignment for radiology report generation. InIEEE Int Conf Comput Vis, pages 2851–2862, 2023

    2023

  40. [48]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Trans Pattern Anal Mach Intelli, pages 1–20, 2024

    2024

  41. [49]

    MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. MedCLIP: Contrastive Learning from Unpaired Medical Images and Text. arXiv:2210.10163, 2022

    2022 arXiv

  42. [50]

    Anatomical Structure-Guided Medical Vision-Language Pre-training

    Qingqiu Li, Xiaohan Yan, Jilan Xu, Runtian Yuan, Yuejie Zhang, Rui Feng, Quanli Shen, Xiaobo Zhang, and Shujun Wang. Anatomical Structure-Guided Medical Vision-Language Pre-training. arXiv:2403.09294, 2024

    2024 arXiv

  43. [51]

    KIA: Knowledge-guided implicit vision- language alignment for chest X-ray report generation

    Heng Yin, Shanlin Zhou, Pandong Wang, Zirui Wu, and Yongtao Hao. KIA: Knowledge-guided implicit vision- language alignment for chest X-ray report generation. In Int Conf Comput Ling, pages 4096–4108, 2025

    2025

  44. [52]

    Content-based medical image retrieval with opponent class adaptive margin loss

    Saban Ozturk, Emin Celik, and Tolga Cukur. Content-based medical image retrieval with opponent class adaptive margin loss. Inf Sci, 637:118938, 2023

    2023

  45. [53]

    Improving joint learning of chest X-Ray and radiology report by word region alignment

    Zhanghexuan Ji, Mohammad Abuzar Shaikh, Dana Moukheiber, Sargur N Srihari, Yifan Peng, and Mingchen Gao. Improving joint learning of chest X-Ray and radiology report by word region alignment. Mach Learn Med Imaging, 12966:110–119, 2021

    2021

  46. [54]

    Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

    Jiawei Chen, Dingkang Yang, Tong Wu, Yue Jiang, Xiaolu Hou, Mingcheng Li, Shunli Wang, Dongling Xiao, Ke Li, and Lihua Zhang. Detecting and Evaluating Medical Hallucinations in Large Vision Language Models. arXiv:2406.10185, 2024

    2024 arXiv

  47. [55]

    Ontology-based data integration between clinical and research systems

    Sebastian Mate, Felix K ¨opcke, Dennis Toddenroth, Marcus Martin, Hans-Ulrich Prokosch, Thomas B ¨urkle, and Thomas Ganslandt. Ontology-based data integration between clinical and research systems. PLOS ONE, 10(1):1–20, 2015

    2015

  48. [56]

    Chexpert: A large chest radiograph dataset with uncer- tainty labels and expert comparison

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncer- tainty labels and expert comparison. In AAAI Conf Artif Intell...

    2019

  49. [57]

    On the role of morphological information for contextual lemmatization

    Olia Toporkov and Rodrigo Agerri. On the role of morphological information for contextual lemmatization. Comput Ling, pages 1–35, 2024

    2024

  50. [58]

    Optimizing the dice score and jaccard index for medical image segmentation: Theory and practice

    Jeroen Bertels, Tom Eelbode, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B Blaschko. Optimizing the dice score and jaccard index for medical image segmentation: Theory and practice. In Med Image Comput Comput Assist Inter, pages 92–100, 2019

    2019

  51. [59]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2020

    2010 arXiv

  52. [60]

    Spd manifold deep metric learning for image set classification

    Rui Wang, Xiao-Jun Wu, Ziheng Chen, Cong Hu, and Josef Kittler. Spd manifold deep metric learning for image set classification. IEEE Trans Neural Netw Learn Syst, pages 1–15, 2024. 17

    2024

  53. [61]

    Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data, 6(1):317, 2019

    2019

  54. [62]

    Wu, Safwan S

    George Shih, Carol C. Wu, Safwan S. Halabi, Marc D. Kohli, Luciano M. Prevedello, Tessa S. Cook, Arjun Sharma, Judith K. Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, Ritu R. Gill, Myrna C.B. Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N. Shah, Dharshan Vummi...

    2019

  55. [63]

    Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B

    Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. Publicly Available Clinical BERT Embeddings. arXiv:1904.03323, 2019

    1904 arXiv

  56. [64]

    Langlotz, Andrew Y

    Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P. Langlotz, Andrew Y . Ng, and Pranav Rajpurkar. Expert-level detec- tion of pathologies from unannotated chest x-ray images via self-supervised learning. Nat Bio Eng, 6(12):1399– 1406, 2022

    2022

  57. [65]

    Medfilip: Medical fine-grained language-image pre-training.IEEE J Biomed Health Inf, 2025

    Xinjie Liang, Xiangyu Li, Fanding Li, Jie Jiang, Qing Dong, Wei Wang, Kuanquan Wang, Suyu Dong, Gongning Luo, and Shuo Li. Medfilip: Medical fine-grained language-image pre-training.IEEE J Biomed Health Inf, 2025

    2025

  58. [66]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Int Conf Comput Vis, pages 618–626, 2017

    2017

  59. [67]

    Muzaffer Ozbey, Onat Dalmaz, Salman U. H. Dar, Hasan A. Bedel, Saban Ozturk, Alper Gungor, and Tolga C ¸ ukur. Unsupervised medical image translation with adversarial diffusion models. IEEE Trans Med Imaging, 42(12):3524–3539, 2023

    2023

  60. [68]

    Self-consistent recursive diffusion bridge for medical image translation

    Fuat Arslan, Bilal Kabas, Onat Dalmaz, Muzaffer Ozbey, and Tolga C ¸ ukur. Self-consistent recursive diffusion bridge for medical image translation. arXiv:2405.06789, 2024

    2024 arXiv

  61. [69]

    Nezhad, Saban Ozturk, Emine U

    Bilal Kabas, Fuat Arslan, Valiyeh A. Nezhad, Saban Ozturk, Emine U. Saritas, and Tolga C ¸ ukur. Physics-driven autoregressive state space models for medical image reconstruction. arXiv:2412.09331, 2024. 18

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.