Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that dynamic soft labels built from text, CheXbert clinical labels, and RadGraph graph embeddings, combined with negation-based hard negatives, make medical CLIP training robust to templated duplicates and negations…

desk verdict Sensible medical-CLIP recipe with a systematic ablation, but the negation benchmark needs a text-only baseline before the clinical-language claims hold. read the letter →

arxiv 2505.22079 v1 pith:PST64OEZ submitted 2025-05-28 cs.CV

Hanbin Ko , Chang-Min Park This is my paper
classification cs.CV
keywords vision-languagepre-trainingmedicalimagingchestX-raycontrastivelearningnegationhandlingdynamicsoftlabelsdataimbalanceCLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard contrastive objective used to train CLIP on chest X-ray reports is hampered by two medical-data quirks: templated duplicate reports that create semantic overlap inside a batch, and frequent negations like "no pneumothorax" that bag-of-words models tend to ignore. To fix this, the authors replace one-hot contrastive targets with dynamic soft labels built from three similarity streams—text embeddings, CheXbert clinical labels, and RadGraph-derived graph embeddings—and add hard negatives that negate a single clinical entity. They also introduce CXR-Align, a benchmark that tests whether a model can reject a wrongly negated report and prefer a complete report over one missing an entity. The paper reports that the combined method achieves state-of-the-art results on zero-shot and fine-tuned classification, report retrieval, adversarial prediction, and the new negation benchmark.

What carries the argument

Dynamic soft contrastive loss: a KL-divergence loss that matches cross-modal logits against thresholded, row-normalized intra-modal similarity matrices computed from three modalities—text embeddings, CheXbert clinical labels, and RadGraph graph embeddings encoded by a two-layer GCN. Negation-based hard negatives are generated by an LLM pipeline that removes all sentences about one CheXbert-positive entity and inserts a template negation, so the hard negative differs from the original report by exactly one entity. Together these replace the one-hot targets of InfoNCE with graded targets that share label mass among clinically similar reports, which is the mechanism that counteracts semantic overlap from duplicate templated reports.

What would settle it

A human-evaluated set of negated reports that decorrelates template phrasing from clinical content would settle the claim: if a model trained with the proposed hard negatives performs at chance when the negation templates are rephrased into non-canonical clinical language (e.g., 'pneumothorax is excluded' or 'no evidence for pneumothorax is present') while still scoring near-perfect on CXR-Align's fixed templates, then the benchmark's claim of advanced negation comprehension would be falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that medical CLIP training can be made robust to data imbalance and negation by softening the contrastive target itself. Instead of treating every other sample in the batch as a hard negative, the model computes intra-modal similarity matrices for text, CheXbert clinical labels, and RadGraph graph embeddings, thresholds these matrices to suppress noise, row-normalizes them, and trains the image-text, image-graph, and text-graph logits against these soft targets using KL divergence. Negation-based hard negatives—reports that differ from the original by exactly one negated entity—are added to the batch, and the dynamic soft labels are what keep those hard negatives from injecting semantic noise. The authors claim this yields state-of-the-art performance across classification, retrieval, adversarial prediction, and negation comprehension, and they introduce CXR-Align as a benchmark dedicated to measuring negation and clinical-entity alignment.

Load-bearing premise

CXR-Align is assumed to actually measure negation comprehension and clinical understanding; the authors themselves note that their hard-negative-trained model's near-perfect score on the negation task may reflect the model exploiting surface template shortcuts rather than true clinical reasoning.

Editorial extensions

If this is right

  • If the dynamic soft-label method is correct, medical CLIP training no longer needs to discard duplicate or templated reports; the soft targets let the model share labels among semantically identical reports, turning imbalance into usable signal.
  • Negation-based hard negatives, when combined with soft labels, improve the model's ability to reject wrongly negated statements and to prefer a complete report over one missing an entity, as measured on CXR-Align.
  • The approach extends across contrastive frameworks: the paper reports gains when the dynamic soft-label and hard-negative components are added on top of CLIP and CLIPN baselines, suggesting the method is a drop-in training modification rather than a new architecture.
  • Graph embeddings from RadGraph contribute stable gains on top of text and clinical similarity, implying that relational structure such as location and severity is a complementary signal to entity-presence labels.
  • On the normal-case detection task, including large numbers of normal, duplicate reports during training enables the model to retrieve a single normal report among thousands of abnormal ones, rather than hurting performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The shortcut-avoidance hypothesis the authors raise for CXR-Align could be tested directly by probing the model with negated reports whose surface form is unusual (e.g., 'the presence of pneumothorax is ruled out'); if performance collapses, the benchmark is measuring template memorization rather than negation reasoning.
  • The same soft-label mechanism might apply to other imbalanced multimodal domains such as pathology or dermatology, where templated language and duplicate normal cases are common—nothing in the method is specific to chest X-rays.
  • One could check whether the dynamic soft labels actually change the learned embedding geometry by testing whether the soft-target distribution predicts the model's confusion matrix on rare entities; if it does, soft labels are the active ingredient behind the gains.
  • Because the method inserts hard negatives into the batch, batch composition changes; an open question is how performance varies with the ratio of hard negatives to original reports, a hyperparameter the paper leaves fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a modified CLIP training pipeline for chest X-ray imaging that combines three additions: dynamic soft contrastive labels built from text, CheXbert clinical-label, and RadGraph graph similarities; negation-based hard negatives generated by an LLM pipeline; and an auxiliary graph-embedding contrastive loss. The authors also introduce CXR-Align, a benchmark with triplets (I, r, r_n) and (I, r, r_r) intended to measure negation comprehension and clinical report alignment. Experiments cover zero-shot and fine-tuned classification, report retrieval, adversarial prediction, normal-case detection, and the new CXR-Align benchmark, with claims of state-of-the-art performance across multiple tasks.

Significance. If the controlled comparisons were in place, the paper would offer a practical contribution: the dynamic soft-label formulation is clearly stated, the ablation structure is monotone and easy to follow, and the normal-case-detection finding that including duplicate normal reports helps rather than hurts is an interesting and publishable observation. The CXR-Align benchmark also targets a genuinely important gap, since negation comprehension is rarely evaluated in medical VLP. The main value is therefore conditional on two things: (i) showing that CXR-Align measures image-grounded negation understanding rather than shallow text mismatch, and (ii) demonstrating SOTA claims under fair, controlled comparisons. The manuscript is strong on transparency of the proposed pipeline (prompts, preprocessing, hyperparameters are documented), but the evaluation protocol currently does not support the central claims as written.

major comments (4)
  1. [Sec. 5.6 and App. E.1] The CXR-Align Task A is confounded by a purely textual cue: r_n is constructed by removing a positive-entity sentence and inserting a template negation, so a text-only model that detects sentence-level mismatch can solve the task without using the image. The manuscript itself states in Sec. 5.6 that the near-perfect score is 'unexpectedly high' and hypothesizes that the model 'may be learning to avoid unnatural negations by exploiting shortcuts,' and App. E.1 shows that accuracy is driven by entity frequency and prompt identity rather than insertion location. No text-only baseline (e.g., BERT similarity, template matching, or self-BLEU with the original report) and no vision-only ablation are reported. Because the paper's headline claim of 'advancing clinical language understanding' rests on this benchmark, the authors must add such baselines, report per-entity and per-prompt accuracy, and either re-validate the benchmark or substantially soften the negation-comprehension claim.
  2. [Tab. 2 and Sec. 5.3] The state-of-the-art comparison in Table 2 is not controlled: SOTA numbers are quoted from their original papers, which use different image encoders, text encoders, preprocessing pipelines, training data, and fine-tuning schedules. The statement in Sec. 5.3 that 'Our approach outperforms other SOTA methods on most benchmark datasets' is therefore not supported unless the comparison models are retrained under the same data split, preprocessing, and evaluation protocol, or unless the table is clearly labelled as a reproduced-versus-published comparison with all discrepancies stated. At minimum, the authors should report which entries were reproduced and which were taken from prior work.
  3. [Sec. 4.2, Eqs. (3)-(5)] The dynamic soft labels are computed from the very text and graph embeddings being optimized (St = T·T^T and Sg = G·G^T), so the KL targets are functions of the current model state. This creates a self-referential training loop that can reinforce existing biases or drift during training. The paper does not analyze whether the soft-label distribution is stable across epochs, nor does it compare against a detached variant where text similarities are computed with a frozen encoder or precomputed offline. Since the method's core novelty is this dynamic target, an experiment separating the effect of the target construction from the effect of the loss form is needed.
  4. [Sec. C.4] The three similarity thresholds (tau_t=0.9, tau_c=0.8, tau_g=0.7) and the loss weights (w_T=w_C=w_G=0.167) are hand-picked, and no sensitivity analysis or validation-based selection is reported. Thresholding is described in Sec. 4.2 as 'crucial,' yet the paper does not show how performance varies with these values or whether the conclusions survive a reasonable range of thresholds. In addition, all experiments appear to be single-seed runs with no error bars and no significance tests, which is especially problematic for the small margins in Table 2 and the fine-tuning comparisons in Table 3.
minor comments (5)
  1. [Abstract] The sentence 'thereby improving clinical comprehension and improving the applicability of contrastive loss' repeats 'improving'; consider rewording.
  2. [Eq. (10)] The double sum over i and j with i != j should be made explicit: as written, the domains of i and j are clear from context but a reader cannot immediately tell that all six ordered cross-modal pairs are included. Please define M_1, M_2, M_3 and state the ordered-pair convention.
  3. [Tab. 6] On MIMIC Task A, CLIPN (hard negatives alone) scores 97.3 while the full model scores 96.5. This non-monotonicity is not discussed; a sentence explaining why the full method slightly lowers Task A accuracy would help.
  4. [Tab. 3 and App. E.3] The Normal Case Detection (NCD) numbers are reported inconsistently: Table 3 says OpenI normal CXRs are retrieved from 2,999 abnormal reports, while Appendix E.3 reports an internal test set and mentions a rank of 68 for the MIMIC-only model. Please clarify which test set each reported NCD value comes from.
  5. [Sec. B.3] The CXR-Align benchmark uses Gemini-Flash for LLM-based transformations, but the version and decoding settings are not given. For reproducibility, please provide the exact model version, any sampling parameters, and the full list of generated negation templates, not only examples.

Circularity Check

2 steps flagged · score 5.0 of 10

Clinical-understanding claims are partially endogenous: CheXbert F1 retrieval and CXR-Align negation evaluation are built from the same labeler/pipeline used to construct training targets, while classification results remain externally benchmarked.

  1. fitted input called prediction [Sec. 4.2 (Eqs. 3-7), Sec. D.2, Sec. 5.7]
    "Since embeddings alone may not fully capture clinical semantics [2], we use 14 labels extracted from reports by CheXbert [28] as an alternative source of clinical information. ... For report retrieval, we use the CheXbert F1 score rather than the standard BERTScore to evaluate how the retrieved or generated report clinically reflects the original report. ... This implies that our model captures more clinically meaningful features from the reports."

    The clinical soft-label stream y_c is computed from CheXbert-derived label embeddings (Eqs. 3-5) and used as the training target for image-text alignment (Eqs. 6-7). The report-retrieval claim is then measured primarily by Macro CheXbert F1, i.e., agreement with the same labeler used to build the training target. A model optimized to align CheXbert label neighborhoods is by construction expected to score well on CheXbert F1, so the 'clinically meaningful retrieval' result is in part a measure of fit to the training surrogate rather than an independent clinical-language benchmark.

  2. other [Sec. 3, Sec. 4.1, Sec. 5.6, Sec. B.3.2]
    "Following the process from Sec. 3, we create hard negatives r n for abnormal CXRs by introducing negations into the report. ... The final dataset contains two triplet structures (I, r, rn) and (I, r, rr). ... Notably, the performance of our model trained with hard-negatives on the negation-related task is unexpectedly high, leading us to hypothesize that the model may be learning to avoid unnatural negations by exploiting shortcuts."

    CXR-Align task A is generated with the same CheXbert entity selection, LLM sentence splitting, entity-sentence removal, and predefined negative-template insertion (Sec. B.3.2) that Sec. 4.1 uses to construct training hard negatives. High CXR-Align accuracy can therefore reflect in-distribution pattern matching of the training augmentation rather than an independent measure of negation understanding. The paper's own 'unexpectedly high' and 'shortcuts' statement admits this, and no text-only or vision-only baseline is reported, so the benchmark cannot independently support the headline clinical-language-comprehension claim.

full rationale

The paper does not rely on load-bearing self-citations or imported uniqueness theorems, and the main zero-shot/fine-tuned classification results (RSNA, SIIM, CheXpert, CXR14) are tested against external labels with standard splits, so the overall method is not a pure derivation. The dynamic soft labels are a training objective rather than a claimed prediction, so I do not count the self-distillation structure itself as formal circularity. However, two evaluation loops are partially endogenous: report retrieval is highlighted through CheXbert F1, the same labeler that supplies the clinical soft-label training target, and CXR-Align's negation task is constructed from the exact pipeline used to make the hard negatives the model trains on. Both are central to the paper's 'clinical language understanding' claims, and the CXR-Align issue is explicitly acknowledged as a possible shortcut by the authors. Score 5 reflects partial circularity in these self-constructed evaluations while recognizing that the classification and adversarial-prediction results remain externally benchmarked.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method depends on auxiliary labelers (CheXbert, RadGraph, ClinicalBERT) and hand-set thresholds, none of which are validated for this purpose in the paper. The hard-negative generation also depends on Gemini-Flash outputs, which are not reproducible in detail.

free parameters (5)
  • text similarity threshold tau_t = 0.9
    Threshold in Equation 4 for text-based soft labels, hand-set in Appendix C.4 with no sensitivity analysis.
  • clinical similarity threshold tau_c = 0.8
    Threshold in Equation 5 for CheXbert-label soft labels, hand-set in Appendix C.4.
  • graph similarity threshold tau_g = 0.7
    Threshold in Equation 9 for graph-embedding soft labels, hand-set in Appendix C.4.
  • uniform loss weights w_T, w_C, w_G = 0.167 each
    Weights in Equation 10, set to near-uniform values in Appendix C.4 without justification.
  • temperature tau = 0.1
    Contrastive temperature used in Equations 1-2, fixed at 0.1 in Appendix C.4.
assumptions (5)
  • domain assumption CheXbert's 14 labels plus an extra no-findings label provide an accurate clinical summary of each report.
    These labels define the clinical similarity matrix S_c in Equation 3; if CheXbert mislabels, the soft targets are wrong.
  • domain assumption RadGraph entity/relation extraction and ClinicalBERT node embeddings capture clinically meaningful structure.
    Graph embeddings G feed the graph similarity S_g in Equation 8; noisy graphs would inject noise into soft labels.
  • domain assumption The LLM (Gemini-Flash) correctly splits reports into single-entity sentences and generates semantically valid negations.
    Both hard negatives (Section 4.1) and CXR-Align (Section 3, Appendix B.3) are built from this pipeline.
  • ad hoc to paper Thresholding at 0.9/0.8/0.7 removes noise rather than discarding valid similar pairs.
    No theoretical or empirical justification is provided for these specific values in the paper.
  • ad hoc to paper KL divergence to soft labels from the model's own embeddings is a stable training objective.
    Used in Equations 6-7; the self-referential nature of the targets may reinforce existing biases without stabilization guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis." pith.science (2026). https://pith.science/paper/PST64OEZ

@misc{pith2026250522079,
  author       = {Pith},
  title        = {Pith review of: Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PST64OEZ}},
  note         = {Machine review of arXiv:2505.22079}
}
read the original abstract

The development of large-scale image-text pair datasets has significantly advanced self-supervised learning in Vision-Language Processing (VLP). However, directly applying general-domain architectures such as CLIP to medical data presents challenges, particularly in handling negations and addressing the inherent data imbalance of medical datasets. To address these issues, we propose a novel approach that integrates clinically-enhanced dynamic soft labels and medical graphical alignment, thereby improving clinical comprehension and the applicability of contrastive loss in medical contexts. Furthermore, we introduce negation-based hard negatives to deepen the model's understanding of the complexities of clinical language. Our approach is easily integrated into the medical CLIP training pipeline and achieves state-of-the-art performance across multiple tasks, including zero-shot, fine-tuned classification, and report retrieval. To comprehensively evaluate our model's capacity for understanding clinical language, we introduce CXR-Align, a benchmark uniquely designed to evaluate the understanding of negation and clinical information within chest X-ray (CXR) datasets. Experimental results demonstrate that our proposed methods are straightforward to implement and generalize effectively across contrastive learning frameworks, enhancing medical VLP capabilities and advancing clinical language understanding in medical imaging.

Figures

Figures reproduced from arXiv: 2505.22079 by the authors.

Figure 1
Figure 1. (a) Standard visual-language pre-training approaches us [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Given a CXR report, CheXbert identifies all positive entities, and one is randomly selected. A language model then (i) splits [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed pipeline. Hard negative reports are created that differ from the original by only one clinical entity. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Counts of clinical entities in the whole MIMIC training [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Counts of clinical entities in reports for the MIMIC train [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Comparison of textual, clinical similarity between reports. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 8
Figure 8. Figure 8: (a) Data with the lowest OOD score in the MIMIC [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The number of selected entities in each dataset for CXR-Align. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Example of the CXR-Align generation process. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Examples of retrieved reports. Blue text represents important entities that should be included in the report. Red text indicates [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Detailed sub-analysis for CXR-Align on MIMIC dataset. (A) Task accuracy for entities that were either negated or removed. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Detailed sub-analysis for CXR-Align on OPENI dataset. (A) Task accuracy for entities that were either negated or removed. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 19 canonical work pages

  1. [1]

    Publicly available clinical bert embeddings.arXiv preprint arXiv:1904.03323, 2019

    Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings.arXiv preprint arXiv:1904.03323, 2019. 5, 6

    arXiv 1904

  2. [2]

    ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics

    Oishi Banerjee, Agustina Saenz, Kay Wu, Warren Clements, Adil Zia, Dominic Buensalido, Helen Kavnoudias, Alain S Abi-Ghanem, Nour El Ghawi, Cibele Luna, et al. Rexamine- global: A framework for uncovering inconsistencies in radiology report generation metrics.arXiv preprint arXiv:2408.16208, 2024. 5

    work page Pith review arXiv 2024

  3. [3]

    Learning to exploit temporal structure for biomed- ical vision-language processing

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomed- ical vision-language processing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15016–15...

    work page 2023

  4. [4]

    Maira-2: Grounded radiology report gen- eration.arXiv preprint arXiv:2406.04449, 2024

    Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Anton Schwaighofer, Sam Bond-Taylor, Maximilian Ilse, Fernando P´erez-Garc´ıa, Valentina Salvatelli, Harshita Sharma, Felix Meissen, et al. Maira-2: Grounded radiology report gen- eration.arXiv preprint arXiv:2406.04449, 2024. 3, 4

    arXiv 2024

  5. [5]

    Llm2vec: Large language models are secretly pow- erful text encoders.arXiv preprint arXiv:2404.05961, 2024

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mos- bach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly pow- erful text encoders.arXiv preprint arXiv:2404.05961, 2024. 8

    arXiv 2024

  6. [6]

    Making the most of text semantics to improve biomedical vision–language processing

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. InEuro- pean conference on computer vision, pages 1–21. Springer,

  7. [7]

    Chexpert plus: Hundreds of thousands of aligned radiology texts, im- ages and patients.arXiv preprint arXiv:2405.19538, 2024

    Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P Langlotz. Chexpert plus: Hundreds of thousands of aligned radiology texts, im- ages and patients.arXiv preprint arXiv:2405.19538, 2024. 2

    arXiv 2024

  8. [8]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 2

    2020

Show all 41 references
  1. [9]

    Preparing a collection of radiology examinations for distribution and re- trieval.Journal of the American Medical Informatics Asso- ciation, 23(2):304–310, 2016

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and re- trieval.Journal of the American Medical Informatics Asso-...

    2016

  2. [10]

    Improving clip training with language rewrites.Advances in Neural Information Processing Sys- tems, 36, 2024

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites.Advances in Neural Information Processing Sys- tems, 36, 2024. 2

    2024

  3. [11]

    Pyramidclip: Hierarchical fea- ture alignment for vision-language model pretraining.Ad- vances in neural information processing systems, 35:35959– 35970, 2022

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Ron- grong Ji, and Chunhua Shen. Pyramidclip: Hierarchical fea- ture alignment for vision-language model pretraining.Ad- vances in neural information processing systems, 35:35959– 35970, 2022. 3

    2022

  4. [12]

    Softclip: Softer cross-modal alignment makes clip stronger

    Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softclip: Softer cross-modal alignment makes clip stronger. InProceed- ings of the AAAI Conference on Artificial Intelligence, pages 1860–1868, 2024. 3, 6, 1

    2024

  5. [13]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36, 2024

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.Advances in neural information processing systems, 36, 2024. 3

    2024

  6. [14]

    Gloria: A multimodal global-local represen- tation learning framework for label-efficient medical image recognition

    Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local represen- tation learning framework for label-efficient medical image recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2...

    2021

  7. [15]

    Llm2clip: Powerful language model unlock richer visual representation.arXiv preprint arXiv:2411.04997, 2024

    Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, et al. Llm2clip: Powerful language model unlock richer visual representation.arXiv preprint arXiv:2411.04997, 2024. 8

    2024

  8. [16]

    Rad- graph: Extracting clinical entities and relations from radiol- ogy reports.arXiv preprint arXiv:2106.14463, 2021

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. Rad- graph: Extracting clinical entities and relations from radiol- ogy reports.arXiv preprint arXiv:2106.14463, 2021. 5, 8

    2021 arXiv

  9. [17]

    Mimic- iv.PhysioNet

    Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic- iv.PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pages 49–55, 2020. 2

    2021

  10. [18]

    Mimic-iii, a freely accessible critical care database

    Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1–9, 2016. 2, 3

    2016

  11. [19]

    Carzero: Cross-attention alignment for radiology zero-shot classifica- tion

    Haoran Lai, Qingsong Yao, Zihang Jiang, Rongsheng Wang, Zhiyang He, Xiaodong Tao, and S Kevin Zhou. Carzero: Cross-attention alignment for radiology zero-shot classifica- tion. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 11137–11146,

  12. [20]

    Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.arXiv preprint arXiv:2110.05208, 2021

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Su- pervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.arXiv preprint arXiv:2110.05208, 2021. 6

    arXiv

  13. [21]

    Mlip: Enhanc- ing medical visual representation with divergence encoder and knowledge-guided contrastive learning

    Zhe Li, Laurence T Yang, Bocheng Ren, Xin Nie, Zhangyang Gao, Cheng Tan, and Stan Z Li. Mlip: Enhanc- ing medical visual representation with divergence encoder and knowledge-guided contrastive learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern ...

    2024

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 6

    2021

  15. [23]

    Crepe: Can vision-language foundation models reason compositionally? InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10910–10921, 2023

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10910–10921, 2023. 3

    2023

  16. [24]

    Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations.Scientific Data, 9(1):429, 2022

    Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, et al. Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations.Scientific Data, 9(1):429, 2022. 6

    2022

  17. [25]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 5

    2018 arXiv

  18. [26]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  19. [27]

    Learn” no” to say” yes” bet- ter: Improving vision-language models via negations.arXiv preprint arXiv:2403.20312, 2024

    Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Learn” no” to say” yes” bet- ter: Improving vision-language models via negations.arXiv preprint arXiv:2403.20312, 2024. 2, 3

    2024 arXiv

  20. [28]

    Chexbert: com- bining automatic labelers and expert annotations for accu- rate radiology report labeling using bert.arXiv preprint arXiv:2004.09167, 2020

    Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren. Chexbert: com- bining automatic labelers and expert annotations for accu- rate radiology report labeling using bert.arXiv preprint arXiv:2004.09167, 2020. 3, 5, 8

    2004 arXiv

  21. [29]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 3

    2023 arXiv

  22. [30]

    Expert-level detection of pathologies from unannotated chest x-ray images via self- supervised learning.Nature Biomedical Engineering, 6(12): 1399–1406, 2022

    Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, An- drew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self- supervised learning.Nature Biomedical Engineering, 6(12): 1399–1406, 2022. 2

    2022

  23. [31]

    Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mo- hammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. InProceedings of the IEEE conference on ...

  24. [32]

    Exploring vision-language models for imbalanced learning

    Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. Exploring vision-language models for imbalanced learning. International Journal of Computer Vision, 132(1):224–237,

  25. [33]

    Generalization beyond data imbalance: A con- trolled study on clip for transferable insights.arXiv preprint arXiv:2405.21070, 2024

    Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, and Xiaojuan Qi. Generalization beyond data imbalance: A con- trolled study on clip for transferable insights.arXiv preprint arXiv:2405.21070, 2024. 3

    2024 arXiv

  26. [34]

    Xlip: Cross-modal attention masked modelling for medical language-image pre-training

    Biao Wu, Yutong Xie, Zeyu Zhang, Minh Hieu Phan, Qi Chen, Ling Chen, and Qi Wu. Xlip: Cross-modal attention masked modelling for medical language-image pre-training. arXiv preprint arXiv:2407.19546, 2024. 1, 2

    2024 arXiv

  27. [35]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 21372–21383, 2023. 1, 2

    2023

  28. [36]

    Worse than random? an embarrassingly simple probing eval- uation of large multimodal models in medical vqa.arXiv preprint arXiv:2405.20421, 2024

    Qianqi Yan, Xuehai He, Xiang Yue, and Xin Eric Wang. Worse than random? an embarrassingly simple probing eval- uation of large multimodal models in medical vqa.arXiv preprint arXiv:2405.20421, 2024. 6, 2

    2024 arXiv

  29. [37]

    Graph convo- lutional networks for text classification

    Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convo- lutional networks for text classification. InProceedings of the AAAI conference on artificial intelligence, pages 7370– 7377, 2019. 5

    2019

  30. [38]

    Cxr-clip: Toward large scale chest x-ray language-image pre-training

    Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K Hong, Woonhyuk Baek, and Byungseok Roh. Cxr-clip: Toward large scale chest x-ray language-image pre-training. InInternational Conference on Medical Im- age Computing and Computer-Assisted Intervention, pages 101–...

    2023

  31. [39]

    When and why vision- language models behave like bags-of-words, and what to do about it?arXiv preprint arXiv:2210.01936, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it?arXiv preprint arXiv:2210.01936, 2022. 2, 3

    2022 arXiv

  32. [40]

    Contrastive learning of medical visual representations from paired images and text

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. InMachine Learning for Healthcare Conference, pages 2–

  33. [41]

    Right large pleural effusion. No pneumothorax

    Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, et al. Clip in medical imaging: A comprehensive sur- vey.arXiv preprint arXiv:2312.07353, 2023. 1 A. Motivation In general-domain datasets, captions involve million...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.