Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Multimodal Medical Image Binding via Shared Text Embeddings

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Medical images from five modalities can be aligned through shared text embeddings, with no paired image data between modalities.

desk verdict Text-anchored binding of five medical CLIP-like models is a plausible idea with broad evaluation, but the central text-pair alignment in Eq. (2) is never specified, which undercuts the method as written. read the letter →

arxiv 2506.18072 v2 pith:JZRNTYPO submitted 2025-06-22 eess.IV cs.AIcs.CV

Yunhao Liu , Suyang Xi , Shiqi Liu , Hong Ding , Chicheng Jin , Chong Zhong , Junjun He , Catherine C. Liu
show 1 more author
Yiqing Shen
This is my paper
classification eess.IVcs.AIcs.CV
keywords multimodalmedicalimagingsharedtextembeddingcontrastivelanguage-imagepretrainingknowledgedistillationzero-shotclassificationfew-shotcross-modalretrievaltext-anchoredalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Medical images come in many modalities, but pairing them with each other for training is expensive and often impossible. This paper argues that text can serve as the common anchor instead: every modality already has clinical reports or descriptions, so aligning the text spaces of five existing image-text models should align the image spaces too. It introduces M3Bind, which fine-tunes five modality-specific CLIP-like models with a text-text alignment loss while preserving each one's image-text alignment, then distills their text encoders into a single model. The reported result is that zero-shot and few-shot classification improve over each modality's original model, and X-ray-to-ECG retrieval works without ever training on paired X-ray-ECG data. A sympathetic reader would take this as evidence that text-bridged alignment can substitute for explicit image pairing in medical multimodal learning.

What carries the argument

The load-bearing mechanism is a two-phase text-anchored alignment. In the first phase, each modality-specific CLIP-like model is fine-tuned via LoRA so its text embeddings are pulled toward text embeddings from other modalities through the MSE loss of Eq. (2), while Eq. (1) preserves the original image-text contrastive alignment; adaptive sampling, learning-rate scaling, and loss weighting balance dataset sizes. In the second phase, Shared Embedding Space Knowledge Distillation (SESKD) compresses the five teacher text encoders into a single BioClinicalBERT student by combining embedding-matching MSE with a contrastive loss against all five image encoders, so the unified text model remains aligned with every modality. The unifying identity is simply that text, which already co-occurs with each image modality, becomes the shared coordinate system for the image spaces.

What would settle it

Train M3Bind twice on the same modalities: once with text pairs that are verified to describe the same finding across modalities, and once with randomly shuffled text pairs. If cross-modal retrieval on a held-out set is not clearly better in the verified-pair run, the shared text space is not actually binding through semantics.

Watch

Extended reading notes

Core claim

M3Bind's central claim is that multiple medical imaging modalities can be brought into one embedding space without any paired data between image modalities, by using text as the bridging anchor. Starting from five pre-trained CLIP-like models — one each for X-ray, CT, retina, ECG, and pathology — the framework aligns their text encoders with an MSE loss on text samples that share semantic content, while a contrastive loss keeps each image encoder aligned to its own text. An adaptive modality-balancing scheme prevents large datasets like X-ray from dominating, and a distillation stage (SESKD) trains a single BioClinicalBERT student to reproduce the teacher text embeddings and to stay contrastively aligned with all five image encoders. The paper reports that the resulting shared space improves or matches the original models on zero-shot and few-shot classification across twelve datasets, and enables cross-modal retrieval between X-ray and ECG without direct image-image pairs.

Load-bearing premise

The load-bearing premise is that text descriptions from different modalities can be matched to the same semantic content without paired images, so that pulling their text embeddings together aligns meanings rather than gluing unrelated reports.

Editorial extensions

If this is right

  • Any two of the five modalities can be compared or retrieved through the shared text space despite never being paired in training.
  • Adding a new modality requires only a CLIP-like model for that modality and alignment of its text encoder to the shared space, avoiding the quadratic number of image-image pairings.
  • The unified distilled text encoder makes inference cheaper than running five separate text encoders while preserving cross-modal alignment.
  • Stronger base CLIP-like models should translate directly into stronger M3Bind performance, since the framework inherits each modality's image-text alignment.
  • Modalities with small datasets, such as ECG and retina, receive extra weight in training, so the framework can support rare modalities alongside data-rich ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If text anchoring generalizes, a natural extension is to bind modalities that share little or no text vocabulary by using a pivot modality's text, or by translating text spaces; the paper does not test such indirect binding.
  • The paper leaves unspecified how text-text pairs with the same semantic content are obtained; one testable implication is that the framework's gains should depend on semantic matching of reports rather than lexical overlap, so shuffling those pairs should degrade cross-modal retrieval.
  • Because the shared space is built from independent CLIP-like models, one could probe whether aligning text encoders alone is sufficient for image-image alignment by checking whether cross-modal image retrieval correlates with text similarity on held-out pairs.
  • The reported gains from adding each modality suggest a scaling effect: more anchor modalities should monotonically improve the shared space, which could be tested by binding additional modalities such as ultrasound or MRI.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes M3Bind, a pre-training framework that aligns five medical imaging modalities (X-ray, CT, retina, ECG, pathology) by binding their modality-specific CLIP-like image-text models through a shared text embedding space, without requiring explicit image-image pairs. The method first fine-tunes the text encoders of the five base models using a contrastive image-text loss plus an MSE loss between text embeddings of 'the same semantic content' across modalities, then distills the five text encoders into a unified BioClinicalBERT-based text encoder via SESKD. The authors evaluate zero-shot and few-shot classification and cross-modal retrieval on 12 datasets and report competitive or state-of-the-art results relative to the base CLIP-like models and several baselines.

Significance. If the central mechanism is valid, the framework is practically attractive because it turns multiple single-modality medical CLIP models into a unified multimodal space without collecting paired cross-modal images, and the evaluation is broad (five modalities, 12 datasets, three task types). The paper also gives credit for explicit adaptive modality balancing and a distillation stage that reduces the number of text encoders at inference. However, the main claim rests on the construction of semantically matched text-text pairs in Eq. (2), which is not specified, and the zero-shot protocol has unresolved data-split questions. The contribution is therefore not yet established even though the empirical scope is commendable.

major comments (3)
  1. [3.2, Eq. (2)] The key alignment loss L_MSE requires text embeddings 'corresponding to the same semantic content i' for two different modalities, but the manuscript never specifies how such cross-modal text-text pairs are constructed. Section 4.2 only states that text-modality alignment used a batch size of 64 with adaptive sampling; if the index i pairs the i-th text from an independently sampled batch of modality m1 with the i-th text from a batch of modality m2, the MSE loss aligns unrelated clinical texts and would corrupt the text space rather than bind modalities. Please specify the exact source of semantic correspondence (shared disease-label prompts, translated reports, same-patient reports across modalities, or another construction), give examples of the paired inputs, and explain how this does not reintroduce the paired-data burden the paper claims to remove.
  2. [4.2-4.3] The zero-shot evaluation protocol is incomplete. The model is fine-tuned on text data from each modality, but the paper does not state which corpus splits are used for M3Bind pre-training (Eq. (6)) and SESKD (Eq. (9)) and which are held out for evaluation. In particular, if text from MIMIC-ECG or CT-RATE contributes to pre-training, then the internal validation results reported in Table 1 for MIMIC-ECG and CT-RATE are not true zero-shot results. Please report exact dataset splits and confirm that no evaluation subjects or reports were seen during fine-tuning or distillation, including at the patient level for the MIMIC-PAIR retrieval experiment in Table 3.
  3. [3.2, 4.1, 4.3] The ECG results depend on an ECG-CLIP model trained by the authors, but the manuscript does not provide enough detail to reproduce it: no architecture, training data split, text preprocessing, loss function, or hyperparameters beyond the generic settings in Section 4.2. Since ECG-CLIP is the only baseline for MIMIC-ECG and PTB-XL zero-shot retrieval in Table 1 and is also a component of M3Bind, the missing details make fair comparison and reproducibility difficult. Please provide the training protocol or release the checkpoint.
minor comments (5)
  1. [3.2] The phrase 'novel pertaining method' should be 'novel pre-training method'.
  2. [Table 1] Table 1 contains typographical issues: 'MESIDOR' should be 'MESSIDOR', 'CT-RA TE' should be 'CT-RATE', and the ECG-CLIP row appears to be missing entries for the MIMIC-ECG and PTB-XL columns that are listed in the header.
  3. [Table 2] In Table 2, the MedBind rows use '−.−' for missing standard deviations; please use an explicit 'not reported' marker and add error bars if available.
  4. [Related Work] The related work cites DeCLIP [39] as leveraging multi-dimensional supervision for data efficiency, but reference [39] is a deepfake localization paper; the intended citation is the earlier DeCLIP work (Li et al., 2021) on multi-dimensional supervision.
  5. [4.6, Table 5] The ablation results in Table 5 are reported without standard deviations or significance tests; given the small differences (e.g., ODIR200x3 πnaive 0.452 vs. 0.478 with AMB), please state the number of seeds and, if possible, include variance.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: M3Bind's cross-modal alignment is produced by explicit training objectives and evaluated on held-out downstream tasks; no reported prediction reduces to a fitted constant or to a self-citation chain.

full rationale

The paper's derivation chain is a training pipeline, not a prediction derived from its own definition. Eq. (1) preserves each modality's image-text contrastive alignment; Eq. (2) adds an MSE loss that pushes text embeddings from different modality encoders together; Eq. (6) combines these with adaptive balancing; and Eqs. (7)-(9) distill the modality-specific text encoders into a BioClinicalBERT student. All of these are training objectives. The paper's claims of cross-modal alignment are then evaluated empirically on held-out zero-shot classification (Table 1), few-shot classification (Table 2), and cross-modal retrieval (Table 3) benchmarks such as CheXpert, MESIDOR, ODIR, PTB-XL, MIMIC-PAIR, and RAD-ChestCT. No table entry is identical by construction to any loss term, and no hyperparameter such as beta=0.5 or lambda=10 is presented as a fitted prediction of the downstream metrics. The framework cites external prior models (MGCA, CT-CLIP, FLAIR, QuiltNet, MedBind, BioClinicalBERT) and does not depend on a self-citation chain or on a 'uniqueness theorem' imported from the authors' own prior work. The main legitimate concern raised in the skeptical note is that Eq. (2) relies on text embeddings 'corresponding to the same semantic content i' without specifying how such text-text matches are constructed; however, that is an implementation and reproducibility gap, not a circular reduction. The central claim is not equivalent to its inputs by definition, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The framework rests on standard contrastive and distillation losses, the reliability of five pretrained CLIP-like models, and the availability of semantically matched text across modalities. The least-supported element is the text-text matching assumption behind Eq. (2); it is load-bearing and not described. All other entries are either standard optimization tools or domain-specific modeling choices.

free parameters (6)
  • beta (modality balancing exponent) = 0.5
    Set empirically in Section 3.3, Eq. (3); controls how strongly underrepresented modalities are oversampled and affects training balance.
  • lambda (inter-modality MSE weight) = 10
    Set in Section 4.2; weights the text-space alignment loss in Eq. (6); the paper says it prioritizes inter-modality text alignment before connecting modalities.
  • temperature tau = not reported
    Used in contrastive losses Eqs. (1) and (8); no value is reported, so the implementation is not fully reproducible.
  • base learning rate = 2e-5
    Set in Section 4.2 for optimization; chosen by hand and not swept.
  • batch sizes = 72 (most), 8 (CT-CLIP), 64 (text alignment)
    Configured in Section 4.2 according to model complexity and hardware constraints; affects contrastive learning and alignment quality.
  • training iterations = 15,000 pretraining; 1,200 plus 1,200 for SESKD
    Set in Section 4.2; the distillation schedule is split into two stages and chosen without reported sensitivity analysis.
assumptions (6)
  • standard math Contrastive losses and MSE are valid objectives for aligning embedding spaces.
    Used in Eqs. (1), (2), (7), and (8); these are standard optimization assumptions in representation learning.
  • domain assumption Pretrained medical CLIP-like models provide usable image-text alignments that survive LoRA fine-tuning.
    Sections 3.2 and 4.1 rely on MGCA, CT-CLIP, FLAIR, ECG-CLIP, and QuiltNet as starting points, and assume fine-tuning preserves their original image-text alignments.
  • ad hoc to paper Text-text matches with the same semantic content exist across modalities without image-image pairs.
    Eq. (2) requires text embeddings Tm1,i and Tm2,i for the same semantic content i; the paper does not specify how these text-text pairs are constructed across datasets.
  • domain assumption Text descriptions across modalities share enough semantics to act as a universal medical anchor.
    Section 1 argues that radiology reports, pathology annotations, retina evaluations, and other clinical text can bridge heterogeneous imaging modalities.
  • ad hoc to paper The inverse-dataset-size balancing scheme with beta=0.5 improves learning for small modalities.
    Section 3.3 introduces this strategy specifically for this framework and sets beta empirically without a sensitivity analysis.
  • domain assumption BioClinicalBERT can consolidate five teacher text spaces without losing image-text alignment.
    Section 3.4 assumes distillation with MSE and contrastive losses into a BioClinicalBERT student preserves the useful semantic structure of all teacher encoders.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Medical Image Binding via Shared Text Embeddings." pith.science (2026). https://pith.science/paper/JZRNTYPO

@misc{pith2026250618072,
  author       = {Pith},
  title        = {Pith review of: Multimodal Medical Image Binding via Shared Text Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JZRNTYPO}},
  note         = {Machine review of arXiv:2506.18072}
}
read the original abstract

Medical image analysis increasingly relies on the integration of multiple imaging modalities to capture complementary anatomical and functional information, enabling more accurate diagnosis and treatment planning. Achieving aligned feature representations across these diverse modalities is therefore important for effective multimodal analysis. While contrastive language-image pre-training (CLIP) and its variant have enabled image-text alignments, they require explicitly paired data between arbitrary two modalities, which is difficult to acquire in medical contexts. To address the gap, we present Multimodal Medical Image Binding with Text (M\textsuperscript{3}Bind), a novel pre-training framework that enables seamless alignment of multiple medical imaging modalities through a shared text representation space without requiring explicit paired data between any two medical image modalities. Specifically, based on the insight that different images can naturally bind with text, M\textsuperscript{3}Bind first fine-tunes pre-trained CLIP-like image-text models to align their modality-specific text embedding space while preserving their original image-text alignments. Subsequently, we distill these modality-specific text encoders into a unified model, creating a shared text embedding space. Experiments on X-ray, CT, retina, ECG, and pathological images on multiple downstream tasks demonstrate that M\textsuperscript{3}Bind achieves state-of-the-art performance in zero-shot, few-shot classification and cross-modal retrieval tasks compared to its CLIP-like counterparts. These results validate M\textsuperscript{3}Bind's effectiveness in achieving cross-image-modal alignment for medical analysis.

Figures

Figures reproduced from arXiv: 2506.18072 by the authors.

Figure 1
Figure 1. We illustrate the M3Bind framework, designed to embed multiple medical imaging modalities (X-ray, CT, retina, ECG, and Pathology) within a cohesive text-anchored space. In this figure, the transition from Phase One to Phase Two represents the five modality-specific CLIP-like models we used. These models start by aligning each modality with the text modality in isolation and then use text as the anchor modality, achi… view at source ↗
Figure 2
Figure 2. Overview of the M3Bind. We start with five modality-specific CLIP-like models (X-ray, CT, retina, ECG, and pathology). The M3Bind integrates these models by maintaining image-text alignment within each modality while aligning text representations across different modalities. LoRA is applied to efficiently fine-tune these pre-trained models while preserving their original capabilities. The Shared Embedding Space Know… view at source ↗
Figure 3
Figure 3. Comparison of Few-Shot Performance Across Various Datasets (ODIR200x3, CheXpert, SICAPv2, RAD-ChestCT) With In [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 43 canonical work pages

  1. [1]

    Yehuda Adler, Philippe Charron, Massimo Imazio, Luigi Badano, Gonzalo Bar ´on-Esquivias, Jan Bogaert, Antonio Brucato, Pascal Gueret, Karin Klingel, Christos Lionis, Bernhard Maisch, Bongani Mayosi, Alain Pavie, Arsen D Risti´c, Manel Sabat´e Tenas, Petar Seferovic, Karl Swedberg, Witold Tomkowski, and ESC Scientific Document Group. 2015 esc guidelines fo...

    work page 2015

  2. [2]

    Publicly available clinical bert embeddings

    Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323, 2019. 5, 6

    arXiv 1904

  3. [3]

    Prior: Prototype representation joint learning from medical images and reports

    Pujin Cheng, Li Lin, Junyan Lyu, Yijin Huang, Wenhan Luo, and Xiaoying Tang. Prior: Prototype representation joint learning from medical images and reports. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 21361–21371, 2023. 6, 7

    work page 2023

  4. [4]

    Quantifying tumour heterogeneity in 18 f-fdg pet/ct imaging by texture analysis

    Sugama Chicklore, Vicky Goh, Musib Siddique, Arunabha Roy, Paul K Marsden, and Gary JR Cook. Quantifying tumour heterogeneity in 18 f-fdg pet/ct imaging by texture analysis. European journal of nuclear medicine and molec- ular imaging, 40:133–140, 2013. 2

    work page 2013

  5. [5]

    As- sessment of tumor heterogeneity: an emerging imaging tool for clinical practice? Insights into imaging , 3:573–589,

    Fergus Davnall, Connie SP Yip, Gunnar Ljungqvist, Mariyah Selmi, Francesca Ng, Bal Sanghera, Balaji Gane- shan, Kenneth A Miles, Gary J Cook, and Vicky Goh. As- sessment of tumor heterogeneity: an emerging imaging tool for clinical practice? Insights into imaging , 3:573–589,

    work page

  6. [6]

    Feedback on a publicly distributed image database: the messidor database

    Etienne Decenci `ere, Xiwei Zhang, Guy Cazuguel, Bruno Lay, B ´eatrice Cochener, Caroline Trone, Philippe Gain, John-Richard Ord´o˜nez-Varela, Pascale Massin, Ali Erginay, et al. Feedback on a publicly distributed image database: the messidor database. Image Analysis & Stereology, pages 231–234, 2014. 7

    work page 2014

  7. [7]

    Maskclip: Masked self- distillation advances contrastive language-image pretraining

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self- distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10995–11005, 2023. 3

    work page 2023

  8. [8]

    Machine-learning-based multiple abnor- mality prediction with large-scale chest computed tomogra- phy volumes

    Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnor- mality prediction with large-scale chest computed tomogra- phy volumes. Medical image analysis, 67:101857, 2021. 7

    work page 2021

Show all 56 references
  1. [9]

    Medbind: Unifying language and multimodal medi- cal data embeddings

    Yuan Gao, Sangwook Kim, David E Austin, and Chris McIn- tosh. Medbind: Unifying language and multimodal medi- cal data embeddings. In International Conference on Med- ical Image Computing and Computer-Assisted Intervention , pages 218–228. Springer, 2024. 3, 4, 6, 7, 8

    2024

  2. [10]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 2, 3

    2023

  3. [11]

    Anal cancer: Esmo–esso–estro clinical practice guidelines for diagnosis, treatment and follow-up

    Robert Glynne-Jones, Per J Nilsson, Carlo Aschele, Vicky Goh, Didier Peiffert, Andr ´es Cervantes, and Dirk Arnold. Anal cancer: Esmo–esso–estro clinical practice guidelines for diagnosis, treatment and follow-up. Radiotherapy and Oncology, 111(3):330–339, 2014. 2

    2014

  4. [12]

    Mimic-iv-ecg: Diagnostic electrocardio- gram matched subset

    Brian Gow, Tom Pollard, Larry A Nathanson, Alistair John- son, Benjamin Moody, Chrystinne Fernandes, Nathaniel Greenbaum, Jonathan W Waks, Parastou Eslami, Tanner Carbonati, et al. Mimic-iv-ecg: Diagnostic electrocardio- gram matched subset. Type: dataset, 6:13–14, 2023. 7

    2023

  5. [13]

    A foundation model utiliz- ing chest ct volumes and radiology reports for supervised- level zero-shot detection of abnormalities

    Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Sim- sar, Mehmet Simsar, et al. A foundation model utiliz- ing chest ct volumes and radiology reports for supervised- level ze...

    2024

  6. [14]

    Distilling the knowledge in a neural net- work

    Geoffrey Hinton. Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531, 2015. 2, 3

    2015 arXiv

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  8. [16]

    Gloria: A multimodal global-local represen- tation learning framework for label-efficient medical image recognition

    Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local represen- tation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, ...

    2021

  9. [17]

    A visual–language foundation model for pathology image analysis using medical twitter

    Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307–2316, 2023. 6, 7

    2023

  10. [18]

    Modern diagnostic imaging technique applications and risk factors in the medical field: a review

    Shah Hussain, Iqra Mubeen, Niamat Ullah, Syed Shahab Ud Din Shah, Bakhtawar Abduljalil Khan, Muhammad Za- hoor, Riaz Ullah, Farhat Ali Khan, and Mujeeb A Sultan. Modern diagnostic imaging technique applications and risk factors in the medical field: a review. BioMed research i...

    2022

  11. [19]

    Quilt-1m: One million image-text pairs for histopathology, 2023

    Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fate- meh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology, 2023. 2, 3, 4, 6, 7

    2023

  12. [20]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Sil- viana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAA...

    2019

  13. [21]

    Tinybert: Distill- ing bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distill- ing bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019. 2

    1909 arXiv

  14. [22]

    Unimed- clip: Towards a unified image-text pretraining paradigm for diverse medical imaging modalities

    Muhammad Uzair Khattak, Shahina Kunhimon, Muzammal Naseer, Salman Khan, and Fahad Shahbaz Khan. Unimed- clip: Towards a unified image-text pretraining paradigm for diverse medical imaging modalities. arXiv preprint arXiv:2412.10372, 2024. 3

    2024 arXiv

  15. [23]

    Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sections

    Katharina Kriegsmann, Frithjof Lobers, Christiane Zgorzel- ski, Joerg Kriegsmann, Charlotte Janssen, Rolf R ¨udinger Meliß, Thomas Muley, Ulrich Sack, Georg Steinbuss, and Mark Kriegsmann. Deep learning for the detection of anatomical tissue structures and neoplasms of the ski...

    2022

  16. [24]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  17. [25]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  18. [26]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390– 23400, 2023. 3

    2023

  19. [27]

    Retinal vascular imaging: a new tool in microvascular dis- ease research

    Gerald Liew, Jie Jin Wang, Paul Mitchell, and Tien Y Wong. Retinal vascular imaging: a new tool in microvascular dis- ease research. Circulation: Cardiovascular Imaging , 1(2): 156–161, 2008. 1

    2008

  20. [28]

    An Open Access Database for Evaluating the Algorithms of Electro- cardiogram Rhythm and Morphology Abnormality Detec- tion

    Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaol- ing Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, Jianqing Li, and Eddie Ng Yin Kwee. An Open Access Database for Evaluating the Algorithms of Electro- cardiogram Rhythm and Morphology Abnormality Det...

    2018

  21. [29]

    Cells, tissues, and disease: principles of general pathology

    Guido Majno and Isabelle Joris. Cells, tissues, and disease: principles of general pathology . Oxford University Press,

  22. [30]

    Electrocardiography

    David M Mirvis and Ary L Goldberger. Electrocardiography. Heart disease, 1:82–128, 2001. 1

    2001

  23. [31]

    Im- proved knowledge distillation via teacher assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Im- proved knowledge distillation via teacher assistant. In Pro- ceedings of the AAAI conference on artificial intelligence , pages 5191–5198, 2020. 2

    2020

  24. [32]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In European conference on computer vision, pages 529–544. Springer, 2022. 2, 3

    2022

  25. [33]

    Imaging biomarker roadmap for cancer studies

    James PB O’connor, Eric O Aboagye, Judith E Adams, Hugo JWL Aerts, Sally F Barrington, Ambros J Beer, Ronald Boellaard, Sarah E Bohndiek, Michael Brady, Gina Brown, et al. Imaging biomarker roadmap for cancer studies. Nature reviews Clinical oncology, 14(3):169–186, 2017. 2

    2017

  26. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  27. [35]

    Computed tomography: revolutionizing the practice of medicine for 40 years

    Geoffrey D Rubin. Computed tomography: revolutionizing the practice of medicine for 40 years. Radiology, 273(2S): S45–S74, 2014. 1

    2014

  28. [36]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    V Sanh. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. 2

    1910 arXiv

  29. [37]

    Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detec- tion

    Julio Silva-Rodr ´ıguez, Adri ´an Colomer, Mar ´ıa A Sales, Rafael Molina, and Valery Naranjo. Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detec- tion. Computer methods and programs in bio...

    2020

  30. [38]

    A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision

    Julio Silva-Rodriguez, Hadi Chakor, Riadh Kobbi, Jose Dolz, and Ismail Ben Ayed. A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision. Medical Image Analysis, 99:103357, 2025. 2, 3, 4, 6, 7

    2025

  31. [39]

    De- clip: Decoding clip representations for deepfake localization

    Stefan Smeu, Elisabeta Oneata, and Dan Oneata. De- clip: Decoding clip representations for deepfake localization. arXiv preprint arXiv:2409.08849, 2024. 3

    2024 arXiv

  32. [40]

    Chest radiography in general practice: indications, diagnostic yield and consequences for patient management

    Anouk M Speets, Yolanda van der Graaf, Arno W Hoes, Sandra Kalmijn, Alfred PE Sachs, Matthieu JCM Rutten, Jan Willem C Gratama, Alexander D Montauban van Swijn- dregt, and Willem PThM Mali. Chest radiography in general practice: indications, diagnostic yield and consequences f...

    2006

  33. [41]

    Dime-fm: Distilling multimodal and efficient foundation models

    Ximeng Sun, Pengchuan Zhang, Peizhao Zhang, Hardik Shah, Kate Saenko, and Xide Xia. Dime-fm: Distilling multimodal and efficient foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 15521–15533, 2023. 3

    2023

  34. [42]

    Alpha- clip: A clip model focusing on wherever you want

    Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha- clip: A clip model focusing on wherever you want. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13019–13029, 2024. 3

    2024

  35. [43]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 2

    2021

  36. [44]

    Rotation equivariant cnns for digital pathology

    Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Co- hen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer As- sisted Intervention–MICCAI 2018: 21st International Con- ference, Granada, Spain, September 16-20, 2018,...

    2018

  37. [45]

    Ptb-xl, a large publicly available electrocardiog- raphy dataset

    Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Di- eter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiog- raphy dataset. Scientific data, 7(1):1–15, 2020. 7

    2020

  38. [46]

    Multi-granularity cross-modal align- ment for generalized medical visual representation learn- ing

    Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanab- huti, and Lequan Yu. Multi-granularity cross-modal align- ment for generalized medical visual representation learn- ing. Advances in Neural Information Processing Systems , 35:33536–33549, 2022. 2, 3, 4, 6, 7

    2022

  39. [47]

    Efficient image captioning for edge devices

    Ning Wang, Jiangrong Xie, Hang Luo, Qinglin Cheng, Jihao Wu, Mingbo Jia, and Linlin Li. Efficient image captioning for edge devices. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2608–2616, 2023. 3

    2023

  40. [48]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European con- ference on computer vision, pages 68–85. Springer, 2022. 2

    2022

  41. [49]

    Tinyclip: Clip dis- tillation via affinity mimicking and weight inheritance

    Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tinyclip: Clip dis- tillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on C...

    2023

  42. [50]

    Clip-kd: An empirical study of clip model distillation

    Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xin- qiang Yu, Han Yang, Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15952–15962, 2024. 3

    2024

  43. [51]

    Clip-cid: Efficient clip distillation via cluster-instance discrimination

    Kaicheng Yang, Tiancheng Gu, Xiang An, Haiqiang Jiang, Xiangzi Dai, Ziyong Feng, Weidong Cai, and Jiankang Deng. Clip-cid: Efficient clip distillation via cluster-instance discrimination. arXiv preprint arXiv:2408.09441, 2024. 3

    2024 arXiv

  44. [52]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 2

    2021 arXiv

  45. [53]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915,

    arXiv

  46. [54]

    Contrastive learning of medical visual representations from paired images and text

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference , pages 2–

  47. [55]

    Bira-net: Bilinear attention net for diabetic retinopathy grading

    Ziyuan Zhao, Kerui Zhang, Xuejie Hao, Jing Tian, Matthew Chin Heng Chua, Li Chen, and Xin Xu. Bira-net: Bilinear attention net for diabetic retinopathy grading. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1385–1389. IEEE, 2019. 6

    2019

  48. [56]

    Clip in medical imaging: A comprehensive sur- vey

    Zihao Zhao, Yuxiao Liu, Han Wu, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, and Ding- gang Shen. Clip in medical imaging: A comprehensive sur- vey. arXiv preprint arXiv:2312.07353, 2023. 2

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.