Pith. sign in

REVIEW 4 major objections 6 minor 68 references

VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Aligning coarse-to-fine visual features with report-, sentence-, and word-level text lets a 3D medical vision-language model beat larger models while pre-training on only 38,875 CT-report pairs.

desk verdict The framework is plausible and the ablation story is mostly coherent, but the SOTA claim is undercut by the data-curation confound and an unexplained name change. read the letter →

arxiv 2508.12108 v1 pith:Z4TQB6YC submitted 2025-08-16 cs.CV

Ziyang Zhang , Yang Yu , Xulei Yang , Si Yong Yeo This is my paper · ORCID
classification cs.CV
keywords vision-languagepre-trainingvolumetricmedicalimaging3DCTscanshierarchicalcontrastivelearningself-supervisedradiologyreportgenerationcross-modalretrievalvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Volumetric medical imaging lacks the large scan-report datasets that power 2D vision-language models, and the paper sets out to show that clever pre-training can substitute for data volume. VELVET-Med predicts that if a text encoder is taught to represent reports, sentences, and words separately, and those levels are aligned one-by-one with coarse-to-fine visual features of a 3D CT scan, then strong transferable encoders can be learned from only 38,875 pairs. The resulting encoders are claimed to reach state-of-the-art performance on 3D semantic segmentation, cross-modal retrieval, visual question answering, and report generation. The paper also argues that a smaller, manually cleaned dataset of anatomically ordered volumes beats a larger noisy one, making data quality a central part of the result.

What carries the argument

The load-bearing mechanism is TriBERT, a BERT-style text encoder augmented with a learnable [SENT] token per sentence and sentence-type embeddings, plus a tri-level attention mask that lets sentence tokens attend only to their own sentence and the global [CLS] token, preventing inter-sentence leakage. Its three text levels (report, sentence, word) are matched by a hierarchical contrastive learning objective to three visual levels (top, middle, bottom feature maps), replacing a single global CLIP-style loss. Uni-modal self-supervised losses on both modalities complete the training signal. This three-layer alignment is what the paper credits for capturing both coarse concepts and fine-grained anatomical detail, and it is the component that carries the reported transferability.

What would settle it

Train VELVET-Med with identical hyperparameters on the unfiltered M3D-CAP set (120,092 pairs) and on M3D-CAP-filtered (38,875 pairs), then compare retrieval and segmentation; if the filtered model does not clearly beat the unfiltered one, the gains are curation rather than the architecture or objectives. A second check is to have independent radiologists re-score the excluded scans: if many excluded volumes have clinically valid anatomy, the filter is introducing selection bias.

Watch

Extended reading notes

Core claim

The central discovery is that cross-modal alignment for volumetric medical data should be hierarchical rather than whole-pair-only. VELVET-Med aligns the three deepest visual feature maps of a 3D Swin-based vision encoder to report-level, sentence-level, and word-level embeddings of a custom text encoder called TriBERT, while simultaneously training each modality alone with self-supervision (masked volume inpainting, rotation prediction, contrastive coding for images; masked language modeling for text). With this combination, the paper reports that the pre-trained encoders transfer to four downstream families of tasks and beat the prior M3D-based models even though pre-training uses only 38,875 scan-report pairs. The paper further claims that filtering the M3D-CAP dataset to exclude scans with disordered z-axis slice order and fewer than 48 slices substantially improves retrieval, indicating that dataset curation is itself a source of the gains.

Load-bearing premise

The entire reported advantage rests on the assumption that the manually filtered M3D-CAP-filtered set of 38,875 scan-report pairs is a fair representation of real medical CT data, rather than an easy subset created by excluding scans the authors judged atypical.

Editorial extensions

If this is right

  • Pre-training 3D medical vision-language models no longer requires millions of paired scans; a few tens of thousands of clean pairs can be enough if the learning objectives exploit volumetric structure.
  • Text encoders for radiology should represent sentence structure explicitly, since whole-report BERT-style encoding misses the sentence-level dependencies that hierarchical contrastive learning uses.
  • Adding uni-modal self-supervision to cross-modal pre-training is expected to help downstream segmentation and retrieval more than cross-modal alignment alone.
  • A modest curated dataset can outperform a larger automatically collected one, so filtering protocols such as z-axis order and slice count deserve to be a standard part of volumetric medical vision-language pre-training pipelines.
  • If the framework is modality-agnostic as claimed, the same recipe should work for MRI or ultrasound volumes paired with reports.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the dataset filter is not automated, so the clean-versus-noisy comparison conflates data curation with the model; a same-architecture ablation on unfiltered data would isolate the curation effect.
  • Editorial inference: the paper's own table shows that multi-modal matching can hurt retrieval when added to cross-modal contrast, suggesting the optimal weighting of the four losses is task-dependent and that a tuned schedule could push results further.
  • Editorial inference: TriBERT's sentence-level masking is a general idea that could extend to any multi-sentence document paired with images, such as pathology reports or longitudinal imaging, and to sentence-level grounding of findings in regions.
  • Editorial inference: scaling these objectives to seven-digit pair counts, with curriculum learning or distillation, is a natural next test; the current 38,875-pair results do not bound what the same objectives could do at larger scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VELVET-Med, a vision-and-language pre-training framework for 3D CT volumes paired with radiology reports. The main contributions are a TriBERT language encoder that models report-, sentence-, and word-level semantics through special input embeddings and an attention mask; a hierarchical contrastive learning objective that aligns top/middle/bottom visual features with report/sentence/word text embeddings; and the inclusion of uni-modal vision and language self-supervised losses alongside cross-modal and multi-modal objectives. The model is pre-trained on M3D-CAP-filtered, a manually curated subset of M3D-CAP containing 38,875 scan-report pairs, and then evaluated on 3D segmentation (AbdomenCT-1K, CT-ORG), cross-modal retrieval, generative and classification-based VQA, and medical report generation. The paper claims state-of-the-art transferability across these tasks.

Significance. If the results hold, the paper makes a useful contribution to data-efficient 3D medical vision-language pre-training. The TriBERT sentence-aware encoding and the hierarchical contrastive alignment are reasonable architectural ideas, and the combination of uni-modal and cross-modal objectives is worth studying in the low-data volumetric setting. The paper also promises to release a curated dataset, which would aid reproducibility. However, the significance is currently conditional: the evaluation does not isolate the proposed method from the dataset curation step, the main generative-task baseline is a re-implementation, all results appear to be single-run point estimates, and the full proposed objective is not the best configuration in the paper's own retrieval ablation. These issues must be addressed before the SOTA claims can be accepted.

major comments (4)
  1. [Sec. 4.1; Appendix A; Table 3] The central claim that the reported gains come from TriBERT, hierarchical contrastive learning, and uni-modal supervision is not separable from the dataset curation step. Pre-training is performed exclusively on M3D-CAP-filtered (38,875 pairs), obtained by manually excluding scans judged to have disordered z-axis slice order and scans with fewer than 48 slices. Appendix A states that scans are stored without metadata and gives no automated criterion for 'disordered,' and the promised release of the filter and filtered dataset is absent. The paper's own data-quality experiment (Table 3) shows that holding the architecture fixed (BERT-B/ViT-B) and moving from the unfiltered M3D retrieval model to CLIP-3D on M3D-CAP-filtered raises SRR R@1 from 19.10 to 26.29, a larger jump than most subsequent architectural or objective changes. Because the filter is validated on the same benchmark used for the SOTA claims and is not externally validated on independent clinical data, the reported margins over prior M3D-based models could be substantially due to selection of easier or cleaner volumes. To support the attribution, the authors should train the M3D retrieval baseline and CLIP-3D on the same filtered data, train the proposed method on unfiltered or intermediate-quality data, and provide an automated, reproducible filter criterion or external validation of the curation decisions.
  2. [Table 3] The full VELVET-Med objective combination is not the best configuration in the paper's own retrieval results. In Table 3, VELVET-Med (Lcm,Lmm,Llan,Lvis) obtains SRR R@1 34.86 and RSR R@1 33.12, while Lcm+Llan+Lvis reaches 36.18 and 36.60, and Lcm+Lvis reaches 36.85 and 35.37. The text says VELVET-Med 'achieves competitive retrieval results,' but the ablation does not support the claim that the complete proposed model is the best or that adding multi-modal supervision helps retrieval. The authors need to explain this degradation, report which configuration is recommended for each downstream task, and state whether the full model was selected on a held-out validation criterion or by another principle.
  3. [Tables 1–4; Appendix B.3] All experiments appear to be single runs without error bars or significance testing, and several reported differences are small (e.g., Table 2 Dice 94.03 vs 93.96; Table 4 classification-based VQA AUC 84.27 vs 84.01). The SOTA claim therefore rests on point estimates that may be within run-to-run noise. The authors should provide results over multiple seeds with means and standard deviations and, where appropriate, significance tests. In addition, the generative VQA and report-generation results are fine-tuned on a 10% subsample disclosed only in Appendix B.3; this caveat should be stated in the main tables and taken into account when claiming that VELVET-Med achieves the best performance on those tasks.
  4. [Table 4; Appendix B.3] The generative VQA and report-generation comparisons are not on equal footing with the official M3D-LaMed baseline. The baseline M3D-LaMed$ is the authors' own re-implementation using the pre-trained vision encoder from CLIP-3D, and both the proposed and baseline generative models are fine-tuned on a 10% subsample of M3D-VQA/M3D-CAP-filtered. The 'state-of-the-art' claim on these tasks therefore depends on a non-official baseline and a small subsample. Please report official M3D-LaMed numbers, evaluate on the full fine-tuning set if feasible, or clearly qualify the claim and state the subsample size in the main text.
minor comments (6)
  1. [Table 3; Table 4; Tables C/D] The row label 'Lcm,Lvis,Llan,Lvis(VELVET-Med)' in Table 3 duplicates Lvis and does not match the objective set in Eq. (4); it should read Lcm,Lmm,Llan,Lvis. The same inconsistency appears in Table 4, and Tables C/D refer to 'De-MedViL' instead of VELVET-Med.
  2. [Appendix A] There is an unresolved placeholder '2D interpolation [cite]' in Appendix A; this citation needs to be completed before publication.
  3. [Sec. 3.5] The notation for visual features is inconsistent: the text writes 'f^k_v in R^{N x c^k_v x h^k x w^k x d^i}' with mixed superscripts and a dangling 'di'. This should be corrected to a single coherent form such as R^{N x c^k_v x h^k x w^k x d^k}.
  4. [Eq. (5)] The total loss is written as an unweighted sum Lvis + Llan + Lcm + Lmm; please state whether all weights are fixed to 1 and whether any sensitivity analysis or tuning of these weights was performed.
  5. [Tables 1 and 3] The cross-modal retrieval evaluation uses a 'M3D-CAP-filtered test set,' but the construction and size of this 2000-pair test set are not fully described in the main text. In addition, Table 1 reports scan sizes 256,256,32 alongside 96,96,96; the meaning of these size tuples should be clarified.
  6. [Throughout] There are several typos and grammatical issues (e.g., 'adatabple' in the introduction, 'parings' in Section 1) that should be corrected in a careful proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: pre-training losses are self-supervised, downstream evaluations are external or held-out, and the dataset-curation comparison is a confound rather than a derivation.

full rationale

VELVET-Med's central claim is that its pre-training objectives and TriBERT architecture produce transferable encoders. The objectives Lvis, Llan, Lcm, and Lmm are self-supervised; they never use downstream task labels, and no equation defines a reported metric as one of these losses. Segmentation is measured on AbdomenCT-1K and CT-ORG with SwinUNETR initialized from the pre-trained encoder; retrieval is evaluated on a held-out M3D-CAP-filtered split without fine-tuning; VQA and report generation use M3D-VQA and M3D-CAP-filtered fine-tuning with frozen vision and language encoders. These are standard transfer evaluations, not reductions of predictions to inputs. The only potentially confounded comparison is the 'Effect of data quality' experiment in Section 4.4 and Table 3, where the manually curated M3D-CAP-filtered set is compared against the M3D retrieval model trained on the original M3D-CAP. This is a dataset-curation attribution issue: the exclusion criteria are manual and described in Appendix A without an automated or externally validated criterion, so the reported gains may partly reflect the filter rather than the architecture. However, this is an experimental confound, not circularity: the filter is not a fitted parameter, no test label is used to choose it, and no prediction reduces by construction to a training input or to a self-citation. The sole self-citation, reference [3], appears in related-work listings and is not load-bearing. The appended limitations section explicitly acknowledges the small pre-training scale and single-modality scope, further indicating that the claims are empirical rather than definitional.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on dataset curation choices, an assumed visual-textual level correspondence, and unablated loss balancing. No new physical or ontological entities are introduced; the TriBERT [SENT] tokens and sentence-type embeddings are architectural components, not postulated entities in the sense of the graviton problem.

free parameters (4)
  • Slice count threshold = 48
    Scans with fewer than 48 slices are excluded from M3D-CAP-filtered, a hand-chosen threshold motivated by avoiding repeated or zero-padded regions. This filtering affects every downstream result.
  • Disordered scan exclusion criterion = Manual inspection, no numeric threshold
    Scans with disordered z-axis slice arrangement are excluded based on visual inspection with no objective, reproducible criterion, introducing selection bias risk.
  • Loss weighting = Equal (implied)
    The final loss L = Lvis + Llan + Lcm + Lmm is used without reported per-term weights or a weighting ablation, so the balance among four objectives is an unexamined design choice.
  • Input caps for text = 50 sentences, 200 words per sentence, 512 words per report
    These caps constrain TriBERT and are chosen without analysis of their effect on long medical reports.
assumptions (4)
  • domain assumption M3D-CAP scan-report pairs are correctly aligned and clinically meaningful
    The entire pre-training and evaluation relies on the correctness of this public dataset's pairing, stated in Section 4.1.
  • ad hoc to paper Top/mid/bottom visual feature levels correspond semantically to report/sentence/word text levels
    The hierarchical contrastive loss in Section 3.5 assumes this correspondence, but no experiment directly measures whether the visual hierarchy matches the text hierarchy.
  • domain assumption Filtering out disordered and few-slice scans improves generalization to real clinical data
    The dataset curation in Section 4.1 and Appendix A assumes that out-of-distribution scans hinder learning; validated only on the filtered distribution, not on external cohorts.
  • ad hoc to paper The four self-supervised losses can be jointly optimized without weighting
    Equation (5) sums Lvis, Llan, Lcm, and Lmm directly; no gradient scaling or loss-balancing scheme is given, leaving the multi-task balance unexamined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine." pith.science (2026). https://pith.science/paper/Z4TQB6YC

@misc{pith2026250812108,
  author       = {Pith},
  title        = {Pith review of: VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4TQB6YC}},
  note         = {Machine review of arXiv:2508.12108}
}
read the original abstract

Vision-and-language models (VLMs) have been increasingly explored in the medical domain, particularly following the success of CLIP in general domain. However, unlike the relatively straightforward pairing of 2D images and text, curating large-scale paired data in the medical field for volumetric modalities such as CT scans remains a challenging and time-intensive process. This difficulty often limits the performance on downstream tasks. To address these challenges, we propose a novel vision-language pre-training (VLP) framework, termed as \textbf{VELVET-Med}, specifically designed for limited volumetric data such as 3D CT and associated radiology reports. Instead of relying on large-scale data collection, our method focuses on the development of effective pre-training objectives and model architectures. The key contributions are: 1) We incorporate uni-modal self-supervised learning into VLP framework, which are often underexplored in the existing literature. 2) We propose a novel language encoder, termed as \textbf{TriBERT}, for learning multi-level textual semantics. 3) We devise the hierarchical contrastive learning to capture multi-level vision-language correspondence. Using only 38,875 scan-report pairs, our approach seeks to uncover rich spatial and semantic relationships embedded in volumetric medical images and corresponding clinical narratives, thereby enhancing the generalization ability of the learned encoders. The resulting encoders exhibit strong transferability, achieving state-of-the-art performance across a wide range of downstream tasks, including 3D segmentation, cross-modal retrieval, visual question answering, and report generation.

Figures

Figures reproduced from arXiv: 2508.12108 by the authors.

Figure 1
Figure 1. Framework of VELVET-Med. The model takes paired CT scans and medical reports as [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Novel TriBERT design. (a) Input Embedding Construction: Unlike standard BERT, TriBERT inserts learnable [SENTi] tokens into the input sequence to encode sentence semantics. It also adds sentence type embeddings to differentiate between sentences. (b) Tri-level Self-Attention Masking: To prevent inter-sentence information leakage, each [SENTi] token attends only to its associated word tokens and the [CLS] token. This… view at source ↗
Figure 3
Figure 3. Visualization of self-attention maps: self- [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons with VELVET-MED and ground truth on open-ended VQA. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 32 canonical work pages

  1. [1]

    Gloria: A multimodal global- local representation learning framework for label-efficient medical image recognition

    Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global- local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021

    work page 2021

  2. [2]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, 2022

    work page 2022

  3. [3]

    Medunifier: Unifying vision-and- language pre-training on medical data with vision generation task using discrete visual representations

    Ziyang Zhang, Yang Yu, Yucheng Chen, Xulei Yang, and Si Yong Yeo. Medunifier: Unifying vision-and- language pre-training on medical data with vision generation task using discrete visual representations. arXiv preprint arXiv:2503.01019, 2025

    arXiv 2025

  4. [4]

    Towards unifying medical vision-and-language pre-training via soft prompts

    Zhihong Chen et al. Towards unifying medical vision-and-language pre-training via soft prompts. In CVPR, 2023

    work page 2023

  5. [5]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  6. [6]

    Conditional prompt learning for vision- language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816–16825, 2022

    2022

  7. [7]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022

    2022

  8. [8]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18134–18144, 2022

    work page 2022

Show all 68 references
  1. [9]

    Learning to exploit temporal structure for biomedical vision-language processing

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the...

    2023

  2. [10]

    Cplip: zero-shot learning for histopathology with comprehensive vision-language alignment

    Sajid Javed, Arif Mahmood, Iyyakutti Iyappan Ganapathi, Fayaz Ali Dharejo, Naoufel Werghi, and Mohammed Bennamoun. Cplip: zero-shot learning for histopathology with comprehensive vision-language alignment. In Proceedings of the IEEE/CVF conference on computer vision and patter...

    2024

  3. [11]

    Lu, Bowen Chen, Andrew Zhang, Drew F

    Ming Y . Lu, Bowen Chen, Andrew Zhang, Drew F. K. Williamson, Richard J. Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF Conference on ...

    2023

  4. [12]

    Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019. 11

    2019

  5. [13]

    Quilt-1m: One million image-text pairs for histopathology

    Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. arXiv preprint arXiv:2306.11207, 2023

    2023 arXiv

  6. [14]

    Towards generalist foundation model for radiology

    Weidi Xie, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, and Yanfeng Wang. Towards generalist foundation model for radiology. 2023

    2023

  7. [15]

    M3d: Advancing 3d medical image analysis with multi-modal large language models

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578, 2024

    2024 arXiv

  8. [16]

    Align before fuse: Vision and language representation learning with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021

    2021

  9. [17]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lan...

    2019

  10. [18]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020

    2020

  11. [19]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR, 2020

    2020

  12. [20]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural i...

    2020

  13. [21]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  14. [22]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020

    2020

  15. [23]

    Unsupervised learning of visual representations by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69–84. Springer, 2016

    2016

  16. [24]

    Colorization as a proxy task for visual understanding

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6874–6883, 2017

    2017

  17. [25]

    Self-supervised representation learning by rotation feature decoupling

    Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning by rotation feature decoupling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10364–10374, 2019

    2019

  18. [26]

    What makes instance discrimination good for transfer learning? arXiv preprint arXiv:2006.06606, 2020

    Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? arXiv preprint arXiv:2006.06606, 2020

    2006 arXiv

  19. [27]

    Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning

    Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. InProceedings of the I...

    2023

  20. [28]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  21. [29]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 12

    1901

  22. [30]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  23. [31]

    Med3d: Transfer learning for 3d medical image analysis

    Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis. arXiv preprint arXiv:1904.00625, 2019

    1904 arXiv

  24. [32]

    Models genesis

    Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang. Models genesis. Medical image analysis, 67:101840, 2021

    2021

  25. [33]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019

    2019

  26. [34]

    Lxmert: Learning cross-modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019

    1908 arXiv

  27. [35]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer, 2020

    2020

  28. [36]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,...

    2020

  29. [37]

    Vinvl: Revisiting visual representations in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588, 2021

    2021

  30. [38]

    Coca: Contrastive captioners are image-text foundation models, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022

    2022

  31. [39]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  32. [40]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  33. [41]

    Multi-modal understanding and generation for medical images and text via vision-language pre-training

    Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young-Hak Kim, and Edward Choi. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE Journal of Biomedical and Health Informatics, 26(12):6070–6080, 2022

    2022

  34. [42]

    Pmc- vqa: Visual instruction tuning for medical visual question answering

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023

    2023 arXiv

  35. [43]

    Slip: Self-supervision meets language- image pre-training

    Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In European conference on computer vision, pages 529–544. Springer, 2022

    2022

  36. [44]

    Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm, 2022

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm, 2022

    2022

  37. [45]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  38. [46]

    Self-supervised pre-training of swin transformers for 3d medical image analysis

    Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognit...

    2022

  39. [47]

    Masked image modeling advances 3d medical image analysis, 2022

    Zekai Chen, Devansh Agarwal, Kshitij Aggarwal, Wiem Safta, Samit Hirawat, Venkat Sethuraman, Mariann Micsinai Balan, and Kevin Brown. Masked image modeling advances 3d medical image analysis, 2022. 13

    2022

  40. [48]

    V oco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis

    Linshan Wu, Jiaxin Zhuang, and Hao Chen. V oco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22873–22882, 2024

    2024

  41. [49]

    Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. CoRR, abs/1604.07379, 2016

    2016 arXiv

  42. [50]

    Unsupervised representation learning by predicting image rotations

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. CoRR, abs/1803.07728, 2018

    2018 arXiv

  43. [51]

    Representation learning with contrastive predictive coding

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018

    2018 arXiv

  44. [52]

    Linear-time wordpiece tokenization

    Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Linear-time wordpiece tokenization. CoRR, abs/2012.15524, 2020

    2012 arXiv

  45. [53]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of...

    2021

  46. [54]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pages 272–284. Springer, 2021

    2021

  47. [55]

    Abdomenct-1k: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6695–6714, 2022

    Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, Shucheng Cao, Qi Zhang, Shangqing Liu, Yunpeng Wang, Yuhui Li, Jian He, and Xiaoping Yang. Abdomenct-1k: Is abdominal organ segmentation a solved problem? IEEE Transac...

    2022

  48. [56]

    Ct-org, a new dataset for multiple organ segmentation in computed tomography

    Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin. Ct-org, a new dataset for multiple organ segmentation in computed tomography. Scientific Data, 7(1):381, 2020

    2020

  49. [57]

    Ledsam, and Olaf Ronneberger

    Stanislav Nikolov, Sam Blackwell, Alexei Zverovitch, Ruheena Mendes, Michelle Livne, Jeffrey De Fauw, Yojan Patel, Clemens Meyer, Harry Askham, Bernardino Romera-Paredes, Christopher Kelly, Alan Karthikesalingam, Carlton Chu, Dawn Carnell, Cheng Boon, Derek D’Souza, Syed Ali M...

    2021

  50. [58]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p...

    2002

  51. [59]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. page 10, 01 2004

    2004

  52. [60]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  53. [61]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  54. [62]

    https://huggingface.co/ContactDoctor/Bio- Medical-Llama-3-8B, 2024

    Bio-medical: A high-performance biomedical language model. https://huggingface.co/ContactDoctor/Bio- Medical-Llama-3-8B, 2024

    2024

  55. [63]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  56. [64]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  57. [65]

    Segvol: Universal and interactive volumetric medical image segmentation

    Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. Segvol: Universal and interactive volumetric medical image segmentation. Advances in Neural Information Processing Systems, 37:110746–110783, 2024

    2024

  58. [66]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 14

    2024

  59. [67]

    Pmc- clip: Contrastive language-image pre-training using biomedical documents

    Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- clip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. ...

    2023

  60. [68]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate, 2022. 15 Table A: Statis...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.