Pith. sign in

REVIEW 5 major objections 5 minor 56 references

M3Ret claims that a single ViT encoder, pretrained with purely visual self-supervision on 867,653 unpaired X-ray, ultrasound, endoscopy, and CT samples, achieves state-of-the-art zero-shot medical image retrieval—and generalizes to MRI it n

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

One self-supervised encoder trained on unpaired X-ray, ultrasound, endoscopy, and CT data gives competitive zero-shot retrieval and seems to generalize to unseen MRI tasks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Promising mixed-modality SSL result undercut by overclaims and unverified private test-set independence; deserves review after disclosure and baseline fixes. the 5 major comments →

arxiv 2509.01360 v1 pith:JQUGXRSB submitted 2025-09-01 cs.CV cs.LG

M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision

Che Liu , Zheng Jiang , Chengyu Fang , Heng Guo , Yan-Jie Zhou , Jiaqi Qu , Le Lu , Minfeng Xu This is my paper
classification cs.CV cs.LG
keywords zero-shot retrievalmedical image retrievalself-supervised learningmultimodal medical imagingmasked autoencoderSimDINOcross-modal generalization4D patchification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

M3Ret sets out to answer whether one vision model can represent every major medical imaging modality at once, without text labels, paired scans, or modality-specific architecture. The authors curate 867,653 unpaired X-ray, ultrasound, endoscopy, and CT samples, flatten all of them into the same spatiotemporal patch grid, and train a single ViT with two off-the-shelf self-supervised objectives, MAE and SimDINO. They report state-of-the-art zero-shot image-to-image retrieval on public benchmarks, beating the text-supervised BMC-CLIP and the 1.7-billion-image DINOv3, and the headline result is retrieval between CT and MRI even though MRI never appeared in pretraining. If this holds, the practical payoff is large: medical image retrieval would no longer depend on radiology reports or paired multimodal datasets, and one pretrained encoder could serve 2D, 3D, and video archives at once.

Core claim

The paper's central claim: unified, modality-agnostic visual representations for medical images can be learned entirely from self-supervision, given heterogeneous data at scale. M3Ret trains one ViT on 867,653 unpaired X-ray, ultrasound, endoscopy, and CT samples by flattening every input into the same 4D patch grid; both masked-autoencoding (MAE) and contrastive self-distillation (SimDINO) produce encoders that transfer zero-shot. On category-level retrieval, the SimDINO variant sets state-of-the-art results on ChestXray14 and Hyper Kvasir; on CT regional-abnormality and lesion-size retrieval it matches or beats supervised baselines with no labels at all. The strongest claim is cross-modal:

What carries the argument

The load-bearing mechanism is unified 4D patchification: every input—2D grayscale X-ray or ultrasound, RGB endoscopy video, 3D CT volume—is reshaped into a tensor X ∈ R^{3×256×256×S} and cut with the same spatiotemporal patch size (3,16,16,4), so a single ViT sees identical token structure regardless of modality. Two SSL objectives train that shared encoder: MAE, which reconstructs masked patches, and SimDINO, which aligns student and teacher [CLS] embeddings from augmented views under coding-rate regularization. The paper's claim is that this input standardization, not a custom loss, is what lets cross-modal alignment emerge.

Load-bearing premise

The private MRI, CT, and X-ray test sets used for the cross-modal results come from the same collaborating hospital as the 867,653-sample pretraining set, and the paper does not document that patients or exams are non-overlapping; if they overlap, the 'unseen MRI' numbers could reflect memorized body-region mappings rather than genuine transfer, and the Qwen2.5-generated DICOM body-part tags that define retrieval ground truth would then be doing the work.

What would settle it

Compute patient-level overlap between the pretraining set and the private cross-modal test sets (6,000 MRI, 4,500 CT, 9,500 X-ray). If the same patients or exams appear on both sides, rerun CT→MRI retrieval on MRI from an independent hospital; an R@1 near the body-region prior would falsify the generalization claim. A second check: shuffle the BodyPartExamined-derived tags that define retrieval ground truth—if retrieval accuracy tracks the coarse body-part labels rather than the image content, the reported cross-modal gains are inflated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval no longer requires radiology reports or image-text pairs; any unlabeled imaging archive becomes usable for pretraining, including modalities where text is scarce.
  • A single encoder handles 2D images, 3D volumes, and videos, making queries that cross dimensionality feasible—an X-ray query against a CT gallery, for instance.
  • Cross-modal alignment (CT↔X-ray, CT↔MRI) arises without paired data, implying that anatomical semantics rather than text are what unify the representation space.
  • Retrieval quality scales with data and model size along a fitted power law, so collecting more diverse heterogeneous scans is a direct route to better retrieval.
  • Contrastive self-distillation (SimDINO) is the stronger recipe for discriminative retrieval; pixel reconstruction (MAE) trails it on most tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The MRI result, if it survives testing on an independent hospital's data, suggests the encoder is effectively doing anatomy-based matching: shared body-region geometry between CT and MRI is what aligns them. That predicts retrieval to functionally different modalities like PET or histology would degrade—a boundary worth testing.
  • The generalization claim is demonstrated at body-part granularity (12 body parts for MRI), not disease-level semantics; disease-level cross-modal transfer remains untested and may not follow.
  • The reliance on Qwen2.5-generated DICOM body-part tags as ground truth raises the possibility that part of the reported cross-modal gain is coarse body-region classification in disguise rather than image-content-based retrieval.
  • The recipe (4D patchification plus SimDINO) is generic and should transfer to other heterogeneous image domains where dimensionalities differ, such as microscopy plus 3D imaging stacks; a negative result there would delimit the phenomenon.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes M3Ret, a unified visual encoder for zero-shot medical image-to-image retrieval. All inputs (2D X-ray/ultrasound, RGB endoscopy video, 3D CT) are mapped through a fixed 4D patchification (3,16,16,4) and trained with either MAE or SimDINO self-supervision on a private corpus of 867,653 samples. The paper evaluates category-level retrieval on ChestXray14, Fetal Planes, Kvasir Capsule and Hyper Kvasir; regional-abnormality and lesion-size retrieval on CT-RATE; and cross-modal CT/X-ray/MRI retrieval on private test sets, with additional scaling and ablation analyses. It claims state-of-the-art performance across all individual modalities and generalization to MRI despite no MRI in pretraining.

Significance. If validated, M3Ret would be a useful step: a simple modality-agnostic input scheme plus purely visual SSL, with no text or cross-modal pairing, yielding transferable representations on a heterogeneous medical corpus. The paper's strengths are its broad public benchmark coverage, comparison against several strong baselines (DINOv2/DINOv3/SigLIP2/BMC-CLIP/UniMiSS+/VoCo/Merlin), and systematic ablations of patch size, local views, embedding, data and model scale. However, the headline SOTA claim is contradicted by the authors' own Table 1, and the most novel result (unseen-MRI transfer) rests on private same-hospital test sets with unverified independence and label quality. These issues prevent the significance from being assessed at face value.

major comments (5)
  1. [Abstract; §4.3, Table 1] The paper claims a new state-of-the-art in zero-shot image-to-image retrieval 'across all individual modalities,' but Table 1 does not support this. On Kvasir Capsule, DINOv2-B has R@1 0.784/R@5 0.898/R@10 0.936, while M3Ret (SimDINO) has 0.647/0.872/0.913; on Fetal Planes, DINOv3-7B has R@1 0.971 vs 0.955 and also higher R@5/R@10. The text in §4.3 already concedes 'outperforms or matches,' which is weaker than the abstract. Please correct the headline claim and, if SOTA is claimed, provide per-metric significance tests or state precisely on which datasets/metrics the method is SOTA.
  2. [§3.4; Appendix C.1, Table 6; Tables 3/7] The central 'unseen MRI' generalization results are obtained on private MRI/CT/X-ray test sets drawn from the same collaborating hospital as the 867,653 pretraining samples, but no patient- or exam-level disjointness is stated. If any patients/exams overlap, the zero-shot evaluation is contaminated; the assertion that no MRI was seen in pretraining is also not audited. Please report a modality-tag and patient-ID check, provide an explicit de-duplication statement, or re-evaluate on independent public MRI data. Without this, Tables 3 and 7 cannot support the claim of generalization to an unseen modality.
  3. [Appendix C.2; Table 3] Cross-modal ground truth is built by using Qwen2.5 to map DICOM BodyPartExamined tags across modalities, with no reported validation. Because a retrieval is counted as correct whenever the mapped body-part tags match, the R@K numbers may reflect coarse body-region classification and mapping noise rather than semantic cross-modal alignment. Please report the mapping accuracy on a manually labeled subset, release the tag-mapping table, or use independently verified labels. This is especially important for the MRI results, where only 9-19 coarse body-part categories define positives.
  4. [§4.4, Fig. 3; Tables 1/2] The design choices for the final model—number of local views, embedding strategy, and patch size—were selected by looking at Recall@5 on ChestXray14 and CT-RATE (Fig. 3). Those same datasets are then used for the final numbers in Tables 1 and 2. This selection-on-test-data makes the 'zero-shot' evaluation optimistic and invalidates the claim that the reported configuration is a priori. Please move tuning to a separate validation partition (or a different dataset) and report final metrics only on untouched test sets.
  5. [§4.3, Table 3] The statement that M3Ret 'outperforms UniMiSS+' in CT↔X-ray retrieval is direction-dependent and not supported for the SimDINO variant. In X-ray→CT, UniMiSS+ achieves R@1=0.344/R@5=0.518 vs M3Ret(SimDINO) 0.285/0.502; M3Ret is superior only in the CT→X-ray direction. Please report the results per direction and avoid the unqualified superiority claim; the same issue affects the abstract's global SOTA claim.
minor comments (5)
  1. [Figure 1] The caption and legend text are garbled ('BMC-CLIPM!RetM!Ret (SimDINO)(MAE)(CVPR2025)'); please fix the LaTeX/PDF rendering and ensure all model names and labels are readable.
  2. [§4.2 and Table 2] VoCo is cited as [18] in Table 2 but described and linked to [35] in the baselines paragraph; clarify which VoCo checkpoint (and which pretraining task set) is used.
  3. [Appendix C.1] Appendix C.1 says 'three increasingly challenging tasks' while §4.1 defines two tasks (Regional Abnormality, Lesion Size); make the count consistent.
  4. [Fig. 4] The 'power law' claim is based on only three data ratios (20%, 60%, 100%) and no error bars; please qualify the claim or provide repeated runs/confidence intervals.
  5. [Appendix D] Qwen2.5-based cleaning of CT-RATE labels is used without manual validation; since it feeds Table 2, report inter-annotator agreement or a manual audit of a random sample.

Circularity Check

1 steps flagged

Reported ChestXray14 and CT-RATE zero-shot numbers are entangled with hyperparameter selection on the same test sets; private same-hospital MRI/CT/X-ray test sets lack stated disjointness from pretraining (correctness risk, not circularity).

specific steps
  1. fitted input called prediction [Section 4.4 'Analysis', Fig. 3; Tables 1 and 2]
    "As shown in Fig. 3 (c), we observe a clear trend: smaller patch sizes consistently improve performance across various downstream tasks. ... Recall@5 for ChestXray14 and CT-RATE datasets. (a) Performance under varying numbers of local crops. (b) Effect of embedding strategies. (c) Comparison of different patch sizes."

    The hyperparameter analysis in Sec. 4.4 evaluates local-view counts, embedding strategies, and patch sizes on ChestXray14 and CT-RATE, the same benchmarks later reported as headline zero-shot retrieval results (Table 1: ChestXray14; Table 2: CT-RATE). The final configuration is chosen after inspecting Recall@5 on these test sets, and then the same test-set numbers are presented as evidence that M3Ret 'sets a new state-of-the-art in zero-shot image-to-image retrieval.' The reported performance on these two datasets is therefore not an independent prediction under a fixed protocol; it is partly a selection-on-test estimate. This is a circular evaluation loop (fitted input renamed as prediction), although it does not directly contaminate the other public datasets or the private MRI cross-moda

full rationale

The paper's core pipeline—unified 4D patchification, MAE/SimDINO pretraining on 867,653 samples, and cosine-similarity retrieval—is self-contained and does not hinge on any equation-level circularity. The SSL objectives are standard, implemented from official code, and no load-bearing uniqueness theorem or author-derived ansatz is invoked. The two self-authored references ([34], [40]) appear only in related-work context and are not used to justify the central claim. The one genuine circular element is the benchmark selection loop: Fig. 3 uses ChestXray14 and CT-RATE to choose hyperparameters (local views, embedding, patch size), and Tables 1 and 2 then report those same datasets as zero-shot retrieval evidence. That makes the ChestXray14 and CT-RATE results partially a product of model selection rather than an independent test. Separately, Appendix C.1 introduces private MRI/CT/X-ray test sets from the same collaborating hospital as the pretraining data but never states that they are disjoint from the 867,653 pretraining samples, and it builds cross-modal ground truth from Qwen2.5-mapped DICOM BodyPartExamined tags without reported validation. This is a correctness risk for the 'unseen MRI' claim, but it is a missing-independence check rather than a circular derivation. Appendix E's limitations discuss modality coverage but do not disclose either the test-set overlap risk or the benchmark selection loop. Overall, the score reflects the partial circularity of the benchmark selection loop, while the method's derivation and the majority of its evaluations retain independent content.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The ledger is dominated by selection-on-test (patch size), undisclosed split independence, and LLM-derived labels rather than new physical or conceptual entities. No invented forces, particles, or dimensions are introduced.

free parameters (5)
  • Patch size (Hp, Wp, Sp) = (16, 16, 4) = 16x16x4 (after comparing 8x8x4 and 32x32x4)
    Chosen using downstream retrieval performance on ChestXray14 and CT-RATE (Fig. 3c), so the final configuration is selected on test data.
  • Number of local views in SimDINO = 10 for 2D, 4 for 3D/video
    Chosen partly for GPU memory; paper shows performance is stable across 2-10 local views (Fig. 3a). Hand-chosen, not fitted.
  • MAE mask ratio alpha = 0.75
    Taken from MAE paper [1], not fit to medical data.
  • SimDINO epsilon = 0.5
    From SimDINO paper [2], not fit to data.
  • Augmentation crop scales (sg, sl) = not reported
    Algorithms 1-3 reference global and local crop scales but no numeric values are given, so the actual augmentation strength is an undisclosed hand-chosen setting.
axioms (4)
  • ad hoc to paper 4D patchification with fixed channel dimension and S (slices/frames) as a token axis creates a shared semantic space across 2D, video, and 3D modalities.
    Section 3.1; this is the core unification mechanism, with no theoretical or empirical proof that semantically aligning a 4-slice X-ray replica and a 64-slice CT volume is meaningful.
  • domain assumption Private pretraining set and private MRI/CT/X-ray test sets are disjoint and from a comparable clinical distribution.
    Appendix C.1; the paper does not state patient/exam-level separation, so the MRI generalization claim depends on this unstated premise.
  • domain assumption Qwen2.5-based cleaning/harmonization of HyperKvasir, RadGenome, and DICOM BodyPartExamined labels is clinically accurate.
    Appendix C.1 and C.2; LLM relabeling could introduce systematic label noise correlated with visual appearance, inflating retrieval metrics.
  • domain assumption SimDINO's coding-rate-regularized contrastive loss and EMA teacher remain non-collapsing and discriminative on mixed-modality medical inputs.
    Section 3.3; the loss is inherited from natural-image SSL and there is no analysis of modality collapse in the shared space.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision." pith.science (2026). https://pith.science/paper/JQUGXRSB

@misc{pith2026250901360,
  author       = {Pith},
  title        = {Pith review of: M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQUGXRSB}},
  note         = {Machine review of arXiv:2509.01360}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Medical image retrieval is essential for clinical decision-making and translational research, relying on discriminative visual representations. Yet, current methods remain fragmented, relying on separate architectures and training strategies for 2D, 3D, and video-based medical data. This modality-specific design hampers scalability and inhibits the development of unified representations. To enable unified learning, we curate a large-scale hybrid-modality dataset comprising 867,653 medical imaging samples, including 2D X-rays and ultrasounds, RGB endoscopy videos, and 3D CT scans. Leveraging this dataset, we train M3Ret, a unified visual encoder without any modality-specific customization. It successfully learns transferable representations using both generative (MAE) and contrastive (SimDINO) self-supervised learning (SSL) paradigms. Our approach sets a new state-of-the-art in zero-shot image-to-image retrieval across all individual modalities, surpassing strong baselines such as DINOv3 and the text-supervised BMC-CLIP. More remarkably, strong cross-modal alignment emerges without paired data, and the model generalizes to unseen MRI tasks, despite never observing MRI during pretraining, demonstrating the generalizability of purely visual self-supervision to unseen modalities. Comprehensive analyses further validate the scalability of our framework across model and data sizes. These findings deliver a promising signal to the medical imaging community, positioning M3Ret as a step toward foundation models for visual SSL in multimodal medical image understanding.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 44 canonical work pages · 6 internal anchors

  1. [1]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of CVPR, 2022, pp. 16000–16009

    work page 2022

  2. [2]

    Simplifying dino via coding rate regularization,

    Z. Wu, J. Zhang, D. Pai, X. Wang, C. Singh, J. Yang, J. Gao, and Y. Ma, “Simplifying dino via coding rate regularization,” arXiv preprint arXiv:2502.10385, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,

    X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of CVPR, 2017, pp. 3462–3471

    work page 2017

  4. [4]

    Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes,

    X. P. Burgos-Artizzu, D. Coronado-Gutiérrez, B. Valenzuela-Alcaraz, E. Bonet-Carne, E. Eixarch, F. Crispi, and E. Gratacós, “Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes,”Scientific Reports, vol. 10, 2020

    work page 2020

  5. [5]

    Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy,

    H. Borgli, V. Thambawita, P. H. Smedsrud, S. Hicks, D. Jha, S. L. Eskeland, K. R. Randel, K. Pogorelov, M. Lux, D. T. D. Nguyenet al., “Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy,” Scientific data, vol. 7, no. 1, p. 283, 2020

    work page 2020

  6. [6]

    Developing generalist foundation models from a multimodal dataset for 3d computed tomography,

    I. E. Hamamci, S. Er, C. Wang, F. Almas, A. G. Simsek, S. N. Esirgun, I. Doga, O. F. Durugol, W. Dai, M. Xu et al., “Developing generalist foundation models from a multimodal dataset for 3d computed tomography,”arXiv preprint arXiv:2403.17834, 2024

    arXiv 2024

  7. [7]

    Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature,

    A. Lozano, M. W. Sun, J. Burgess, L. Chen, J. J. Nirschl, J. Gu, I. Lopez, J. Aklilu, A. W. Katzer, C. Chiu, A. Rau, X. Wang, Y. Zhang, A. S. Song, R. Tibshirani, and S. Yeung-Levy, “Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature,”ArXiv, vol. abs/2501.07171, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Unimiss+: Universal medical self-supervised learning from cross-dimensional unpaired data,

    Y. Xie, J. Zhang, Y. Xia, and Q. Wu, “Unimiss+: Universal medical self-supervised learning from cross-dimensional unpaired data,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10021–10035, 2024

    work page 2024

  9. [9]

    A review of content-based image retrieval systems in medical applications—clinical benefits and future directions,

    H. Müller, N. Michoux, D. Bandon, and A. Geissbuhler, “A review of content-based image retrieval systems in medical applications—clinical benefits and future directions,”International journal of medical informatics, vol. 73, no. 1, pp. 1–23, 2004

    work page 2004

  10. [10]

    Image retrieval: Ideas, influences, and trends of the new age,

    R. Datta, D. Joshi, J. Li, and J. Z. Wang, “Image retrieval: Ideas, influences, and trends of the new age,”ACM Computing Surveys (Csur), vol. 40, no. 2, pp. 1–60, 2008

    work page 2008

  11. [11]

    Content- based image retrieval by using deep learning for interstitial lung disease diagnosis with chest ct,

    J. Choe, H. J. Hwang, J. B. Seo, S. M. Lee, J. Yun, M.-J. Kim, J. Jeong, Y. Lee, K. Jin, R. Parket al., “Content- based image retrieval by using deep learning for interstitial lung disease diagnosis with chest ct,”Radiology, vol. 302, no. 1, pp. 187–197, 2022

    work page 2022

  12. [12]

    Medical image retrieval using deep convolutional neural network,

    A. Qayyum, S. M. Anwar, M. Awais, and M. Majid, “Medical image retrieval using deep convolutional neural network,” Neurocomputing, vol. 266, pp. 8–20, 2017

    work page 2017

  13. [13]

    Large-scale retrieval for medical image analytics: A comprehensive review,

    Z. Li, X. Zhang, H. Müller, and S. Zhang, “Large-scale retrieval for medical image analytics: A comprehensive review,” Medical Image Analysis, vol. 43, pp. 66–84, 2018

    work page 2018

  14. [14]

    Large- scale domain-specific pretraining for biomedical vision-language processing,

    S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wonget al., “Large- scale domain-specific pretraining for biomedical vision-language processing,”arXiv preprint arXiv:2303.00915, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Radiology objects in context (roco): A multimodal image dataset,

    O. Pelka, S. Koitka, J. Rückert, F. Nensa, and C. Friedrich, “Radiology objects in context (roco): A multimodal image dataset,” inCVII-STENT/LABELS@MICCAI, 2018

    work page 2018

  16. [16]

    Rad-dino: Exploring scalable medical image encoders beyond text supervision,

    F. P’erez-Garc’ia, H. Sharma, S. Bond-Taylor, K. Bouzid, V. Salvatelli, M. Ilse, S. Bannur, D. C. Castro, A. Schwaighofer, M. P. Lungren, M. T. Wetscherek, N. Codella, S. L. Hyland, J. Alvarez-Valle, and O. Oktay, “Rad-dino: Exploring scalable medical image encoders beyond text supervision,”ArXiv, vol. abs/2401.10815, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Self-supervised pre-training of swin transformers for 3d medical image analysis,

    Y. Tang, D. Yang, W. Li, H. R. Roth, B. A. Landman, D. Xu, V. Nath, and A. Hatamizadeh, “Self-supervised pre-training of swin transformers for 3d medical image analysis,”Proceedings of CVPR, pp. 20698–20708, 2021

    work page 2021

  18. [18]

    Voco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis,

    L. Wu, J. Zhuang, and H. Chen, “Voco: A simple-yet-effective volume contrastive learning framework for 3d medical image analysis,” inProceedings of CVPR, 2024, pp. 22873–22882

    work page 2024

  19. [19]

    Dira: Discriminative, restorative, and adversarial learning for self-supervised medical image analysis,

    F. Haghighi, M. R. H. Taher, M. B. Gotway, and J. Liang, “Dira: Discriminative, restorative, and adversarial learning for self-supervised medical image analysis,” inProceedings of CVPR, 2022, pp. 20824–20834

    work page 2022

  20. [20]

    The bitter lesson,

    R. Sutton, “The bitter lesson,”Incomplete Ideas (blog), 2019. [Online]. Available: http://www.incompleteideas. net/IncIdeas/BitterLesson.html

    work page 2019

  21. [21]

    Content-based medical image retrieval,

    T. W. Cai, J. Kim, and D. D. Feng, “Content-based medical image retrieval,” inBiomedical information technology. Elsevier, 2008, pp. 83–113

    work page 2008

  22. [22]

    Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data,

    A. Kumar, J. Kim, W. Cai, M. Fulham, and D. Feng, “Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data,”Journal of digital imaging, vol. 26, pp. 1025–1039, 2013

    work page 2013

  23. [23]

    Rocov2: Radiology objects in context version 2, an updated multimodal image dataset,

    J. Rückert, L. Bloch, R. Brüngel, A. Idrissi-Yaghir, H. Schäfer, C. S. Schmidt, S. Koitka, O. Pelka, A. B. Abacha, A. G. S. de Herrera, H. Müller, P. A. Horn, F. Nensa, and C. M. Friedrich, “Rocov2: Radiology objects in context version 2, an updated multimodal image dataset,”Scientific Data, vol. 11, 2024

    work page 2024

  24. [24]

    Medical Image Retrieval Using Pretrained Embeddings

    F. K. Jush, T. Truong, S. Vogler, and M. Lenga, “Medical image retrieval using pretrained embeddings,”arXiv preprint arXiv:2311.13547, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [25]

    Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query

    H. H. Lee, A. Santamaria-Pang, J. Merkow, O. Oktay, F. Pérez-García, J. Alvarez-Valle, and I. Tarapov, “Region- based contrastive pretraining for medical image retrieval with anatomic query,”arXiv preprint arXiv:2305.05598, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    X-mir: Explainable medical image retrieval,

    B. Hu, B. Vasu, and A. Hoogs, “X-mir: Explainable medical image retrieval,” inProceedings of WACV, 2022, pp. 440–450

    work page 2022

  27. [27]

    Sketch-based semantic retrieval of medical images,

    K. Kobayashi, L. Gu, R. Hataya, T. Mizuno, M. Miyake, H. Watanabe, M. Takahashi, Y. Takamizawa, Y. Yoshida, S. Nakamuraet al., “Sketch-based semantic retrieval of medical images,”Medical Image Analysis, vol. 92, p. 103060, 2024

    work page 2024

  28. [28]

    Bimcv-r: A landmark dataset for 3d ct text-image retrieval,

    Y. Chen, C. Liu, X. Liu, R. Arcucci, and Z. Xiong, “Bimcv-r: A landmark dataset for 3d ct text-image retrieval,” in Proceedings of MICCAI, 2024

    work page 2024

  29. [29]

    3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology

    A. B. Abacha, A. Santamaría-Pang, H. H. Lee, J. T. Merkow, Q. Cai, S. T. Devarakonda, A. Islam, J. Gong, M. P. Lungren, T. Lin, N. Codella, and I. Tarapov, “3d-mir: A benchmark and empirical study on 3d medical image retrieval in radiology,”ArXiv, vol. abs/2311.13752, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Big self-supervised models advance medical image classification,

    S. Azizi, B. Mustafa, F. Ryan, Z. Beaver, J. Freyberg, J. Deaton, A. Loh, A. Karthikesalingam, S. Kornblith, T. Chenet al., “Big self-supervised models advance medical image classification,” inProceedings of ICCV, 2021, pp. 3478–3488

    work page 2021

  31. [31]

    Representing part-whole hierarchies in foundation models by learning localizability composability and decomposability from anatomy via self supervision,

    M. R. H. Taher, M. B. Gotway, and J. Liang, “Representing part-whole hierarchies in foundation models by learning localizability composability and decomposability from anatomy via self supervision,” inProceedings of CVPR, 2024, pp. 11269–11281

    work page 2024

  32. [32]

    Revisiting rubik’s cube: self-supervised learning with volume-wise transformation for 3d medical image segmentation,

    X. Tao, Y. Li, W. Zhou, K. Ma, and Y. Zheng, “Revisiting rubik’s cube: self-supervised learning with volume-wise transformation for 3d medical image segmentation,” inProceedings of MICCAI. Springer, 2020, pp. 238–248

    work page 2020

  33. [33]

    Models genesis,

    Z. Zhou, V. Sodha, J. Pang, M. B. Gotway, and J. Liang, “Models genesis,”Medical Image Analysis, vol. 67, p. 101840, 2021

    work page 2021

  34. [34]

    Anatomical invariance modeling and semantic alignment for self-supervised learning in 3d medical image analysis,

    Y. Jiang, M. Sun, H. Guo, X. Bai, K. Yan, L. Lu, and M. Xu, “Anatomical invariance modeling and semantic alignment for self-supervised learning in 3d medical image analysis,” inProceedings of ICCV, 2023, pp. 15859– 15869

    work page 2023

  35. [35]

    Large-scale 3d medical image pre-training with geometric context priors,

    L. Wu, J. Zhuang, and H. Chen, “Large-scale 3d medical image pre-training with geometric context priors,”arXiv preprint arXiv:2410.09890, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    A unified visual information preservation framework for self-supervised pre-training in medical image analysis,

    H.-Y. Zhou, C. Lu, C. Chen, S. Yang, and Y. Yu, “A unified visual information preservation framework for self-supervised pre-training in medical image analysis,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8020–8035, 2023

    work page 2023

  37. [37]

    Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier,

    Y. Xie, J. Zhang, Y. Xia, and Q. Wu, “Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier,” inProceedings of ECCV, 2021

    work page 2021

  38. [38]

    Merlin: A vision language foundation model for 3d computed tomography,

    L. Blankemeier, J. P. Cohen, A. Kumar, D. V. Veen, S. J. S. Gardezi, M. Paschali, Z. Chen, J.-B. Delbrouck, E. P. Reis, C. A. M. Truyts, C. Bluethgen, M. E. K. Jensen, S. Ostmeier, M. Varma, J. M. J. Valanarasu, Z. Fang, Z. Huo, Z. Nabulsi, D. Ardila, W.-H. Weng, E. A. Junior, N. Ahuja, J. A. Fries, N. H. Shah, A. Johnston, R. D. Boutin, A. Wentland, C. P...

    work page 2024

  39. [39]

    Pmc-clip: Contrastive language-image pre-training using biomedical documents,

    W. Lin, Z. Zhao, X. Zhang, C. Wu, Y. Zhang, Y. Wang, and W. Xie, “Pmc-clip: Contrastive language-image pre-training using biomedical documents,” inProceedings of MICCAI. Springer, 2023, pp. 525–536

    work page 2023

  40. [40]

    Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyond

    Y. Zhang, P. Hager, C. Liu, S. Shit, C. Chen, D. Rueckert, and J. Pan, “Towards cardiac mri foundation models: Comprehensive visual-tabular representations for whole-heart assessment and beyond,”arXiv preprint arXiv:2504.13037, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [41]

    Multi-modal Vision Pre-training for Medical Image Analysis

    S. Rui, L. Chen, Z. Tang, L. Wang, M. Liu, S. Zhang, and X. Wang, “Brainmvp: Multi-modal vision pre-training for brain image analysis using multi-parametric mri,”arXiv preprint arXiv:2410.10604, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Continual self-supervised learning: Towards universal multi-modal medical data representation learning,

    Y. Ye, Y. Xie, J. Zhang, Z. Chen, Q. Wu, and Y. Xia, “Continual self-supervised learning: Towards universal multi-modal medical data representation learning,” inProceedings of CVPR, 2024, pp. 11114–11124

    work page 2024

  43. [43]

    Unified Medical Image Pre-training in Language-Guided Common Semantic Space

    X. He, Y. Yang, X. Jiang, X. Luo, H. Hu, S. Zhao, D. Li, Y. Yang, and L. Qiu, “Unified medical image pre-training in language-guided common semantic space,”ArXiv, vol. abs/2311.14851, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    Segmentation of multivariate mixed data via lossy data coding and compression,

    Y. Ma, H. Derksen, W. Hong, and J. Wright, “Segmentation of multivariate mixed data via lossy data coding and compression,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 9, pp. 1546–1562, 2007

    work page 2007

  45. [45]

    Learning diverse and discriminative representations via the principle of maximal coding rate reduction,

    Y. Yu, K. H. R. Chan, C. You, C. Song, and Y. Ma, “Learning diverse and discriminative representations via the principle of maximal coding rate reduction,”Advances in neural information processing systems, vol. 33, pp. 9422–9434, 2020

    work page 2020

  46. [46]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” inProceedings of ICLR, 2021

    work page 2021

  47. [47]

    Kvasir-capsule, a video capsule endoscopy dataset,

    P. H. Smedsrud, V. L. Thambawita, S. Hicks, H. L. Gjestang, O. O. Nedrejord, E. Næss, H. Borgli, D. Jha, T. J. D. Berstad, S. L. Eskeland, M. Lux, H. N. Espeland, A. Petlund, D. T. D. Nguyen, E. Garcia-Ceja, D. Johansen, P. T. Schmidt, E. Toth, H. L. Hammer, T. de Lange, M. Riegler, and P. Halvorsen, “Kvasir-capsule, a video capsule endoscopy dataset,”Sci...

    work page 2020

  48. [48]

    Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis,

    X. Zhang, C. Wu, Z. Zhao, J. Lei, Y. Zhang, Y. Wang, and W. Xie, “Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis,”ArXiv, vol. abs/2404.16754, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Noubyet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafaet al., “Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,”arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Siméoni, H

    O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoaet al., “Dinov3,”arXiv preprint arXiv:2508.10104, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    Vision foundation models for computed tomography,

    S. Pai, I. Hadzic, D. Bontempi, K. Bressem, B. H. Kann, A. Fedorov, R. H. Mak, and H. J. Aerts, “Vision foundation models for computed tomography,”arXiv preprint arXiv:2501.09001, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” inProceedings of ICCV, 2021, pp. 9650–9660

    work page 2021

  54. [54]

    Unsupervised learning of visual features by contrasting cluster assignments,

    M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,”Advances in neural information processing systems, vol. 33, pp. 9912–9924, 2020

    work page 2020

  55. [55]

    Qwen2.5-vl technical report,

    S. Bai, K. qin Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025

    work page 2025

  56. [56]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024. Acknowledgement The LaTeX template is built upon Meta’s original template. Appendix Contents A Pretraining Data Distribution 15 B Details of Data Augmentation in SimDINO 15 C Downstream Task Configu...

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.