Pith. sign in

REVIEW 7 cited by

Exploring scalable medical image encoders beyond text supervision

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.10815 v3 pith:DXNSS4Q3 submitted 2024-01-19 cs.CV

Fernando Pérez-García , Harshita Sharma , Sam Bond-Taylor , Kenza Bouzid , Valentina Salvatelli , Maximilian Ilse , Shruthi Bannur , Daniel C. Castro
show 7 more authors
Anton Schwaighofer Matthew P. Lungren Maria Teodora Wetscherek Noel Codella Stephanie L. Hyland Javier Alvarez-Valle Ozan Oktay
This is my paper
classification cs.CV
keywords rad-dinobiomedicalimagingmedicalsupervisiondatafeaturesimage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, the computed features are limited by the information contained in the text, which is particularly problematic in medical imaging, where the findings described by radiologists focus on specific observations. This challenge is compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general-purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language-supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder. Model weights of RAD-DINO trained on publicly available datasets are available at https://huggingface.co/microsoft/rad-dino.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding

    cs.CV 2026-07 conditional novelty 7.0 of 10

    RadPRISM trains chest-X-ray vision-language models on a 19-concept clinician schema by aligning each concept's text spans in its own visual subspace, improving zero-shot AUROC from 0.717 to 0.868 over global alignment...

  2. CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A chest X-ray VLM co-trained with classification and grounding heads, tuned with DAPO reinforcement learning, and augmented with deterministic measurement tools outperforms prior radiology VLMs on report generation, V...

  3. RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture

    cs.CV 2026-01 unverdicted novelty 6.0 of 10

    RadJEPA learns chest X-ray encoders from unlabeled images via latent prediction in a joint embedding architecture, exceeding prior state-of-the-art on classification, segmentation, and report generation.

  4. M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision

    cs.CV 2025-09 conditional novelty 6.0 of 10

    One self-supervised encoder trained on unpaired X-ray, ultrasound, endoscopy, and CT data gives competitive zero-shot retrieval and seems to generalize to unseen MRI tasks.

  5. Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ALTA adapts a frozen masked-pretrained X-ray encoder to language with 8% trainable parameters and temporal-multiview inputs, improving medical retrieval and zero-shot classification.

  6. VET-DINO: Learning Anatomical Understanding Through Multi-View Distillation in Veterinary Imaging

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Using real multi-view radiographs from the same study as self-supervised training pairs yields better anatomical representations and downstream veterinary task performance than synthetic single-image augmentations.

  7. Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation

    eess.IV 2025-06 conditional novelty 4.0 of 10

    MedRegion-CT integrates region-representative tokens, mask-driven segmentation tokens, and patient-specific attribute prompts into a multimodal LLM, reporting state-of-the-art scores on RadGenome-Chest CT report generation.

Pith tools