REVIEW 15 cited by
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using around 200K data). Our code is available at https://github.com/RyanWangZf/MedCLIP.
Forward citations
Cited by 15 Pith papers
-
CheXTemporal: A Dataset for Temporally-Grounded Reasoning in Chest Radiography
CheXTemporal supplies paired chest X-rays with explicit temporal progression taxonomy and spatial grounding to benchmark and improve models on longitudinal reasoning tasks.
-
Tables Guide Vision: Learning to See the Heart through Tabular Data
Tabular clinical data guides contrastive learning on cardiac MR images to build better visual representations by identifying patient similarities, outperforming image-only augmentation on downstream disease prediction tasks.
-
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
BiomedCLIP, pretrained on the new 15-million-pair PMC-15M dataset, achieves state-of-the-art performance on diverse biomedical vision-language tasks and even outperforms radiology-specific models on chest X-ray pneumo...
-
CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
A chest X-ray VLM co-trained with classification and grounding heads, tuned with DAPO reinforcement learning, and augmented with deterministic measurement tools outperforms prior radiology VLMs on report generation, V...
-
Cross-Contextual Vision-Language Adaptation with LoRA for Personalized Severe Adverse Event Detection in Clinical Wound Monitoring
Cross-contextual dual-stream LoRA on BiomedCLIP plus multi-signal temporal OOD scoring detects personalized SAEs in longitudinal diabetic foot ulcer images better than unimodal baselines on one clinical trial dataset.
-
MAM-CLIP: Vision-Language Pretraining on Mammography Atlases for BI-RADS Classification
Contrastive pretraining on mammography atlas image-text pairs improves BI-RADS classification F1 by 1-14% especially in low-label regimes, outperforming equivalent numbers of direct labels in some settings.
-
GLAM: Geometry-Guided Local Alignment for Multi-View VLP in Mammography
GLAM adds geometry-guided local alignment to mammography visual-language pre-training and outperforms prior VLP baselines on EMBED, VinDr, and RSNA.
-
Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
A 3D encoder pretrained with GPT-4V slice captions and partial optimal transport alignment beats vision-only SSL baselines on several medical tasks, but a key evaluation dataset may overlap with pretraining.
-
Primus: Enforcing Attention Usage for 3D Medical Image Segmentation
Primus and PrimusV2 are Transformer-centric models that match or exceed nnU-Net and top CNNs on nine 3D medical segmentation datasets by enforcing attention usage.
-
UniVAD v2: Unified Visual Anomaly Detection via Support-Conditioned Boundary Construction
UniVAD v2 improves 1N-shot mean image-level AUC from 83.0% to 84.5% (85.7% with one abnormal reference) via support-conditioned boundary construction on six datasets.
-
CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift
CADRE combines LoRA with self-scaling similarity-aware EWC and an anchor-to-prior penalty to achieve lower forgetting and positive backward transfer when adapting a medical VLM across dissimilar imaging modalities whi...
-
BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
BiomedAP improves robustness of biomedical VLMs to prompt variations using gated cross-modal fusion and dual-anchor constraints, outperforming baselines on 11 benchmarks.
-
Align then Refine: Text-Guided 3D Prostate Lesion Segmentation
A text-guided multi-encoder U-Net with alignment loss, heatmap calibration, and confidence-gated cross-attention refiner sets new state-of-the-art 3D prostate lesion segmentation performance on the PI-CAI dataset.
-
Automated Radiology Report Generation Based on Topic-Keyword Semantic Guidance
A topic-keyword semantic guidance framework improves automated radiology report generation and reaches state-of-the-art on two public chest X-ray benchmarks.
-
Navigating Distribution Shifts in Medical Image Analysis: A Survey
Survey categorizing DL methods for distribution shifts in MedIA by clinical scenarios, with analysis indicating constrained gains as domain information decreases and a shift toward uncertainty-aware modeling.
Discussion (0). Sign in to comment.