Pith. sign in

REVIEW 8 cited by

MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06542 v1 pith:OE422LH6 submitted 2024-10-09 eess.IV cs.CV

MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

Noel C. F. Codella , Ying Jin , Shrey Jain , Yu Gu , Ho Hin Lee , Asma Ben Abacha , Alberto Santamaria-Pang , Will Guyman
show 23 more authors
Naiteek Sangani Sheng Zhang Hoifung Poon Stephanie Hyland Shruthi Bannur Javier Alvarez-Valle Xue Li John Garrett Alan McMillan Gaurav Rajguru Madhu Maddi Nilesh Vijayrania Rehaan Bhimai Nick Mecklenburg Rupal Jain Daniel Holstein Naveen Gaur Vijay Aski Jenq-Neng Hwang Thomas Lin Ivan Tarapov Matthew Lungren Mu Wei
This is my paper
classification eess.IV cs.CV
keywords medimageinsightmedicalsotaimagingothersearchachievesacross
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Semantically Calibrated Evidence Composition for CT Vision-Language Learning

    cs.CV 2026-07 conditional novelty 6.0

    SCOPE composes organ-level CT evidence under whole-volume context and calibrates it with diagnostic-summary supervision, reaching 85.0/72.2 macro AUC on CT-RATE and RadChestCT.

  2. A Scale-adaptive Vision Model Links C. elegans Neuronal Morphology to Behavior for Neurotoxicity Assessment

    cs.CE 2026-07 conditional novelty 6.0

    A scale-adaptive vision model links C. elegans neuron morphology to behavior, predicts dopamine-dependent deficits, and identifies the benzimidazole moiety as a neurotoxicity determinant.

  3. Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning

    cs.CV 2026-06 unverdicted novelty 6.0

    ConQuer augments global CLIP alignment with independent per-concept contrastive losses on anatomical regions extracted from reports, producing Jolia which outperforms CLIP baselines on classification, report generatio...

  4. Frozen Foundation-Model Embeddings Discard Small-Lesion Signal in Chest Radiography: Implications for Pre-Deployment Evaluation

    cs.CV 2026-06 unverdicted novelty 6.0

    Frozen ViT embeddings in chest radiography suppress small-lesion signal at the CLS token but recover it via patch-local pooling on the same forward pass across multiple models and large cohorts.

  5. Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation

    cs.CV 2026-02 unverdicted novelty 6.0

    MARL-Rad trains region-specific and global agents with reinforcement learning on clinical rewards to produce more accurate radiology reports than prior methods on MIMIC-CXR and IU X-ray datasets.

  6. CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis

    cs.CV 2026-05 unverdicted novelty 5.0

    CapCLIP uses pathology-aware text captions to align WCE images in a vision-language space, outperforming standard models in zero-shot classification and retrieval on unseen data.

  7. Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification

    cs.CV 2025-12 conditional novelty 5.0

    A frozen 2D vision foundation model with lightweight LoRA adapters and attention-based slice fusion achieves state-of-the-art 3D medical image classification across 12 tasks with about 1M trainable parameters per task.

  8. 3D Foundation Model for Generalizable Disease Detection in Head Computed Tomography

    cs.CV 2025-02 unverdicted novelty 5.0

    A 3D self-supervised foundation model trained on over 360k head CT scans improves downstream disease classification on limited-label internal and external datasets versus scratch-trained and prior models.