Pith. sign in

REVIEW 2 major objections 7 minor 106 references

Cleaner medical data beats bigger data: 11M curated pairs top 24M

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 02:47 UTC pith:EGQNXQP3

load-bearing objection Solid data infrastructure paper with a real attribution gap in its central experiment the 2 major comments →

arxiv 2607.07673 v1 pith:EGQNXQP3 submitted 2026-07-08 cs.CV cs.LG

MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

Hyunjae Kim , Dain Kim , Pan Xiao , Serina S. Applebaum , Younjoon Chung , Xuguang Ai , Yu Yin , Roy Jiang
show 20 more authors
Yuexi Du Yawen Wei Yiming Kong Tuo Guo Zhiyuan Cao Mengmeng Du Yuelei Fu Yan Hu Rui Shi Gui Yang Kevin W. Jin Yuntian Liu Yuxuan Tian Jonathan Marquez Zhen Chen Sheng Zhang Hoifung Poon Hua Xu Jaewoo Kang Qingyu Chen
This is my paper
classification cs.CV cs.LG
keywords medicalmultimodalmodelsacrossdatamedpmcbenchmarksclinical
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the quality of image-text pairs matters more than raw quantity when training medical vision-language models. The authors built MedPMC, a five-stage automated pipeline that mines PubMed Central's open-access biomedical literature for clinically relevant images. The pipeline screens out non-medical figures (graphs, charts, molecular diagrams), decomposes compound multi-panel figures into individual subimages, aligns each subimage with its corresponding subcaption, and filters the result for medical relevance. Applied to 6.1 million articles, it produced 11 million image-text pairs with 95.3% medical relevance, versus 19.7% in a prior 24-million-pair dataset. The central claim is tested through a controlled experiment: a CLIP-style model trained on the 11M MedPMC corpus, using identical architecture and hyperparameters to a baseline trained on the 24M BIOMEDICA corpus, improved average zero-shot AUC by 7.1 percentage points across 26 benchmarks spanning 11 medical specialties, improved medical visual question-answering by up to 16.9 percentage points when used as the vision encoder in a multimodal language model, and improved morphology-to-image retrieval by 11.7 percentage points on 10,524 internal clinical dermatology photographs. The paper frames data curation not as preprocessing but as reusable, continuously updatable infrastructure, with modular components that can be individually benchmarked, swapped, and refined as new literature is published.

Core claim

The paper's central discovery is that systematically removing non-medical images, decomposing compound figures into panel-level units, and aligning each panel with its specific subcaption yields training data that produces stronger medical AI models than datasets more than twice as large but lacking this curation. The mechanism is signal-to-noise: when 80% of a corpus consists of graphs, charts, and schematics rather than clinically relevant images, the contrastive learning objective wastes capacity on irrelevant visual content. By filtering to 95.3% medical relevance and achieving panel-level image-text correspondence, each training pair provides supervision at the level where clinical视觉特征最

What carries the argument

The five-stage curation pipeline (initial screening via PubMedBERT text classifier, multi-panel figure detection via Vision Transformer, figure separation via YOLOv10, caption separation and alignment via supervised InternVL-2.5-4B, medical figure classification via Vision Transformer) is the central object. The controlled comparison design—identical CLIP architecture, initialization, hyperparameters, and training schedule, with only the training corpus changed—is the mechanism that isolates dataset quality as the causal variable. The modular benchmark suite for each pipeline stage is the infrastructure that makes the framework auditable and continuously improvable.

Load-bearing premise

The controlled comparison attributing all performance gains to dataset quality assumes that the only meaningful difference between MedPMC's 11M pairs and BIOMEDICA's 24M pairs is the curation applied. However, three of the five curation pipeline stages use GPT-4 Turbo to generate synthetic training labels, and if those synthetic labels encode systematic biases or medical knowledge that the downstream CLIP model then absorbs, the performance gain could partly reflect distilled

What would settle it

If a CLIP model trained on a random 11M subset of BIOMEDICA (without MedPMC's curation but matched for size) achieved comparable performance to MedPMC-CLIP, the curation pipeline's contribution would be confounded with corpus size effects. Alternatively, if the pipeline's synthetic GPT-4T labels were replaced with human annotations and the final CLIP model's performance changed substantially, the gains would be attributable to label quality rather than data curation per se.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If data fidelity dominates data scale for medical multimodal pretraining, then the field's emphasis on building ever-larger corpora may be less productive than investing in curation pipelines that improve signal-to-noise ratio.
  • The modular, continuously updatable pipeline design means that as new medical imaging modalities, diseases, and terminology emerge in the literature, the corpus can be refreshed semiannually without rebuilding from scratch, keeping pretrained models current.
  • The finding that literature-derived images transfer to clinical dermatology retrieval suggests that curated biomedical publications can serve as a pretraining substrate that complements, rather than competes with, institution-specific clinical datasets.
  • The release of component-level benchmarks for each curation stage establishes a standardized evaluation framework that could drive competition and improvement in individual curation subtasks, much as task-specific benchmarks have advanced other areas of machine learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The use of GPT-4 Turbo to generate synthetic training labels for three of the five pipeline stages means the curation pipeline's quality is partly bounded by GPT-4T's own medical knowledge. If GPT-4T systematically mislabels certain image types or introduces domain-specific biases, these errors propagate into the curated corpus and could be learned by downstream models. The paper does not ablate t
  • The 7.1-point AUC gain over BMC-CLIP is attributed solely to dataset quality, but the two corpora differ in composition (MedPMC has 11M pairs, BIOMEDICA has 24M), and the smaller corpus may benefit from a more favorable signal-to-noise ratio per gradient update rather than per pair. A scaling-curve comparison at matched corpus sizes would more cleanly isolate curation quality from dataset size eff
  • The embedding-space analysis showing MedPMC dermatology images overlap more with clinical photographs than existing public dermatology datasets do suggests that biomedical literature may capture visual diversity that curated clinical datasets miss, but this could also reflect publication bias toward visually distinctive or representative cases rather than the full distribution of routine clinical

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This manuscript introduces MedPMC, a five-stage modular pipeline for curating medical image-text pairs from permissively licensed PubMed Central articles. The pipeline performs initial screening, multi-panel figure detection, figure separation, joint caption separation and alignment, and medical figure classification, yielding 11M pairs from 6.1M articles. The authors train MedPMC-CLIP using the exact architecture and training protocol of BMC-CLIP (which trains on the 24M-pair BIOMEDICA corpus) to isolate the effect of the dataset. They report a 7.1 pp improvement in average zero-shot AUC across 26 benchmarks spanning 11 specialties, gains in two MLLM QA benchmarks when MedPMC-CLIP replaces the vision encoder in LLaVA-Med, and an 11.7 pp improvement in Recall@5 on a morphology-to-image retrieval task using 10,524 internal clinical dermatology photographs. The framework, corpus, benchmarks, and model checkpoints are publicly released.

Significance. The paper makes a substantial contribution as a resource: a reproducible, continuously updatable curation pipeline with component-level benchmarks, a large curated medical image-text corpus, and pretrained models all publicly released. The controlled CLIP experiment—matching architecture, initialization, hyperparameters, and training schedule to BMC-CLIP—is a well-designed end-to-end test of whether curation quality translates into downstream gains. The inclusion of 95% confidence intervals via paired bootstrap across benchmarks, the independent clinical dermatology evaluation on internal patient data, and the embedding-space analysis of distributional alignment with clinical images are all commendable. The component-level benchmark suite comparing against prior pipeline approaches (PMC-OA, MedICaT, GPT-4T zero-shot) adds engineering value and positions the framework for community refinement.

major comments (2)
  1. §2.4, paragraph on training details: The paper states that the head-to-head comparison with BMC-CLIP 'isolated the effect of dataset quality.' However, the two corpora differ in at least three load-bearing ways simultaneously: (1) medical image density (MedPMC is 95.3% medical vs. 19.7% for BIOMEDICA, yielding ~10.5M vs. ~4.7M medical pairs—a roughly 2.2× difference in medical training examples), (2) total pair count (11M vs. 24M), and (3) curation depth (multi-panel decomposition, joint caption alignment). The 7.1 pp AUC gain is attributed to 'high-fidelity curation,' but without an ablation that controls for the number of medical training pairs—e.g., training on a medical-filtered subset of BIOMEDICA without decomposition or alignment—the evidence supports the broad claim that curation helps but does not specifically validate that the distinctive pipeline stages (beyond filtering) are载
  2. §2.1, Caption separation and alignment: The paper states that samples where the number of generated subcaptions does not match the number of input subfigures are removed to 'prioritize pair fidelity.' The pipeline goes from 29M subfigures to 12.5M subfigure-subcaption pairs at this stage—a 57% reduction. This is a substantial filter that could introduce systematic selection bias (e.g., excluding figures with many panels or complex layouts, which may represent important clinical content). The paper does not quantify the mismatch rate, analyze what types of figures are dropped, or discuss the potential impact on dataset coverage. This information is needed to assess whether the resulting corpus is representative of the medical literature or biased toward simpler figure layouts.
minor comments (7)
  1. §2.1, Initial screening: The paper reports F1=93.2 for initial screening but does not report precision and recall separately in the main text. Given that this stage gates all downstream processing, reporting these would help readers assess the false-negative rate (medically relevant figures lost) and false-positive rate (wasted downstream computation).
  2. §4.1, Validation and model selection: The validation sets include synthetic annotations generated by the same GPT-4T procedure used for training data. While the test sets use manual labels, the use of synthetic labels in validation could bias model selection toward models that align with GPT-4T's labeling tendencies. A brief discussion of this potential bias would strengthen the methodology section.
  3. Fig. 2a: The BIOMEDICA sample size (432 images) is much smaller than the MedPMC sample (2,906 images). The confidence intervals on the BIOMEDICA composition estimates are correspondingly wider. It would be helpful to note this asymmetry or report CIs on the category proportions.
  4. §2.4, Downstream medical QA: The MMMU improvement of 1.9 pp has a 95% CI of [-4.1, 8.0], which includes zero. The text acknowledges this, but the abstract states the improvement without qualification ('improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks'). The abstract should note that the MMMU result is not statistically significant.
  5. Extended Data Table 2: The caption separation and alignment training set lists '1,664 (M)' from MedICaT, but §4.1 states '1,361 manually annotated samples from the MedICaT dataset.' Please reconcile these numbers.
  6. §4.4, Morphology-to-skin image retrieval: The set-based matching criterion for retrieval evaluation is described, but the specific threshold or matching rule (subset vs. exact match of concept sets) could be stated more precisely in the main text rather than requiring the reader to consult the cited references.
  7. The paper uses 'GPT-4' and 'GPT-4T' somewhat interchangeably in places (e.g., §4.1 multi-panel figure detection says 'GPT-4' while other sections say 'GPT-4T'). Standardizing the terminology would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee raises two major comments: (1) the CLIP comparison with BMC-CLIP does not isolate which aspects of curation drive the observed gains, and (2) the caption alignment filter drops 57% of subfigures without analysis of potential selection bias. Both points are well-taken. We will address them through a combination of new analysis and revised language in the manuscript.

read point-by-point responses

  1. Referee: §2.4: The head-to-head comparison with BMC-CLIP does not isolate the effect of curation quality because the corpora differ in medical image density, total pair count, and curation depth simultaneously. An ablation controlling for the number of medical training pairs is needed.

    Authors: The referee is correct that the three differences between MedPMC and BIOMEDICA (medical pair density, total pair count, and curation depth) are confounded in our current experimental design. We agree that the phrase 'isolated the effect of dataset quality' overstates what the comparison demonstrates. The experiment isolates the effect of the training corpus as a whole, not the individual contributions of filtering, decomposition, or alignment. We will revise the language in §2.4 and the Discussion to accurately characterize the comparison as isolating the effect of the training corpus while acknowledging that the corpora differ along multiple dimensions simultaneously. Regarding the proposed ablation—training on a medical-filtered subset of BIOMEDICA without decomposition or alignment—we agree this would be informative and will attempt it as a supplementary analysis. However, we note a practical constraint: BIOMEDICA does not provide per-image medical relevance labels, so constructing such a subset requires applying our own medical figure classifier to the BIOMEDICA corpus. This means the ablation would partially use MedPMC's own curation components, making it a cleaner test of 'filtering alone vs. filtering plus decomposition and alignment' rather than a fully independent test. We will be transparent about this limitation if the ablation is included. Regardless of whether the ablation is feasible within the revision timeline, we will soften the causal claim and explicitly enumerate the confounded factors. revision: partial

  2. Referee: §2.1: The caption separation and alignment stage drops 57% of subfigures (from 29M to 12.5M) when subcaption counts do not match subfigure counts. This could introduce systematic selection bias toward simpler layouts. The paper does not quantify the mismatch rate, analyze dropped figure types, or discuss impact on coverage.

    Authors: This is a fair and important point. We will add analysis to the revised manuscript. Specifically, we plan to: (1) quantify the mismatch rate as a function of panel count, showing how the drop-off concentrates among figures with many subpanels; (2) sample and manually categorize a subset of dropped figures to characterize what types of content are disproportionately excluded; and (3) add a discussion paragraph acknowledging the potential for selection bias toward simpler layouts and its implications for dataset coverage. We agree that the current manuscript does not adequately address this trade-off between fidelity and coverage. The revised text will explicitly note that the filter prioritizes pair-level fidelity at the cost of excluding complex multi-panel figures, which may underrepresent certain clinical content types. We will also discuss potential future approaches to recovering some of the dropped pairs (e.g., partial alignment or relaxed matching criteria) as future work. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; central claims evaluated on independent external benchmarks and manual test sets.

full rationale

The paper's central claims are not circular. The CLIP comparison (MedPMC-CLIP vs BMC-CLIP) uses 26 external public benchmarks and an independent YNHHS clinical dermatology cohort, with architecture and training protocol held fixed — the improvement is measured against data not used in training. The pipeline component benchmarks do use GPT-4T-generated synthetic labels for training data, and the validation sets also include synthetic annotations from the same procedure (e.g., initial screening validation is 9,781 synthetic examples with zero manual labels per Extended Data Table 2). This creates a partial loop where GPT-4T influences both training and model selection. However, the paper explicitly states: 'Importantly, the final reported performance of each curation component was assessed on held-out real instances with manual labels,' and the test sets per Extended Data Table 2 are composed of existing benchmark data (ImageCLEF, MedICaT, DocFigure) and newly manually annotated samples (L), not synthetic labels. The MLLM evaluation uses MMMU and OmniMedVQA, both external. No self-citation chain is load-bearing: BMC-CLIP (ref 31, Lozano et al.) and LLaVA-Med (ref 29, Li et al.) are external works by different author groups. The skeptic's concern about the controlled comparison not disentangling medical-image filtering from higher-fidelity curation stages is an attribution/mechanism gap, not a circularity issue. The reader's concern about GPT-4T grading its own distillates applies to validation, not to the final test results. Score 1 reflects the minor validation-set concern that does not propagate to the reported findings.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, particles, or forces. It is an engineering and data science contribution. The 'invented' elements are the pipeline models themselves, which are instantiations of existing architectures (PubMedBERT, ViT, YOLOv10, InternVL). The primary axioms are domain assumptions about the sufficiency of text for medical filtering and the reliability of GPT-4T for synthetic label generation.

free parameters (2)
  • Pipeline model hyperparameters = Various (e.g., lr=3e-5, batch=64)
    Standard hyperparameters for fine-tuning PubMedBERT, ViT, YOLOv10, InternVL-2.5-4B; not free parameters in the theoretical sense but fitted to the task.
  • CLIP training config = batch=8192, lr=1e-6
    Inherited from BMC-CLIP to isolate dataset variable; not tuned for MedPMC specifically.
axioms (4)
  • domain assumption Biomedical figure captions and inline reference text contain sufficient signal to distinguish medical from non-medical figures before image download.
    Underlies the initial screening stage (§2.1, Methods 4.1). Validated by F1=93.2 on held-out manual labels.
  • domain assumption A supervised 4B MLLM can jointly perform caption separation and subfigure-subcaption alignment more accurately than sequential rule-based or CLIP-based methods.
    Underlies the caption separation stage (§2.1, Methods 4.1). Validated by F1=81.4 vs PMC-OA F1=48.5.
  • ad hoc to paper GPT-4 Turbo generates sufficiently accurate synthetic labels for training pipeline components.
    Used to generate 99,321 training labels (Methods 4.1). No direct ablation on the effect of GPT-4T label noise on final pipeline output quality is provided.
  • domain assumption Public benchmark datasets are largely independent of the PMC-derived pretraining corpus.
    Stated in Contamination-aware evaluation design; provenance checked but image-level overlap cannot be fully excluded.

pith-pipeline@v1.1.0-glm · 29441 in / 2472 out tokens · 627027 ms · 2026-07-09T02:47:35.088331+00:00 · methodology

0 comments
read the original abstract

Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.

Figures

Figures reproduced from arXiv: 2607.07673 by Dain Kim, Gui Yang, Hoifung Poon, Hua Xu, Hyunjae Kim, Jaewoo Kang, Jonathan Marquez, Kevin W. Jin, Mengmeng Du, Pan Xiao, Qingyu Chen, Roy Jiang, Rui Shi, Serina S. Applebaum, Sheng Zhang, Tuo Guo, Xuguang Ai, Yan Hu, Yawen Wei, Yiming Kong, Younjoon Chung, Yuelei Fu, Yuexi Du, Yuntian Liu, Yuxuan Tian, Yu Yin, Zhen Chen, Zhiyuan Cao.

Figure 1
Figure 1. Figure 1: MedPMC framework for dataset curation and its application to multimodal model training. a, Overview of the proposed five-step framework for constructing a large-scale medical image-text dataset from PMC articles. This pipeline yields 11M medical image-text pairs with improved data quality. b, Model development pipeline. A CLIP-style vision-language model is first trained on the curated dataset via contrast… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of MedPMC dataset composition, modality distribution, and temporal growth. a, Distribution of image categories in MedPMC compared to BIOMEDICA, highlighting a substantial reduction in non-medical images (e.g., graphs and charts) and increased coverage of clinically relevant modalities. b, Distribution of radiology imaging modalities within the MedPMC dataset. c, The number of publications and extr… view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation results. a, Image classification performance across 11 medical specialties, where scores are averaged over 26 benchmarks; MedPMC-CLIP consistently outperforms prior models across accuracy, F1 score, and AUC. b, Comparison of MedPMC-CLIP and BMC-CLIP on image classification performance across medical specialties; MedPMC-CLIP outperformed BMC-CLIP in 10 of 11 specialties. c, Performance comparison… view at source ↗
Figure 4
Figure 4. Figure 4: Embedding-space comparison of external dermatology image sources with internal clinical dermatology images. a, Empirical cumulative distribution of nearest-neighbor cosine distances from internal clinical dermatology images to each external image source in the DINOv2 embedding space. b, Fraction of internal clinical dermatology images whose nearest neighbor was found in each external source. c–f, UMAP proj… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

106 extracted references · 106 canonical work pages · 7 internal anchors

  1. [1]

    N., Falcone, G

    Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical ai.Nat. medicine 28, 1773–1784 (2022)

    work page 2022

  2. [2]

    Moor, M.et al.Foundation models for generalist medical artificial intelligence.Nature616, 259–265 (2023)

    work page 2023

  3. [3]

    Cui, C.et al.Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review.Prog. Biomed. Eng.5, 022001 (2023)

    work page 2023

  4. [4]

    & Lungren, M

    Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine3, 136 (2020)

    work page 2020

  5. [5]

    He, Y .et al.Foundation model for advancing healthcare: challenges, opportunities and future directions.IEEE Rev. Biomed. Eng.18, 172–191 (2024)

    work page 2024

  6. [6]

    Rajendran, S.et al.Learning across diverse biomedical data modalities and cohorts: Challenges and opportunities for innovation.Patterns5(2024)

    work page 2024

  7. [7]

    image analysis105, 103621 (2025)

    Schouten, D.et al.Navigating the landscape of multimodal ai in medicine: a scoping review on technical challenges and clinical applications.Med. image analysis105, 103621 (2025)

    work page 2025

  8. [8]

    & Metaxas, D

    Zhang, S. & Metaxas, D. On the challenges and perspectives of foundation models for medical image analysis.Med. image analysis91, 102996 (2024)

    work page 2024

  9. [9]

    Acevedo, A.et al.A dataset of microscopic peripheral blood cell images for development of automatic recognition systems.Data brief30, 105474 (2020)

    work page 2020

  10. [10]

    Sq.rs–3 (2025)

    Huang, S.-C.et al.A systematic review and implementation guidelines of multimodal foundation models in medical imaging.Res. Sq.rs–3 (2025)

    work page 2025

  11. [11]

    & Hernandez-Boussard, T

    Koul, A., Duran, D. & Hernandez-Boussard, T. Synthetic data, synthetic trust: navigating data challenges in the digital revolution.The Lancet Digit. Heal.7(2025)

    work page 2025

  12. [12]

    & Panagiotakos, D

    Liu, F. & Panagiotakos, D. Real-world data: a brief review of the methods, applications, challenges and opportunities.BMC Med. Res. Methodol.22, 287 (2022)

    work page 2022

  13. [13]

    & Missier, P

    Mandreoli, F., Ferrari, D., Guidetti, V ., Motta, F. & Missier, P. Real-world data mining meets clinical practice: Research challenges and perspective.Front. big Data5, 1021621 (2022)

    work page 2022

  14. [14]

    InProceedings of the AAAI conference on artificial intelligence, vol

    Irvin, J.et al.Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. InProceedings of the AAAI conference on artificial intelligence, vol. 33, 590–597 (2019)

    work page 2019

  15. [15]

    Neural Inf

    Huang, S.-C.et al.Inspect: a multimodal dataset for patient outcome prediction of pulmonary embolisms.Adv. Neural Inf. Process. Syst.36, 17742–17772 (2023). 22/32

    work page 2023

  16. [16]

    Ai-readi: rethinking ai data collection, preparation and sharing in diabetes research and beyond.Nat

    AI-READI Project Managers & NIH Program Scientists. Ai-readi: rethinking ai data collection, preparation and sharing in diabetes research and beyond.Nat. Metab.6, 2210–2212 (2024)

    work page 2024

  17. [17]

    & V ogt, J

    Sokol, K., Fackler, J. & V ogt, J. E. Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap.npj Digit. Medicine8, 345 (2025)

    work page 2025

  18. [18]

    Jiang, R.et al.Are multimodal llms ready for clinical dermatology? a real-world evaluation in dermatology.arXiv preprint arXiv:2605.04098(2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  19. [19]

    Medicine1–9 (2026)

    Bedi, S.et al.Holistic evaluation of large language models for medical tasks with medhelm.Nat. Medicine1–9 (2026)

    work page 2026

  20. [20]

    D.et al.Addressing barriers in fair data practices for biomedical data.Sci

    Hughes, L. D.et al.Addressing barriers in fair data practices for biomedical data.Sci. Data10, 98 (2023)

    work page 2023

  21. [21]

    & Kou, T

    Huang, X., Zhu, S., Lv, Z., Zhou, Q. & Kou, T. The open sharing operation mechanism of health data in the digital healthcare era: A study based on grounded theory and interpretative structural modeling method.Digit. Heal.11, 20552076251353694 (2025)

    work page 2025

  22. [22]

    Cao, Z.et al.The development landscape of large language models for biomedical applications. Annu. review biomedical data science8(2025)

    work page 2025

  23. [23]

    Zhang, X.et al.Pmc-vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    Chen, Z.et al.From compound figures to composite understanding: Developing a multi-modal llm from biomedical literature with medical multiple-image benchmarking and validation.arXiv preprint arXiv:2511.22232(2025)

    work page arXiv 2025

  25. [25]

    About pmc

    National Library of Medicine. About pmc. https://pmc.ncbi.nlm.nih.gov/about/intro/. Accessed: 2026-05-31

    work page 2026

  26. [26]

    InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 525–536 (Springer, 2023)

    Lin, W.et al.Pmc-clip: Contrastive language-image pre-training using biomedical documents. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 525–536 (Springer, 2023)

    work page 2023

  27. [27]

    & De Melo, G

    Eslami, S., Meinel, C. & De Melo, G. Pubmedclip: How much does clip benefit visual question answering in the medical domain? InFindings of the Association for Computational Linguistics: EACL 2023, 1181–1193 (2023)

    work page 2023

  28. [28]

    & Sun, J

    Wang, Z., Wu, Z., Agarwal, D. & Sun, J. Medclip: Contrastive learning from unpaired medical images and text. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 3876–3887 (2022)

    work page 2022

  29. [29]

    Li, C.et al.Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Adv. Neural Inf. Process. Syst.36, 28541–28564 (2023)

    work page 2023

  30. [30]

    Qin, Z.et al.V olmo: Versatile and open large models for ophthalmology.arXiv preprint arXiv:2603.23953(2026)

    work page arXiv 2026

  31. [31]

    InProceedings of the Computer Vision and Pattern Recognition Conference, 19724–19735 (2025)

    Lozano, A.et al.Biomedica: An open biomedical image-caption archive, dataset, and vision- language models derived from scientific literature. InProceedings of the Computer Vision and Pattern Recognition Conference, 19724–19735 (2025)

    work page 2025

  32. [32]

    In International conference on machine learning, 8748–8763 (PmLR, 2021)

    Radford, A.et al.Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763 (PmLR, 2021). 23/32

    work page 2021

  33. [33]

    & Katlam, A

    Karthik, K., Kamath, S., Supreetha, R. & Katlam, A. Content-based medical retrieval systems with evidence-based diagnosis for enhanced clinical decision support.Expert. Syst. with Appl.272, 126678 (2025)

    work page 2025

  34. [34]

    & Liang, X

    Zhou, X., Han, X., Li, H., Wang, J. & Liang, X. Cross-domain image retrieval: methods and applications.Int. J. Multimed. Inf. Retr.11, 199–218 (2022)

    work page 2022

  35. [35]

    M.et al.Multimodal generative ai for medical image interpretation.Nature639, 888–896 (2025)

    Rao, V . M.et al.Multimodal generative ai for medical image interpretation.Nature639, 888–896 (2025)

    work page 2025

  36. [36]

    J.et al.Medical image processing, analysis and visualization in clinical research

    McAuliffe, M. J.et al.Medical image processing, analysis and visualization in clinical research. InProceedings 14th IEEE symposium on computer-based medical systems. CBMS 2001, 381–386 (IEEE, 2001)

    work page 2001

  37. [37]

    Artificial intelligence in clinical medicine: challenges across diagnostic imaging, clinical decision support, surgery, pathology, and drug discovery.Clin

    Ogut, E. Artificial intelligence in clinical medicine: challenges across diagnostic imaging, clinical decision support, surgery, pathology, and drug discovery.Clin. practice15, 169 (2025)

    work page 2025

  38. [38]

    & Müller, H

    García Seco de Herrera, A., Schaer, R., Bromuri, S. & Müller, H. Overview of the ImageCLEF 2016 medical task. InWorking Notes of CLEF 2016 (Cross Language Evaluation Forum)(2016)

    work page 2016

  39. [39]

    In Findings of the Association for Computational Linguistics: EMNLP 2020, 2112–2120 (2020)

    Subramanian, S.et al.Medicat: A dataset of medical images, captions, and textual references. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2112–2120 (2020)

    work page 2020

  40. [40]

    & Jawahar, C

    Jobin, K., Mondal, A. & Jawahar, C. Docfigure: A dataset for scientific document figure classification. In2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), vol. 1, 74–79 (IEEE, 2019)

    work page 2019

  41. [41]

    I.et al.A multimodal data processing pipeline for mimic-iv dataset.arXiv preprint arXiv:2601.11606(2026)

    Adiba, F. I.et al.A multimodal data processing pipeline for mimic-iv dataset.arXiv preprint arXiv:2601.11606(2026)

    work page arXiv 2026

  42. [42]

    Wornow, M.et al.The shaky foundations of large language models and foundation models for electronic health records.npj digital medicine6, 135 (2023)

    work page 2023

  43. [43]

    ACM Transactions on Comput

    Gu, Y .et al.Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Comput. for Healthc. (HEALTH)3, 1–23 (2021)

    work page 2021

  44. [44]

    InInternational Conference on Learning Representations(2021)

    Dosovitskiy, A.et al.An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations(2021)

    work page 2021

  45. [45]

    InProceedings of the 2024 SIAM International Conference on Data Mining (SDM), 118–126 (SIAM, 2024)

    Meng, S.et al.Yolo-ocr: End-to-end compound figure separation and label recognition of images in scientific publications. InProceedings of the 2024 SIAM International Conference on Data Mining (SDM), 118–126 (SIAM, 2024)

    work page 2024

  46. [46]

    GPT-4o System Card

    Wang, A.et al.Yolov10: Real-time end-to-end object detection.Adv. neural information processing systems37, 107984–108011 (2024). 47.Hurst, A.et al.Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024). 48.Chen, Z.et al.Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    & Lee, Y

    Liu, H., Li, C., Wu, Q. & Lee, Y . J. Visual instruction tuning.Adv. neural information processing systems36, 34892–34916 (2023)

    work page 2023

  48. [48]

    In European Conference on Computer Vision, 304–323 (Springer, 2024)

    McKinzie, B.et al.Mm1: methods, analysis and insights from multimodal llm pre-training. In European Conference on Computer Vision, 304–323 (Springer, 2024). 24/32

    work page 2024

  49. [49]

    M.et al.A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.Nat

    Zambrano Chaves, J. M.et al.A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.Nat. Commun.16, 3108 (2025)

    work page 2025

  50. [50]

    & Atkins, M

    Sadeghi, M., Chilana, P., Yap, J., Tschandl, P. & Atkins, M. S. Using content-based image retrieval of dermoscopic images for interpretation and education: A pilot study.Ski. Res. Technol.26, 503–512 (2020)

    work page 2020

  51. [51]

    Gassner, M.et al.Saliency-enhanced content-based image retrieval for diagnosis support in derma- tology consultation: Reader study.JMIR dermatology6, e42129 (2023)

    work page 2023

  52. [52]

    & Geissbuhler, A

    Müller, H., Michoux, N., Bandon, D. & Geissbuhler, A. A review of content-based image retrieval systems in medical applications—clinical benefits and future directions.Int. journal medical informatics73, 1–23 (2004)

    work page 2004

  53. [53]

    R., Antani, S., Deserno, T

    Long, L. R., Antani, S., Deserno, T. M. & Thoma, G. R. Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions.Int. J. Healthc. Inf. Syst. Informatics (IJHISI)4, 1–16 (2009)

    work page 2009

  54. [54]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1820–1828 (2021)

    Groh, M.et al.Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1820–1828 (2021)

    work page 2021

  55. [55]

    Open7, e2446615 (2024)

    Ward, A.et al.Creating an empirical dermatology dataset through crowdsourcing with web search advertisements.JAMA Netw. Open7, e2446615 (2024). 58.DermNet New Zealand Trust. Dermnet. Accessed: 2026-06-17

    work page 2024

  56. [56]

    & Friedrich, C

    Pelka, O., Koitka, S., Rückert, J., Nensa, F. & Friedrich, C. M. Radiology objects in context (roco): A multimodal image dataset. InInternational Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, 180–189 (Springer, 2018)

    work page 2018

  57. [57]

    Zhang, S.et al.A multimodal biomedical foundation model trained from fifteen million image–text pairs.NEJM AI2, AIoa2400640 (2025)

    work page 2025

  58. [58]

    S., Ashkezari, S., Dolatabadi, E

    Baghbanzadeh, N., Islam, M. S., Ashkezari, S., Dolatabadi, E. & Afkanpour, A. Open-pmc-18m: A high-fidelity large scale medical dataset for multimodal representation learning.arXiv preprint arXiv:2506.02738(2025)

    work page arXiv 2025

  59. [59]

    InMachine learning for health (ML4H), 353–367 (PMLR, 2023)

    Moor, M.et al.Med-flamingo: a multimodal medical few-shot learner. InMachine learning for health (ML4H), 353–367 (PMLR, 2023)

    work page 2023

  60. [60]

    medicine30, 3129–3141 (2024)

    Zhang, K.et al.A generalist vision–language foundation model for diverse biomedical tasks.Nat. medicine30, 3129–3141 (2024)

    work page 2024

  61. [61]

    MedGemma Technical Report

    Lin, T.et al.Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. InInternational Conference on Machine Learning, 37975–37995 (PMLR, 2025). 65.Sellergren, A.et al.Medgemma technical report.arXiv preprint arXiv:2507.05201(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  62. [62]

    84, 1388–1395 (2024)

    Wang, Z.et al.Nci cancer research data commons: resources to share key cancer data.Cancer Res. 84, 1388–1395 (2024)

    work page 2024

  63. [63]

    In2024 IEEE international conference on bioinformatics and biomedicine (BIBM), 3138–3143 (IEEE, 2024)

    Dai, D.et al.Pa-llava: A large language-vision assistant for human pathology image understanding. In2024 IEEE international conference on bioinformatics and biomedicine (BIBM), 3138–3143 (IEEE, 2024). 25/32

    work page 2024

  64. [64]

    C., Massiris, M

    Nievas Offidani, M., Roffet, F., González Galtier, M. C., Massiris, M. & Delrieux, C. An open-source clinical case dataset for medical image classification and multimodal ai applications.Data10, 123 (2025)

    work page 2025

  65. [65]

    Wu, D.et al.Gestaltmml: Enhancing rare genetic disease diagnosis through multimodal machine learning combining facial images and clinical texts.ArXivarXiv–2312 (2024)

    work page 2024

  66. [66]

    InProceedings of the 31st ACM international conference on multimedia, 547–556 (2023)

    Yuan, Z.et al.Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training. InProceedings of the 31st ACM international conference on multimedia, 547–556 (2023)

    work page 2023

  67. [67]

    & Wynn, R

    Nissen, T. & Wynn, R. The clinical case report: a review of its merits and limitations.BMC research notes7, 264 (2014)

    work page 2014

  68. [68]

    InThe Twelfth International Conference on Learning Representations(2024)

    Xu, H.et al.Demystifying clip data. InThe Twelfth International Conference on Learning Representations(2024)

    work page 2024

  69. [69]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7346– 7370 (2024)

    Chen, J.et al.Towards injecting medical visual knowledge into multimodal llms at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7346– 7370 (2024)

    work page 2024

  70. [70]

    Bedi, S.et al.Testing and evaluation of health care applications of large language models: a systematic review.Jama333, 319–328 (2025)

    work page 2025

  71. [71]

    communications16, 3280 (2025)

    Chen, Q.et al.Benchmarking large language models for biomedical natural language processing applications and recommendations.Nat. communications16, 3280 (2025)

    work page 2025

  72. [72]

    & Kostakos, P

    Sheikhi, S., Loven, L. & Kostakos, P. Beyond the leaderboard: A survey of the science of evaluation, benchmarking, and methodologies for large language models.IEEE Access(2026)

    work page 2026

  73. [73]

    Commun.(2026)

    Li, A.et al.Memorization in large language models in medicine prevalence characteristics and implications.Nat. Commun.(2026)

    work page 2026

  74. [74]

    Chang, Y .et al.A survey on evaluation of large language models.ACM transactions on intelligent systems technology15, 1–45 (2024)

    work page 2024

  75. [75]

    Liu, X.et al.A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.The lancet digital health1, e271–e297 (2019)

    work page 2019

  76. [76]

    PyTorch Image Models, 2019

    Wightman, R. Pytorch image models. https://github.com/rwightman/pytorch-image-models, DOI: 10.5281/zenodo.4414861 (2019)

    work page doi:10.5281/zenodo.4414861 2019

  77. [77]

    & Hutter, F

    Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. InInternational Conference on Learning Representations(2017)

    work page 2017

  78. [78]

    Yu, J.et al.Coca: Contrastive captioners are image-text foundation models.Transactions on Mach. Learn. Res.(2022)

    work page 2022

  79. [79]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2818–2829 (2023)

    Cherti, M.et al.Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2818–2829 (2023)

    work page 2023

  80. [80]

    Neural Inf

    Lozano, A.et al.Micro-bench: A microscopy benchmark for vision-language understanding.Adv. Neural Inf. Process. Syst.37, 30670–30685 (2024)

    work page 2024

Showing first 80 references.