REVIEW 5 major objections 5 minor 3 references
Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM)
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A single self-supervised vision model, pretrained on 3.3 million unlabelled images across seven modalities, reports a mean AUROC of 0.935 and matches or beats specialty-specific medical imaging models on most tasks tested.
desk verdict Broad medical SSL benchmark that deserves a referee, but missing pretraining provenance and internal number inconsistencies make the headline AUROC claims uncheckable as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a memory-augmented joint-embedding teacher-student self-supervised learning loop. A student ViT and a teacher ViT encode multiple augmented views of each image; only the student is updated by gradients, while the teacher is an exponential moving average of the student. A dynamic memory module, a non-differentiable FIFO store of 65,536 compact representation vectors, holds recent embeddings from all modalities. The student's views are compared with stored representations, and the model enforces that similarity distributions over memory blocks agree across views. Balanced modality- and specialty-aware batch sampling prevents chest X-rays or other large modalities from dominating training. This combination is what the paper credits with stabilising joint training, preventing mode collapse and catastrophic forgetting, and letting a single encoder learn transferable features across seven modalities without labels.
What would settle it
Run an exact-duplicate and perceptual near-duplicate search between the 3.3 million pretraining images and the test images of the public evaluation sets (RSNA, SIIM, TBX11K, IQ-OTH/NCCD, SARS-COV-2, BUSI, BreakHis, APTOS2019, IDRiD, HAM10000, Dermnet, OCTID, and OCTDL). If even a small number of evaluation images, or near-duplicates of them, appear in the pretraining pool, the central comparison is no longer evidence for the model's generalization.
Extended reading notes
Core claim
The paper's central claim is that a self-supervised, vision-only foundation model can be genuinely multimodal and multi-disease without the usual label burden. MerMED-FM is a ViT-B encoder pretrained on 3.3 million unlabelled images spanning CT, chest X-ray, ultrasound, histopathology patches, colour fundus photography, OCT, and dermoscopy. After fine-tuning with a small task-specific head, it reports the best mean AUROC among the compared models, 0.935, above BiomedCLIP's 0.919 and DINO's 0.933, with per-modality AUROCs of 0.988 (OCT), 0.982 (pathology), 0.951 (ultrasound), 0.943 (CT), 0.931 (skin), 0.894 (CFP), and 0.858 (CXR). The paper's authors interpret this as showing that a shared visual encoder trained with a memory-augmented self-supervised objective can transfer across specialties and match or beat models built for one modality, while explicitly noting that it did not surpass the pathology or dermatology specialists and has not yet been tested on true same-patient multimodal reasoning.
Load-bearing premise
The load-bearing assumption is that the 3.3 million pretraining images, taken from publicly available unlabelled datasets named only in Supplementary Table 6, are disjoint from the 25 public evaluation datasets; if any evaluation images were also in pretraining, the reported AUROC improvements would be inflated by leakage.
Editorial extensions
If this is right
- If the central claim holds, a hospital could deploy one vision encoder for CT, CXR, ultrasound, pathology, fundus photography, OCT, and dermoscopy instead of a separate specialized model per department, simplifying maintenance and oversight.
- The self-supervised pretraining route would lower the annotation barrier: with 10–50% of the fine-tuning data, MerMED-FM retains most of its full-data AUROC, so new diseases and modalities could be added more quickly in low-resource settings.
- Because MerMED-FM needs no text prompts, it can be applied to raw images at acquisition time, before a report exists, which suits triage and screening workflows.
- The claim is not that one model beats every specialist: the paper reports parity with but not superiority over the pathology and dermatology specialists, so the all-in-one benefit is bought with some specialty-specific trade-offs.
Reading between the lines
- The paper's 'multimodal' claim is about a shared encoder, not integrated multimodal reasoning; it explicitly has not tested pairing, say, a patient's CT and biopsy in one decision. A natural next experiment is to feed same-patient images from two modalities and check whether diagnostic accuracy improves over either modality alone.
- A testable consequence of the memory module is continual learning: because stored representations are FIFO and sampling is balanced, the same training loop could in principle absorb a new imaging modality later without full retraining; the paper does not demonstrate this, but the mechanism invites it.
- If a leakage screen comes back clean, the result would support a stronger inference than the paper draws: text supervision and paired image-text data may be unnecessary for strong medical imaging encoders, with broad unlabelled modality coverage plus self-supervision doing the work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents MerMED-FM, a vision-only self-supervised foundation model trained on 3.3 million unlabelled medical images spanning CT, CXR, ultrasound, histopathology, color fundus photography, OCT, and dermatology, using a teacher–student ViT architecture augmented with a FIFO memory module. The authors report downstream fine-tuning results across 25 public and seven local datasets, comparing against DINO, BiomedCLIP, and six single-modality foundation models, with a headline mean AUROC of 0.935 versus 0.933 for DINO and 0.919 for BiomedCLIP. They also report data-efficiency experiments at 10%, 30%, and 50% of the fine-tuning data and claim general equivalence or superiority to specialty models on several tasks.
Significance. If the reported benchmark results and pretraining hygiene are confirmed, this is a substantial contribution: a single vision-only SSL encoder that matches or exceeds several specialty foundation models across seven imaging modalities would be practically useful, and the paper is admirably candid about the tasks where the model does not win, such as pathology and dermatology. The authors also provide a relatively large pretraining corpus, a memory-augmented SSL recipe, and a downstream evaluation protocol with five independent runs, paired t-tests, confidence intervals, and effect sizes. However, the manuscript as submitted does not enumerate the pretraining corpus or provide an overlap audit, and several reported numbers are mutually inconsistent; these issues currently block verification of the central claim. The missing Supplementary Table 6 is not a cosmetic omission because the reported margin over DINO is only 0.002 AUROC.
major comments (5)
- [Methods — Pre-Training Dataset / Supplementary Table 6] The pretraining corpus is not enumerated. The Methods state that the 3.3 million images come from 'publicly available unlabelled datasets' and defer the list to Supplementary Table 6, but that table is absent from the submitted manuscript. Because the 25 evaluation datasets include widely used public collections such as RSNA, SIIM, APTOS2019, IDRiD, HAM10000, and BUSI, and because several of these are plausible components of the stated 401,059 dermoscopic and 333,700 fundus pretraining images, the manuscript cannot rule out that a fraction of evaluation images appeared in pretraining. Since the reported overall margin over DINO is 0.002 AUROC, even a small overlap could account for the headline difference. This is a load-bearing transparency issue: please provide the full corpus enumeration, a deduplication procedure, and an overlap analysis against every evaluation dataset.
- [Abstract / Results / Table 1] Several headline numbers disagree across the manuscript. The Abstract reports modality AUROCs of 0.943 (CT) and 0.858 (CXR), whereas Table 1 gives a CT public AUROC of 0.990 and a CXR public AUROC of 0.908; Table 1 also reports histopathology 1.000 while the main text reports a mean AUROC of 0.999. In addition, the Results state that MerMED-FM achieved a mean AUROC of 0.935 overall, but no table presents this overall mean or its confidence interval. Please reconcile these values and state exactly which experiment each number comes from.
- [Results — Ocular disease diagnosis (OCTDL)] The OCTDL paragraph is internally inconsistent. The text says MerMED-FM 'achieved an AUROC of 0.911 (CI: 0.990-0.992)' and 'mean AUROC of 0.988' for OCT, but Table 2 lists the same OCTDL entry as 0.991 with CI 0.990-0.992. The printed 0.911 is incompatible with its own confidence interval and with Table 2. The sentence also appears grammatically incomplete ('where achieved an AUROC'). Please correct the value and recheck the surrounding OCTDL CIs and p-values.
- [Results — CT lung carcinoma / Table 3] The claimed statistical significance for the IQ-OTHNCCD comparison with BiomedCLIP is not supported by Table 3. The text states that MerMED-FM outperformed BiomedCLIP 'by 0.93% (p<0.01, T = 0.94, Cohen's d = 0.951)', but Table 3 reports for the same comparison a T statistic of 0.94 and p = 7.46e-01, which is not p<0.01. The same Table 3 row is garbled ('0.939 0.93 0.94 7.46E-01 0.951'), and several other rows contain AUROC-like numbers where mean differences are expected. Please repair Table 3 and ensure that every p-value quoted in the text is the one reported in the table.
- [Methods — Model Architecture (memory module)] The memory module is presented as a key contribution, but the only support for its benefit is the sentence 'The memory size was fixed at K = 65536, as determined through ablation studies.' No ablation study, table, or comparison with and without the memory module is reported anywhere in the manuscript. Since the memory size K, block size Nb, and temperature schedules are design choices, the reader cannot verify that these choices, rather than pretraining data composition or fine-tuning protocol, drive the reported results. Please include the ablation evidence or temper the claim accordingly.
minor comments (5)
- [Throughout] Please standardize the baseline name ('Dino' vs 'DINO') and specify the exact architecture and checkpoint used for each baseline, including patch size and pretraining dataset, since the comparison is load-bearing and the current text refers only to 'Dino' via a general reference.
- [Throughout] There are numerous typographical slips, including 'its to leading', 'with a AUROC', 'carder diagnosis', 'Supplemantary', 'carcioma', and 'both both'. Please run a careful proofreading pass.
- [Figure 1] The normalization formula is malformed: it reads '3𝑥−𝑚𝑖𝑥(𝑥)8' and '0.2+ ... 0.8max...' with unbalanced parentheses. Please rewrite it in standard mathematical notation.
- [Data Availability] The statement 'Additional data may reasonably be requested from the corresponding author' is insufficient for reproducibility; please specify whether the pretraining corpus list, fine-tuning data splits, random seeds, and model weights will be released.
- [Conclusion / Discussion] The Conclusion states that MerMED-FM 'outperforms existing single-modality and multi-specialty models', which is stronger than the Discussion's own limitations, where the authors concede that the model did not outperform UNI in pathology and PanDERM in dermatology. Please qualify the claim to match the reported results.
Circularity Check
No derivation-level circularity: MerMED-FM's claims rest on external fine-tuning benchmarks, not on self-referential definitions or fitted inputs.
full rationale
The paper's central claim is an empirical benchmark result: MerMED-FM is pretrained by self-supervised learning on unlabelled images, then fine-tuned separately on each downstream training split and evaluated on held-out test portions of 25 public datasets plus local hospital datasets. The statement 'MerMED-FM achieved the highest overall performance, with a mean AUROC of 0.935, outperforming BiomedCLIP (0.919) and Dino (0.933)' is a measured comparison, not a quantity derived from the model's own definitions. The memory-module hyperparameters (K = 65536, block size 16384) are presented as ablation-derived architecture choices, not as parameters fitted to the evaluation labels, and the fine-tuning protocol is applied identically to comparator models. No equation defines X in terms of Y and then presents Y as independent; no fitted value is renamed as a prediction; no uniqueness theorem or ansatz is imported from the authors' prior work to force the choice of architecture. The one substantive caveat is provenance: Methods states that pretraining used 'publicly available unlabelled datasets' and defers the specific list to Supplementary Table 6, which is not present in the provided text, and no overlap analysis against the evaluation datasets is described. If evaluation images were included in pretraining, the reported AUROC margins could be inflated; but this is a data-contamination and transparency risk that cannot be established from the manuscript text and is not a circularity in the derivation chain. Under the hard requirement to quote a specific reduction, no circular step can be identified.
Assumptions & free parameters
free parameters (5)
- Memory size K =
65536
- Memory block size Nb =
16384
- Teacher temperature schedule =
0.04 to 0.07 over first 30 epochs
- Student temperature =
0.1
- Weight decay schedule =
0.04 to 0.4 cosine
assumptions (5)
- domain assumption A shared ViT-B encoder can embed seven heterogeneous modalities into one latent space without modality-specific modules or text supervision.
- domain assumption Two-dimensional CT and US slices are sufficient for diagnostic classification; volumetric context is not required.
- ad hoc to paper The FIFO memory queue with stochastic partition improves generalization and prevents catastrophic forgetting.
- domain assumption Public benchmark labels and local hospital labels are correct and patient-independent from pretraining data.
- domain assumption Teacher-student EMA stabilization transfers from natural-image SSL to medical images.
invented entities (1)
-
Memory module (non-differentiable FIFO representation store)
Cite this review
Pith. "Pith review of Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM)." pith.science (2026). https://pith.science/paper/P4YISSVV
@misc{pith2026250700185,
author = {Pith},
title = {Pith review of: Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM)},
year = {2026},
howpublished = {\url{https://pith.science/paper/P4YISSVV}},
note = {Machine review of arXiv:2507.00185}
}
read the original abstract
Current artificial intelligence models for medical imaging are predominantly single modality and single disease. Attempts to create multimodal and multi-disease models have resulted in inconsistent clinical accuracy. Furthermore, training these models typically requires large, labour-intensive, well-labelled datasets. We developed MerMED-FM, a state-of-the-art multimodal, multi-specialty foundation model trained using self-supervised learning and a memory module. MerMED-FM was trained on 3.3 million medical images from over ten specialties and seven modalities, including computed tomography (CT), chest X-rays (CXR), ultrasound (US), pathology patches, color fundus photography (CFP), optical coherence tomography (OCT) and dermatology images. MerMED-FM was evaluated across multiple diseases and compared against existing foundational models. Strong performance was achieved across all modalities, with AUROCs of 0.988 (OCT); 0.982 (pathology); 0.951 (US); 0.943 (CT); 0.931 (skin); 0.894 (CFP); 0.858 (CXR). MerMED-FM has the potential to be a highly adaptable, versatile, cross-specialty foundation model that enables robust medical imaging interpretation across diverse medical disciplines.
Figures
Reference graph
Works this paper leans on
-
[1]
These datasets feature a myriad of diseases that are widely recognized for training or benchmarking domain-specific disease diagnosis were used for evaluation. Evaluation was conducted on annotated datasets with diabetic retinopathy (APTOS2019, IDRiD and MESSIDOR2 datasets), glaucoma (Glaucoma-Fundus and PAPILA datasets) and other retinal diseases (CRFO-v...
arXiv 2024
-
[8]
with large effect sizes of Cohen’s d 6.04, 6.31 and 6.61 respectively.8,22,23 MerMED-FM (AUROC = 0.937, CI: 0.932-0.942) also outperformed RETFound (AUROC = 0.849, CI: 0.835-0.863) in identifying AMD on CFP by a statistically significant difference of 10.7% (p<0.01, T = 14.1) and great effect of Cohen’s d 10.6.8 On the other hand, MerMED-FM (AUROC = 0.816...
work page 2020
-
[23]
Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore. *These authors contributed equally to this work as joint first authors †Corresponding author Corresponding Author: Assoc Prof. Daniel Ting Shu Wei Director, AI Office, Singapore Health Service Singapore National Eye Centre, Singapore Eye Research Institute, Singap...
work page 2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.