REVIEW 2 major objections 4 minor 1 cited by
Foundation Models in Radiology: What, How, When, Why and Why Not
T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This review argues that radiology needs a shared foundation-model vocabulary, and supplies one together with training and evaluation pathways.
desk verdict A competent and useful review of radiology foundation models, but the 'standardized terminology' claim is undercut by an inconsistent definition of what counts as a foundation model. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the taxonomy itself: the four-property definition of a foundation model — large-scale architecture and data, multimodality, self-supervision, and emergent abilities — inside a what, how, when, why, and why-not structure. This framework does the work of turning a scattered literature into a shared checklist: what counts as a foundation model, how training data and pre-training objectives determine capabilities, when datasets are sufficient, and why caution is needed. The same taxonomy is used to derive concrete data requirements (3D/4D and longitudinal multi-site datasets with demographics) and evaluation categories (coarse- and fine-grained tasks, visual question answering, generative similarity, human and model-based assessment, and subgroup bias analysis).
What would settle it
A careful empirical study that trains radiology foundation models at increasing scale on the same multi-site data and finds that diagnostic accuracy plateaus or drops beyond a modest model size, while smaller task-specific models match them, would falsify the scaling-law premise the review relies on, although the terminology it proposes could still stand.
Extended reading notes
Core claim
The review's central claim is that existing and future radiology foundation models can be understood through a single descriptive framework, rather than as a list of isolated systems. It identifies four defining characteristics — large-scale architectures and data, multimodal data integration, self-supervised training, and emergent abilities — and describes the building blocks that produce them: modality-specific encoders, fusion modules, and multimodal decoders, trained generatively or contrastively. It then argues that pre-trained models are adapted to clinical tasks by zero-shot inference, linear probing, fine-tuning, instruction tuning, or reinforcement learning from human feedback, and that evaluation must cover discriminative accuracy, generative similarity, human ratings, and bias. The paper's conclusion is that radiology-specific foundation models are feasible and potentially valuable, provided training data are large, multi-site, and representative, and deployment is governed by explicit safeguards.
Load-bearing premise
The review's load-bearing premise is that the scaling laws and self-supervised methods that work on web-scale general data also transfer to radiology, where datasets are smaller, labels are expensive, and imaging varies across sites and machines.
Editorial extensions
If this is right
- Papers and regulators gain a common set of terms, so different radiology foundation models can be compared on the same axes: scale, modality coverage, training paradigm, adaptability, and evaluation.
- Training radiology foundation models should prioritize large multi-site datasets spanning 3D/4D imaging, ultrasound, demographics, and longitudinal follow-ups, since current resources mostly cover 2D chest imaging.
- Evaluation should combine automatic discriminative benchmarks, generative content-similarity measures, human radiologist review or interactive demos, and subgroup and bias analysis before clinical deployment.
- Responsible deployment requires safeguards against hallucinated findings, automation bias and overreliance, anthropomorphism of model outputs, and cost-driven centralization of AI development.
- Future research the review points to includes efficient architectures, privacy-preserving training such as federated learning and differential privacy, continual learning, and monitoring for data drift after deployment.
Reading between the lines
- If the scaling-law assumption carries over to radiology, a natural next step would be compute-optimal training studies on radiology-specific data, since the review stops short of quantifying how much data or compute a target capability requires.
- The terminology could serve as a shared language for regulatory bodies and payers, even though the review itself does not engage specific approval or reimbursement pathways.
- A testable extension would be a benchmark suite that scores a radiology foundation model on all five capability categories and all three risk categories at once, making the review's separate checklists operational.
- The claim that emergent abilities are clinically valuable depends on their reliability; a targeted evaluation of emergent zero-shot diagnostic capabilities across rare diseases would test that dependency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper is a narrative review of foundation models (FMs) in radiology. It proposes a framework of four defining properties (large scale, multimodality, self-supervision, emergence), surveys pre-training and adaptation methods (generative and contrastive pre-training, zero-shot inference, fine-tuning, instruction tuning, RLHF), catalogs public datasets and their limitations, lists clinical capabilities and evaluation strategies, and discusses risks (hallucination, bias, automation bias, environmental cost) and future directions. The stated aim is to standardize terminology and align technical development with clinical deployment.
Significance. The review fills a gap for a consolidated, clinically oriented overview of FMs in radiology. It is well referenced, organizes a large literature into a coherent structure, and includes useful figures summarizing training and adaptation pathways. Its strengths include the emphasis on dataset requirements, evaluation benchmarks, and responsible deployment, as well as the broad coverage of both technical and ethical considerations. If the terminology were clarified, it could serve as an accessible entry point for radiologists and machine learning researchers. However, as currently written, the definitional inconsistency prevents the paper from fully achieving its central claim of a standardized terminology.
major comments (2)
- [Section 2 vs. Sections 3 and 5] The paper's central claim of establishing a standardized terminology is not supported by the definitional structure. Section 2 introduces four properties as characteristics of FMs, including multimodality and emergent abilities. Yet Section 3.1 presents masked autoencoders (e.g., Medical MAE) and Section 3.2 presents SimCLR and BYOL as FM pre-training methods, and Section 5 explicitly refers to Medical SAM as a foundation model—all of which are single-modality models with no demonstrated emergent capabilities beyond their training objectives. If the four properties are intended as necessary, these models are excluded; if they are optional, the definition does not provide a basis for distinguishing FMs from conventional self-supervised models. The manuscript should clarify which properties are necessary, sufficient, or merely typical, or revise the terminology to be consistent with the models actually discussed.
- [Section 2, emergent abilities] The definition of emergent abilities cites Wei et al. (ref. 16) as an established phenomenon, but Schaeffer et al. (2023) have shown that apparent emergence in large language models can be an artifact of the evaluation metric used. Since emergence is one of the four defining properties in the paper's framework, presenting it without this caveat is a contested choice and weakens the standardization claim. The authors should either acknowledge the debate and justify their definitional stance or reframe emergence as a contested/optional property.
minor comments (4)
- [Section 6] The heading 'Human-Computer Reaction with FMs' should be 'Human-Computer Interaction with FMs'; 'Reaction' appears to be a typo.
- [Section 4.1, datasets] The statement that 'Recently released datasets (63) consist of around 20,000 CXRs' appears inconsistent with the cited reference (CANDID-PTX), which is a small curated collection. Please verify the dataset and the number, or correct the reference to an appropriate large dataset.
- [Section 2, scaling laws] The sentence 'Model performance follows power laws, consistently improving as model and data size increase' states a general-domain empirical observation as a universal fact; the authors should qualify this as an observation from NLP/general vision that may not transfer to radiology, where data sizes are smaller and distribution shifts are more severe.
- [Section 3.2] The description of BYOL as bringing 'the representations of the two views closer in the representation space' is correct but could be more precise by noting that BYOL uses a predictor network and stop-gradient to avoid collapse; a brief clarification would help readers unfamiliar with the method.
Circularity Check
No circularity: this is a review paper with descriptive claims and external citations; no prediction or derivation reduces to its own inputs.
full rationale
The paper is a narrative review, not a derivation. Its central claim is to propose standardized terminology for foundation models in radiology and to outline training, evaluation, and deployment considerations. Every substantive assertion is supported by external references or by descriptive examples; no quantity is fit from data and then used as a prediction, no equation is defined in terms of its own output, and no load-bearing argument depends on a self-citation. The four listed FM properties (large scale, multimodality, self-supervision, emergent abilities) are presented as a framework, not as an empirically derived result. The paper does not compute or predict any outcome, so there is no fitted input being renamed as a prediction. Self-citations, such as CheXagent and RaLEs, appear only as examples of existing work and are not used to justify the review's core claims in a way that would make those claims depend circularly on the authors' prior results. The skeptical observation that the proposed terminology may not be fully operational is a coherence or precision concern, not circularity: the paper never claims that its own terminology is derived from the models it describes in a way that is logically forced by definition. Therefore the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Scaling laws from general-domain deep learning transfer to medical imaging.
- ad hoc to paper The four listed properties (large-scale, multimodal, self-supervised, emergent) are the defining characteristics of foundation models.
- domain assumption Emergent abilities appear in models as scale increases and are not present in smaller models.
Cite this review
Pith. "Pith review of Foundation Models in Radiology: What, How, When, Why and Why Not." pith.science (2026). https://pith.science/paper/NB7O2P3D
@misc{pith2026241118730,
author = {Pith},
title = {Pith review of: Foundation Models in Radiology: What, How, When, Why and Why Not},
year = {2026},
howpublished = {\url{https://pith.science/paper/NB7O2P3D}},
note = {Machine review of arXiv:2411.18730}
}
read the original abstract
Recent advances in artificial intelligence have witnessed the emergence of large-scale deep learning models capable of interpreting and generating both textual and imaging data. Such models, typically referred to as foundation models, are trained on extensive corpora of unlabeled data and demonstrate high performance across various tasks. Foundation models have recently received extensive attention from academic, industry, and regulatory bodies. Given the potentially transformative impact that foundation models can have on the field of radiology, this review aims to establish a standardized terminology concerning foundation models, with a specific focus on the requirements of training data, model training paradigms, model capabilities, and evaluation strategies. We further outline potential pathways to facilitate the training of radiology-specific foundation models, with a critical emphasis on elucidating both the benefits and challenges associated with such models. Overall, we envision that this review can unify technical advances and clinical needs in the training of foundation models for radiology in a safe and responsible manner, for ultimately benefiting patients, providers, and radiologists.
Forward citations
Cited by 1 Pith paper
-
From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
A PRISMA-ScR scoping review of 144 studies finds the field shifting from text-only LLMs to multimodal AI in medicine, with evaluation and data diversity still the main bottlenecks.
Reference graph
Works this paper leans on
-
[1]
Introduction Advancements in artificial intelligence (AI) have led to models that excel in specific tasks, often outperforming humans in controlled environments. For instance, given input radiological images and labels by human experts, traditional AI models have been trained using supervised learning to perform tasks such as disease detection and image s...
work page 2020
-
[15]
Demirer M, Candemir S, Bigelow MT, Yu SM, Gupta V, Prevedello LM, White RD, Yu JS, Grimmer R, Wels M. A user interface for optimizing radiologist engagement in image data curation for artificial intelligence. Radiology: Artificial Intelligence 2019;1(6):e180095. 16. Wei J, Tay Y, Bommasani R, Raffel C, Zoph B, Borgeaud S, Yogatama D, Bosma M, Zhou D, Metz...
work page 2019
-
[32]
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin J, Chang M-W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805 2018. 33. Wei C, Fan H, Xie S, Wu C-Y, Yuille A, Feichtenhofer C. Masked feature prediction for self-supervised visual pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...
work page 2018
-
[47]
A simple episodic linear probe improves visual recognition in the wild
Liang Y, Zhu L, Wang X, Yang Y. A simple episodic linear probe improves visual recognition in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition2022; p. 9559-9569. 48. Sanh V, Webson A, Raffel C, Bach SH, Sutawika L, Alyafeai Z, Chaffin A, Stiegler A, Scao TL, Raja A. Multitask prompted training enables zero-shot t...
work page 2021
-
[62]
SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million masks
Ye J, Cheng J, Chen J, Deng Z, Li T, Wang H, Su Y, Huang Z, Chen J, Jiang L. SA-Med2D-20M Dataset: Segment Anything in 2D Medical Imaging with 20 Million masks. arXiv preprint arXiv:231111969 2023. 63. Feng S, Azzollini D, Kim JS, Jin C-K, Gordon SP, Yeoh J, Kim E, Han M, Lee A, Patel A. Curation of the candid-ptx dataset with free-text reports. Radiology...
-
[78]
RaLEs: a Benchmark for Radiology Language Evaluations
Chaves JMZ, Bhaskhar N, Attias M, Delbrouck J-B, Rubin D, Loening AM, Langlotz C, Chaudhari AS. RaLEs: a Benchmark for Radiology Language Evaluations. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track2023. 79. Harrer S. Attention is not all you need: the complicated case of ethically using large language mode...
work page 2023
-
[95]
Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models
Yu S, Muñoz JP, Jannesari A. Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models. arXiv preprint arXiv:230511414 2023. 96. Yi H, Qin Z, Lao Q, Xu W, Jiang Z, Wang D, Zhang S, Li K. Towards General Purpose Medical AI: Continual Learning Medical Foundation Model. arXiv preprint arXiv:230306580 2023. 97. Wang L, Zhang ...
work page 2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.