REVIEW 3 major objections 5 minor 36 references
Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that fine-tuning recent natural-domain foundation models—AIMv2 above all—improves medical image classification across mammography, skin lesions, diabetic retinopathy, and chest X-rays.
desk verdict Useful benchmark extension, but the central claim that AIMv2 wins every configuration is contradicted by the paper's own tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrier of the argument is a controlled transfer-learning protocol: each encoder is kept as a feature extractor, a simple linear head or a multi-layer attention head is placed on top, and the backbone is either frozen or fine-tuned end to end. For grayscale inputs, the authors modify the patch-embedding layer by keeping only the first RGB channel's weights, allowing models pre-trained on color images to accept single-channel X-rays and mammograms without architectural change. The comparison across model families and backbone sizes, all evaluated at 224 by 224 resolution on the same four datasets with the same optimizer and learning-rate search, is what lets the authors attribute performance differences to the pre-training method. The multi-layer attention head, applied to patch embeddings rather than the classification token, is the component that most changes behavior between frozen and unfrozen modes.
What would settle it
Recompute all four benchmarks under a patient-exclusive split, where every image of a given patient is assigned to one fold. If AIMv2's margin over DINOv2 and SAM2 shrinks or the order changes when this split is enforced, the paper's central ranking would be shown to depend on data leakage rather than on model quality.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a clean empirical ranking: AIMv2 achieves the highest ROC-AUC, quadratic Cohen kappa, or recall in all four datasets under both frozen and unfrozen configurations, with DINOv2 and SAM2 close behind. The paper interprets this as evidence that each generation of natural-domain pre-training, from CLIP's contrastive language-image training to MAE's masked reconstruction to DINOv2's self-distillation and AIMv2's multimodal autoregression, improves the features that medical classifiers can reuse. It also reports that unfreezing the backbone consistently improves results and that a multi-layer attention head is better than a linear head when features are frozen. The result is intended to show that domain-specific medical pre-training is not required for strong classification performance and that advances in the natural domain transfer directly.
Load-bearing premise
The results assume that a random 80/10/10 split of image files, rather than patients, gives unbiased test scores; with CheXpert and CBIS-DDSM containing repeated images of the same patient, leakage could inflate the reported metrics and alter the ranking.
Editorial extensions
If this is right
- If the ranking holds, practitioners should prefer AIMv2, DINOv2, or SAM2 as backbones for medical classification rather than older CLIP or MAE encoders, even on small labeled datasets.
- Fine-tuning the backbone should be considered a default choice: the paper finds it consistently improves over frozen-feature classification, often making linear heads competitive with attention heads.
- With frozen backbones, the multi-layer attention head is the better classifier; the paper reports it outperforms linear heads on all four datasets in this configuration.
- Model-size choice depends on dataset size: base and large backbones win on smaller datasets, while the largest backbones win on the larger ISIC2019 and CHEXPERT collections.
- The gains of AIMv2, DINOv2, and SAM2 come with lower training time than CoCa, making the best-performing models also the more efficient ones in these experiments.
Reading between the lines
- The paper's random 80/10/10 split may leak patient information: CheXpert and CBIS-DDSM contain multiple images per patient, and without patient-level separation the reported AUC and kappa could be inflated; a patient-exclusive split is the natural stress test of the ranking.
- The claim that natural-domain progress transfers to medicine is inferred from a historical correlation across model families; the paper does not isolate which pre-training objective, contrastive, masked, autoregressive, or segmentation, drives the improvement.
- The single-channel patch-embedding trick could be tested beyond classification, for example in medical segmentation or detection, where SAM2's own dense features may transfer even better than the classification heads used here.
- Because only one or two backbone sizes per family are tested, the ranking may partly reflect model capacity and training budget rather than pre-training method; matching parameter counts across families would separate those factors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates seven recent vision foundation models (CLIP, DINOv2, MAE, VMamba, CoCa, SAM2, AIMv2) on four medical image classification datasets (CBIS-DDSM, APTOS2019, ISIC2019, CHEXPERT), under four configurations: frozen or unfrozen backbones with linear or multi-layer attention heads. The main empirical claim, stated in Section 3.2 and the Conclusion, is that AIMv2 achieves the highest performance across both frozen and unfrozen configurations in all four datasets, with DINOv2 and SAM2 following closely, and that recent advances in natural-domain training transfer positively to medical imaging. The paper also reports configuration-dependent findings on linear versus attention heads and on model-size trends, and it makes the code publicly available.
Significance. If substantiated, the paper would provide practically useful guidance for selecting foundation models for medical classification transfer, especially the recommendation of AIMv2, DINOv2, and SAM2 over older models such as CLIP and MAE. The study covers a broad and current set of models, applies four standard medical benchmarks, and releases code, which are strengths. However, the central ranking claim is contradicted by the paper's own tables in at least two configurations, and the evaluation lacks error bars, multiple-seed statistics, or significance tests. The absence of patient-level splitting for datasets with multiple images per patient is a further validity threat. The underlying benchmark is reproducible in principle, but the headline conclusions are not yet supported as stated.
major comments (3)
- [Section 3.2, Tables 1-4] The claim that 'AIMv2 achieved the highest performance across both frozen and unfrozen configurations in all four datasets' is contradicted by the reported numbers. In Table 2 (APTOS2019), the frozen-linear maximum is DINOv2 ViT-L at 0.889, while AIMv2 ViT-L is 0.888; in the unfrozen-attention column, CLIP ViT-B ties AIMv2 ViT-L at 0.907. In Table 3 (ISIC2019), the frozen-attention maximum is MAE ViT-H at 0.802, exceeding AIMv2 ViT-H at 0.795. The text in Section 3.2 and the Conclusion should be corrected to describe the actual per-configuration maxima or should explicitly qualify the ranking with these exceptions.
- [Section 3.2 and Conclusion] The evaluation provides no error bars, no multiple-seed statistics, and no significance tests, yet the Conclusion states that the models 'significantly enhance' outcomes and the ranking labels models as 'highest.' Several margins used for ranking are 0.001-0.003, for example APTOS2019 unfrozen-linear AIMv2 0.915 versus DINOv2 0.913, CHEXPERT unfrozen-linear AIMv2 0.819 versus DINOv2 0.814, and CBIS-DDSM unfrozen-linear AIMv2 0.968 versus DINOv2 0.966. These differences are within plausible run-to-run variation, so the ranking claims are not statistically supported. Please report multiple runs with confidence intervals or paired significance tests, and soften the language accordingly.
- [Section 3.1, data split] The data split is described as a random 80/10/10 division of images, without patient-level separation. CheXpert and CBIS-DDSM contain multiple images per patient, so images from the same patient can appear in both training and test sets, which can inflate AUC and kappa values through leakage. This could materially change the reported rankings, especially at the small margins observed. The authors should use patient-exclusive stratified splits, or provide a clear justification for why such leakage is negligible for the reported comparisons.
minor comments (5)
- [Section 2.2, ISIC2019] The text states that ISIC2019 includes 'nine diagnostic categories' and then lists 'benign keratosis' and 'seborrheic keratosis' as separate entries. In the standard ISIC 2019 challenge, there are eight lesion categories, with benign keratosis (BKL) encompassing seborrheic keratosis. Please correct the description and the class count.
- [Section 3.1, attention head] The phrase 'we used a pre-trained transformer [12]' is misleading: reference [12] is a paper on transfer learning for medical images, not a transformer architecture definition. Please cite the appropriate ViT or transformer reference, or rephrase to say a multi-layer Transformer head was used.
- [Section 2.2, APTOS2019] The metric is called 'Quadratic Cohen Kappa'; the standard term is 'quadratic weighted Cohen's kappa.' Please use the correct name.
- [Section 3.1, grayscale adaptation] For grayscale datasets, the authors remove the second and third channel weights from the patch embedding layer. This is a reasonable design choice, but the paper does not compare it with alternatives such as averaging the three channels or replicating the gray image to three channels. A brief sensitivity note would strengthen the methodology.
- [Figure 1] The text in Section 3.4 refers to 'fig. 1' for the model-size comparison, but the figure and its caption are not included in the submitted text. Please ensure the figure is present and that the axes and compared configurations are clearly described.
Circularity Check
No circularity: the paper is an empirical benchmark of external foundation models on public medical datasets, and its self-citations are contextual rather than load-bearing.
full rationale
The paper's claimed derivation chain is purely empirical: it takes publicly released pretrained checkpoints (CLIP, DINOv2, MAE, VMamba, CoCa, SAM2, AIMv2), fine-tunes them on four public medical datasets, and reports measured metrics (AUC, recall, kappa). No equation in the paper defines any reported metric as a function of a fitted parameter, and no target quantity is constructed from the model outputs by definition. The central ranking claim is that 'AIMv2 achieved the highest performance across both frozen and unfrozen configurations in all four datasets' (Section 3.2), but this is a read-off from Tables 1-4, not a derivation; even if the claim is contradicted by the paper's own tables (e.g., DINOv2 ViT-L exceeds AIMv2 in APTOS frozen-linear, and MAE ViT-H exceeds AIMv2 ViT-H in ISIC2019 frozen-attention), that is an internal-consistency or correctness concern, not circularity. The two self-citations (Refs. 16 and 18, both polyp-segmentation papers by overlapping authors) are listed among prior attempts to adapt foundation models to the medical domain and are not used to justify the reported rankings or any methodological choice. There is no self-citation chain, no imported uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result. Therefore the paper is not circular; its empirical content stands or falls on the validity of its experimental setup, not on any definitional or self-referential reduction.
Assumptions & free parameters
free parameters (2)
- Learning rate =
Not reported per model; searched in range 1e-3 to 1e-5
- Classification head type =
Linear vs attention head, chosen by validation performance
assumptions (3)
- domain assumption Random 80/10/10 image-level splitting produces unbiased test estimates.
- domain assumption Downsampling all inputs to 224x224 and replacing RGB patch embeddings with the first-channel weights preserves transferable information.
- domain assumption The chosen evaluation metrics (AUC, recall, quadratic kappa) capture clinically meaningful performance.
Cite this review
Pith. "Pith review of Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models." pith.science (2026). https://pith.science/paper/ENBIYVGK
@misc{pith2026250519779,
author = {Pith},
title = {Pith review of: Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/ENBIYVGK}},
note = {Machine review of arXiv:2505.19779}
}
read the original abstract
Using massive datasets, foundation models are large-scale, pre-trained models that perform a wide range of tasks. These models have shown consistently improved results with the introduction of new methods. It is crucial to analyze how these trends impact the medical field and determine whether these advancements can drive meaningful change. This study investigates the application of recent state-of-the-art foundation models, DINOv2, MAE, VMamba, CoCa, SAM2, and AIMv2, for medical image classification. We explore their effectiveness on datasets including CBIS-DDSM for mammography, ISIC2019 for skin lesions, APTOS2019 for diabetic retinopathy, and CHEXPERT for chest radiographs. By fine-tuning these models and evaluating their configurations, we aim to understand the potential of these advancements in medical image classification. The results indicate that these advanced models significantly enhance classification outcomes, demonstrating robust performance despite limited labeled data. Based on our results, AIMv2, DINOv2, and SAM2 models outperformed others, demonstrating that progress in natural domain training has positively impacted the medical domain and improved classification outcomes. Our code is publicly available at: https://github.com/sajjad-sh33/Medical-Transfer-Learning.
Reference graph
Works this paper leans on
-
[13]
Bert: Pre-training of deep bidirectional transformers for language understand- ing,
J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understand- ing,” inProceedings of naacL-HLT, vol. 1, no. 2. Minneapolis, Minnesota, 2019
work page 2019
-
[1]
INTRODUCTION Foundation models are large-scale, pre-trained models designed to perform a wide range of tasks by learning from massive datasets. In recent years, foundation models have gained significant interest in the research community, becoming a cornerstone in various fields of artificial intelligence [1]. Their development began with ground- breaking...
-
[2]
[8], AIMv2 [9], and CoCa (Cooperative Convolutional Attention)
-
[3]
EXPERIMENTS In this section, we detail the comprehensive experimental setup and results of the introduced models and medical image datasets dis- cussed in the previous section. 3.1. Implementation Details As mentioned earlier, two distinct configurations were employed for designing the classification head. We used both linear and multi- layer attention he...
-
[4]
METHODOLOGY Our methodology explores the potential of fine-tuning advanced foundation models such as CoCa, VMamba, AIMv2, MAE, SAM2, DINOv2, and CLIP to improve the performance and reliability of medical image classification tools. By fine-tuning these models on specific medical datasets, we aim to uncover their effectiveness in distinguishing various med...
work page Pith review arXiv 2025
-
[5]
Masked autoencoders are scalable vision learners,
K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” inPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009
2022
-
[6]
Dinov2: Learning robust visual features without super- vision,
M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without super- vision,”arXiv preprint arXiv:2304.07193, 2023
arXiv 2023
-
[7]
For DDSM and CheXpert, the image channels were set to 1 since they are grayscale images
with the model backbone frozen and only the classification head trained, and 2) with the entire model fine-tuned. For DDSM and CheXpert, the image channels were set to 1 since they are grayscale images. To handle this, instead of increasing the image channels, similar to [13], we modified the patch embeddings of the models to work with single-channel imag...
Show all 36 references
-
[8]
By fine-tuning these models on diverse datasets such as CBIS-DDSM, ISIC2019, APTOS2019, and CHEXPERT, we have demonstrated their capabil- ity to improve classification performance
CONCLUSION This study highlights the significant impact of state-of-the-art foun- dation models, including DINOv2, MAE, VMamba, CoCa, SAM2, AIMv2, and CLIP, on medical image classification. By fine-tuning these models on diverse datasets such as CBIS-DDSM, ISIC2019, APTOS2019,...
-
[9]
On the opportunities and risks of foundation models,
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021
2021 arXiv
-
[10]
These computer vision foundation models exhibit robust performance, even in scenarios involving limited labeled data, by leveraging pre-trained knowledge
have set brand-new benchmarks for various visual tasks. These computer vision foundation models exhibit robust performance, even in scenarios involving limited labeled data, by leveraging pre-trained knowledge. The medical imaging field often encounters difficulties due to a l...
-
[11]
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019
2019
-
[12]
We use the ViT-B and ViT-L models as the backbone for feature extraction
and MSCOCO [21] object detection, surpassing its previous state-of-the-art models. We use the ViT-B and ViT-L models as the backbone for feature extraction. •MAE[5]: MAE leverages an encoder-decoder framework tai- lored to representation learning tasks. A total of 130 million ...
-
[14]
Learning transferable visual models from natural language su- pervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language su- pervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763
2021
-
[15]
Vmamba: Visual state space model,
Y . Liu, Y . Tian, Y . Zhao, H. Yu, L. Xie, Y . Wang, Q. Ye, J. Jiao, and Y . Liu, “Vmamba: Visual state space model,”arXiv preprint arXiv:2401.10166, 2024
2024 arXiv
-
[16]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[17]
Mul- timodal autoregressive pre-training of large vision encoders,
E. Fini, M. Shukor, X. Li, P. Dufter, M. Klein, D. Haldimann, S. Aitharaju, V . G. T. da Costa, L. B´ethune, Z. Ganet al., “Mul- timodal autoregressive pre-training of large vision encoders,” arXiv preprint arXiv:2411.14402, 2024
2024 arXiv
-
[18]
Coca: Contrastive captioners are image-text foun- dation models,
J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foun- dation models,”arXiv preprint arXiv:2205.01917, 2022
2022 arXiv
-
[19]
Pre- trained vits yield versatile representations for medical images,
C. Matsoukas, J. F. Haslum, M. S¨oderberg, and K. Smith, “Pre- trained vits yield versatile representations for medical images,” arXiv preprint arXiv:2303.07034, 2023
2023 arXiv
-
[20]
What makes transfer learning work for medical im- ages: Feature reuse & other factors,
C. Matsoukas, J. F. Haslum, M. Sorkhei, M. S ¨oderberg, and K. Smith, “What makes transfer learning work for medical im- ages: Feature reuse & other factors,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2022, pp. 9225–9234
2022
-
[21]
Are natural domain foundation models useful for medical image classification?
J. P. Huix, A. R. Ganeshan, J. F. Haslum, M. S ¨oderberg, C. Matsoukas, and K. Smith, “Are natural domain foundation models useful for medical image classification?” inProceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 7634–7643
2024
-
[22]
Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation,
P. Shi, J. Qiu, S. M. D. Abaxi, H. Wei, F. P.-W. Lo, and W. Yuan, “Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation,”Diagnostics, vol. 13, no. 11, p. 1947, 2023
1947
-
[23]
Medclip: Con- trastive learning from unpaired medical images and text,
Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Con- trastive learning from unpaired medical images and text,”arXiv preprint arXiv:2210.10163, 2022
2022 arXiv
-
[24]
Polyp sam 2: Advancing zero shot polyp segmentation in colorectal cancer detection,
M. Mansoori, S. Shahabodini, J. Abouei, K. N. Plataniotis, and A. Mohammadi, “Polyp sam 2: Advancing zero shot polyp segmentation in colorectal cancer detection,”arXiv preprint arXiv:2408.05892, 2024
2024 arXiv
-
[25]
Towards general purpose medical ai: Con- tinual learning medical foundation model,
H. Yi, Z. Qin, Q. Lao, W. Xu, Z. Jiang, D. Wang, S. Zhang, and K. Li, “Towards general purpose medical ai: Con- tinual learning medical foundation model,”arXiv preprint arXiv:2303.06580, 2023
2023 arXiv
-
[26]
Self-prompting polyp segmentation in colonoscopy using hybrid yolo-sam 2 model,
M. Mansoori, S. Shahabodini, J. Abouei, K. N. Plataniotis, and A. Mohammadi, “Self-prompting polyp segmentation in colonoscopy using hybrid yolo-sam 2 model,”arXiv preprint arXiv:2409.09484, 2024
2024 arXiv
-
[27]
Unleashing the potential of sam2 for biomedical images and videos: A survey,
Y . Zhang and Z. Shen, “Unleashing the potential of sam2 for biomedical images and videos: A survey,”arXiv preprint arXiv:2408.12889, 2024
2024 arXiv
-
[28]
An image is worth 16x16 words: Trans- formers for image recognition at scale,
A. Dosovitskiy, “An image is worth 16x16 words: Trans- formers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
-
[29]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, ...
2014
-
[30]
A curated mammography data set for use in computer-aided detection and diagnosis research,
R. S. Lee, F. Gimenez, A. Hoogi, K. K. Miyake, M. Gorovoy, and D. L. Rubin, “A curated mammography data set for use in computer-aided detection and diagnosis research,”Scientific data, vol. 4, no. 1, pp. 1–9, 2017
2017
-
[31]
Aptos 2019 blindness detection,
M. Karthik and S. Dane, “Aptos 2019 blindness detection,” Kaggle https://kaggle. com/competitions/aptos2019-blindness- detection Go to reference in, p. 5, 2019
2019
-
[32]
Imagenet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255
2009
-
[33]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023
2023 arXiv
-
[34]
N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittleret al., “Skin lesion analysis toward melanoma de- tection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the i...
2017
-
[35]
Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,
J. Irvin, P. Rajpurkar, M. Ko, Y . Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskayaet al., “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” inProceedings of the AAAI conference on artificial intelligence,...
2019
-
[36]
The digital database for screening mammogra- phy,
M. Heath, K. Bowyer, D. Kopans, R. Moore, and W. P. Kegelmeyer, “The digital database for screening mammogra- phy,” inProceedings of the Fifth International Workshop on Digital Mammography, 2001, pp. 212–218
2001
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.