Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that on the MedMNIST collection, end-to-end fine-tuning of pre-trained foundation models consistently beats linear probing, that ViT-based backbones achieve most top results, and that previous benchmark conclusions about…

desk verdict A carefully run MedMNIST benchmark with a genuine protocol-sensitivity finding, weakened by best-on-test learning-rate selection for the end-to-end arm; deserves review with a request for validation-based tuning. read the letter →

arxiv 2501.14685 v1 pith:OX7JFJOH submitted 2025-01-24 eess.IV cs.AIcs.CVcs.LG

Fuping Wu , Bartlomiej W. Papiez This is my paper · ORCID
classification eess.IVcs.AIcs.CVcs.LG
keywords foundationmodelsmedicalimageclassificationMedMNISTtransferlearningfine-tuninglinearprobingvisiontransformersbenchmarkstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that how a pre-trained foundation model is transferred to medical image classification can change the answer to which model is best. On the 12-dataset MedMNIST benchmark, running 12 backbones with a constant classifier learning rate and a separately tuned encoder learning rate, the authors find that end-to-end fine-tuning consistently beats frozen-feature linear probing, that ViT-based models win most top spots, and that a prior study using a different optimizer-scheduler setup reached opposite conclusions. The practical point is that model selection and claims about self-supervised or CNN models are pipeline-dependent, so benchmark conclusions should state the exact training protocol.

What carries the argument

The machinery is a controlled transfer setup: 12 pre-trained encoders (VGG16, ResNet-18, DenseNet-121, EfficientNet-B4, ViT-B/16, CLIP ViT-B/16, SAM-C, SAM, EVA-02, OpenAI ViT-B/16, DINO, DINOv2) each capped with a one-layer linear classifier and trained for 15,000 AdamW iterations under two strategies, namely linear probing with a frozen encoder and end-to-end fine-tuning with a small constant encoder learning rate. The comparison is carried by three quantitative contrasts: the consistent end-to-end versus linear-probing gap, the per-family optimal encoder learning rate, and the head-to-head reproduction of a different fine-tuning pipeline (0.0001 learning rate with single-cycle cosine annealing) that yields different model rankings. Resizing and data-efficiency ablations on DermaMNIST extend the same setup to image resolution, resizing mode, and training-set size.

What would settle it

Re-run the same 12 models on MedMNIST but choose the encoder learning rate on a held-out portion of the training set, or fix one learning rate for all models, then measure test accuracy and AUC. If end-to-end fine-tuning no longer beats linear probing on every dataset, or if CNN models regain the top spots, the paper's central claims are falsified. A second decisive check is to reproduce the exact prior pipeline, AdamW at 0.0001 with single-cycle cosine annealing, and see whether ViT-based models still outperform CNNs under end-to-end fine-tuning.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that the choice of transfer strategy reverses published guidance: end-to-end fine-tuning outperforms linear probing in both accuracy and AUC on all 12 MedMNIST datasets, with the sole exception of PathMNIST AUC where linear-probed DINO ViT-B/16 reaches 99.78% versus 99.75% for fine-tuning. Under both strategies most best results come from ViT-based models, with ViT-B/16 leading the classification-targeted group and DINO or DINOv2 leading the feature-extraction group; DenseNet-121 and EfficientNet-B4 win only on PneumoniaMNIST accuracy and BloodMNIST AUC. Direct comparison on eight common models shows the previous study's linear probing results are similar but its end-to-end fine-tuning results are substantially worse, reversing its conclusions that CNNs beat ViTs in fine-tuning and that CLIP or DINO style self-supervised models only help with linear probing. Auxiliary findings are that an encoder learning rate of 1e-4 for CNNs and 1e-5 for ViTs is generally optimal, scaling small images to 224x224 beats zero-padding under fine-tuning, and classification accuracy still rises sharply as training data grows from 20 to 600 images per class.

Load-bearing premise

The rankings and the end-to-end versus linear-probing conclusions assume that choosing each model's best test accuracy among three encoder learning rates, with no held-out validation split or selection penalty, is a fair comparison; if that selection advantage differs across models, the ranking and even the main pipeline conclusion could shift.

Editorial extensions

If this is right

  • End-to-end fine-tuning is the better default transfer strategy for MedMNIST classification; the single path exception on PathMNIST AUC is only a 0.03% difference.
  • ViT-based pre-trained backbones should generally be preferred over ImageNet CNN backbones for these tasks when fine-tuned this way, since they take most top accuracy and AUC slots across the 12 datasets.
  • Encoder learning rate can be set by architecture family rather than per model: roughly 1e-4 for CNNs and 1e-5 for ViTs.
  • Earlier benchmark conclusions about CNN versus ViT superiority and about self-supervised models only helping with linear probing do not survive a change in the fine-tuning pipeline, so published guidance should be treated as protocol-specific.
  • When smaller images must be fed to a 224x224 model, scaling rather than zero-padding is generally the better resizing choice for end-to-end fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if training pipeline can flip CNN versus ViT rankings, then other head-to-head medical imaging benchmark conclusions should include full training-protocol details before being used for model selection.
  • The observed low data efficiency, with accuracy climbing steeply from 20 to 600 images per class, suggests the common expectation that foundation models need little data does not hold for these medical tasks, so few-shot medical classification likely needs task-specific adaptation methods rather than simply larger pre-trained backbones.
  • A testable extension of the paper's pipeline-dependence finding would be to ablate the scheduler alone: keep the same optimizer and learning rates but compare constant learning rate with single-cycle cosine annealing to see whether the scheduler, not the learning-rate choice, drives the reversal of the prior study's conclusions.
  • Another testable extension is to check whether the size of the end-to-end versus linear-probing gap correlates with dataset difficulty across the 12 MedMNIST datasets, which would turn the observed gap on DermaMNIST, OCTMNIST, OrganC/S, and TissueMNIST into a predictive rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a benchmark study of 12 foundation models (four CNN-based, eight ViT-based) on the 12 MedMNIST v2 classification datasets under two transfer strategies: linear probing and end-to-end fine-tuning. The central claims are that end-to-end fine-tuning consistently outperforms linear probing across datasets, that ViT-based models generally achieve the best performance, and that the authors' training pipeline reverses several conclusions of Doerrich et al. (2024). The paper also reports experiments on image resizing (scaling vs. zero-padding) and data efficiency on DermaMNIST, and compares against three medical-domain foundation models from Zhang et al. (2024).

Significance. If the claims hold, the paper makes a useful contribution: it provides a large, consistently executed benchmark across 12 datasets and 12 models, and it concretely demonstrates that model rankings for medical image classification can depend strongly on the training pipeline, which is an important caution for the community. The paper's strengths include the breadth of models and datasets, the explicit identification of a concrete pipeline difference from Doerrich et al. (2024), the three-run reporting with mean and standard deviation, and the additional analyses of resizing strategies and data efficiency. However, the headline comparisons are currently undermined by the test-set-based selection of encoder learning rates and by the asymmetric treatment of the linear probing baseline; these issues must be addressed before the central claims can be considered reliable.

major comments (3)
  1. [Section 3.2.2 / Table 1 and Tables 4-6] The reported 'best end-to-end fine-tuning results' are selected per model and per dataset from three encoder learning rates (Section 3.2.1), but no separate validation split or selection penalty is described. Because the selection appears to be made on the test set, the end-to-end numbers may be optimistically biased, and the magnitude of this bias can vary across models. The linear probing baseline (Section 2.2) uses a single fixed classifier learning rate with a fixed decay schedule and receives no analogous hyperparameter scan. This asymmetry is load-bearing: the claim that end-to-end fine-tuning consistently outperforms linear probing could be driven partly by giving one arm three chances to look good on the test metric. I request that the authors either introduce a validation-based selection protocol (e.g., select lre on a held-out split and report the corresponding test performance), or report results for a fixed, pre-specified learning rate in addition to the best-of-three, and show that the headline conclusions survive that check.
  2. [Section 3.2.3] The comparison between CNN and ViT models is confounded by the use of different learning-rate grids: CNNs are tuned over {10^-3, 10^-4, 10^-5} while ViTs are tuned over {10^-4, 10^-5, 10^-6}. Since each family effectively gets its own best-of-three test-set selection, the observed ViT advantage could reflect a difference in tuning budget or grid placement rather than an architectural advantage. A fairer comparison would use a common set of candidate learning rates for both families, or at least report results for a matched learning rate (e.g., 10^-4 for both) to show that the ViT-vs-CNN ranking is not an artifact of the different grids.
  3. [Section 3.2.4] The claim that the training pipeline reverses the conclusions of Doerrich et al. (2024) is not isolated by the experimental design. The comparison varies multiple components simultaneously: the learning-rate schedule (cosine annealing single cycle with lr 0.0001 vs. the authors' constant encoder LR and step-decayed classifier LR), the selection protocol (whether and how encoder learning rates are tuned), and potentially preprocessing and checkpoint sources. The presented experiments show only that two different protocols yield different numbers; they do not identify which pipeline component causes the reversal. An ablation (e.g., running the authors' models under Doerrich et al.'s LR schedule, or vice versa) is needed to support the causal attribution implied by the text.
minor comments (4)
  1. [Figure 9 caption] The caption reads 'RSAM-C ViT-B/16' but should be 'SAM-C ViT-B/16'.
  2. [Tables 3 and 5, and several section headings] There are recurring formatting typos: 'EV A-02' should be 'EVA-02', 'V alidation' should be 'Validation', 'T ransferring' should be 'Transferring', and 'L VD-142M' should be 'LVD-142M'.
  3. [Section 4.3 / Table 1] The comparison with Zhang et al. (2024) is incomplete: only accuracy is reported for their models, no AUC values, and no linear probing results are available. The manuscript acknowledges the absence of linear probing, but the lack of AUC makes the comparison with the main tables asymmetric and should be stated more prominently.
  4. [Section 5] The limitations listed in the Conclusion (restriction to MedMNIST, resizing analysis limited to DermaMNIST, lack of per-class analysis, and unexamined model biases) are appropriate, but they should be reflected earlier in the paper (e.g., in the Introduction or Discussion) so that the scope and claims of the benchmark are clear to the reader from the outset.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports measured benchmark results and does not derive predictions from fitted inputs or self-citations.

full rationale

This is an empirical benchmark study whose claims are direct summaries of measured tables: end-to-end fine-tuning versus linear probing, CNN versus ViT-based models, and a comparison with Doerrich et al. (2024). No quantity is derived from another by construction, no fitted parameter is renamed as a prediction, and no load-bearing argument rests on a self-citation. The closest concern is methodological rather than circular: Section 3.2.1 tunes the encoder learning rate per model, and Section 3.2.2 reports 'the best end-to-end fine-tuning results' on the same test metric without describing a validation split, while the linear probing baseline uses a fixed classifier learning rate. That asymmetry is a test-set selection and fairness risk for the headline comparisons, but it is not circular reasoning because the reported numbers remain measurements rather than quantities forced by the paper's own definitions. Similarly, the reversal of Doerrich et al.'s conclusions changes several pipeline components at once, which is a controlled-comparison limitation, not a circular step. The only apparent self-citation, Alloula et al. (2024), appears in a closing limitation sentence about model biases and is not load-bearing. The paper is therefore self-contained as an empirical benchmark report, with no identifiable circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central benchmark rests on empirical protocol choices rather than mathematical axioms. The listed assumptions are the representativeness of MedMNIST, the transfer-learning setup, the three-run statistics, and the timm checkpoint provenance. The only tuned quantities are hyperparameters (learning rates, iterations) chosen by hand; no invented entities are introduced.

free parameters (3)
  • Encoder learning rate per model (lre) = 10^-4 for CNNs, 10^-5 for ViT-based (selected per model from {1e-3,1e-4,1e-5} or {1e-4,1e-5,1e-6})
    The authors choose per-model encoder learning rates by scanning values and then report the best end-to-end results (Section 3.2.1 and 3.2.2). This choice directly controls the headline comparisons.
  • Classifier learning rate schedule = 0.001, decayed by 0.9 every 200 iterations
    Fixed by hand for all models (Section 2.2); not varied across models or datasets.
  • Number of training iterations = 15,000
    Fixed for all models (Section 2.2); a protocol choice that could favor some models over others.
assumptions (4)
  • domain assumption MedMNIST test set performance is the correct yardstick for comparing foundation models for medical image classification.
    The paper restricts validation to the MedMNIST collection (Section 3.1), so the conclusions about model selection are contingent on this suite being representative.
  • domain assumption Pre-trained encoders plus a linear/fine-tuning head are a valid operationalization of transferability.
    Section 2.2 appends a linear classifier and either freezes or fine-tunes the encoder; 'transferability' is defined by accuracy and AUC on MedMNIST.
  • domain assumption Three random repeats suffice to compare models.
    All experiments are run three times on a single A100 GPU (Section 3.1); no significance testing is used and the reported plus/minus standard deviations are used to rank models.
  • domain assumption Pretrained model checkpoints obtained from timm are equivalent to the originally published models.
    Table 3 references timm identifiers; the paper inherits whatever training details those checkpoints embody.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST." pith.science (2026). https://pith.science/paper/OX7JFJOH

@misc{pith2026250114685,
  author       = {Pith},
  title        = {Pith review of: Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OX7JFJOH}},
  note         = {Machine review of arXiv:2501.14685}
}
read the original abstract

Foundation models are widely employed in medical image analysis, due to their high adaptability and generalizability for downstream tasks. With the increasing number of foundation models being released, model selection has become an important issue. In this work, we study the capabilities of foundation models in medical image classification tasks by conducting a benchmark study on the MedMNIST dataset. Specifically, we adopt various foundation models ranging from convolutional to Transformer-based models and implement both end-to-end training and linear probing for all classification tasks. The results demonstrate the significant potential of these pre-trained models when transferred for medical image classification. We further conduct experiments with different image sizes and various sizes of training data. By analyzing all the results, we provide preliminary, yet useful insights and conclusions on this topic.

Figures

Figures reproduced from arXiv: 2501.14685 by the authors.

Figure 1
Figure 1. Framework of our study. We evaluate the performance using the MedMNIST dataset collection and select foundation models from a representative pool. SA-1B 1 for segmentation, its representations were demonstrated to be useful for various tasks. (2) Four models trained for feature extraction: OpenAI ViT-B/16 (Ilharco et al., 2021) was trained on publicly available image-caption data to maximize the similarity of (image… view at source ↗
Figure 2
Figure 2. Accuracy of DINO ViT-B/16 on DermaMNIST with various numbers of training data for each class, and “full” means using all training data. 4.3. Transferring Foundation Models Pre-trained on Medical Data Zhang et al. (2024) fine-tuned three foundation models pre-trained on medical data, includ￾ing BioMedGPT (Zhang et al., 2024), BioMedCLIP Zhang et al. (2023) and MedSAM (Ma et al., 2024), on seven datasets from the MedM… view at source ↗
Figure 3
Figure 3. Comparing the Accuracy of VGG16 with the learning rate of the encoder ranging in {10−3 , 10−4 , 10−5} on different datasets [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (31 more)
Figure 4
Figure 4. Figure 4: Comparing the Accuracy of ResNet-18 with the learning rate of the encoder ranging in {10−3 , 10−4 , 10−5} on different datasets. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Comparing the Accuracy of DenseNet-121 with the learning rate of the encoder ranging in {10−3 , 10−4 , 10−5} on different datasets [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Comparing the Accuracy of EfficientNet-B4 with the learning rate of the encoder ranging in {10−3 , 10−4 , 10−5} on different datasets. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Tuning the learning rate for the encoder of ViT-B/16 ranging in {10−4 , 10−5 , 10−6} on different datasets [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Comparing the Accuracy of CLIP ViT-B/16 with the learning rate of the encoder ranging in {10−4 , 10−5 , 10−6} on different datasets. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Comparing the Accuracy of RSAM-C ViT-B/16 with the learning rate of the encoder ranging in {10−4 , 10−5 , 10−6} on different datasets [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Comparing the Accuracy of SAM ViT-B/16 with the learning rate of the encoder ranging in {10−4 , 10−5 , 10−6} on different datasets. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Comparing the Accuracy of EVA-02 ViT-B/14 with the learning rate of the encoder ranging in {10−4 , 10−5 , 10−6} on different datasets [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Comparing the Accuracy of OpenAI ViT-B/16 with the learning rate of the encoder ranging in {10−4 , 10−5 , 10−6} on different datasets. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Comparing the Accuracy of DINO ViT-B/16 with the learning rate of the en￾coder ranging in {10−4 , 10−5 , 10−6} on different datasets [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Comparing the Accuracy of DINOv2 ViT-B/14 with the learning rate of the encoder ranging in {10−4 , 10−5 , 10−6} on different datasets. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Performance comparison of foundation models on DermaMNIST with different image sizes, when using end-to-end fine-tuning and scaling for image resizing. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Performance comparison of foundation models on DermaMNIST with different image sizes, when using end-to-end fine-tuning and zero-padding for image re￾sizing [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Performance comparison of foundation models on DermaMNIST with different image sizes, when using linear probing and scaling for image resizing. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Performance comparison of foundation models on DermaMNIST with different image sizes, when using linear probing and zero-padding for image resizing [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Performance comparison for zero-padding and scaling on DermaMNIST with an image size of 28 × 28, when using end-to-end fine-tuning. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Performance comparison for zero-padding and scaling on DermaMNIST with an image size of 64 × 64, when using end-to-end fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Performance comparison for zero-padding and scaling on DermaMNIST with an image size of 128 × 128, when using end-to-end fine-tuning. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: Performance comparison for zero-padding and scaling on DermaMNIST with an image size of 28 × 28, when using linear probing [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: Performance comparison for zero-padding and scaling on DermaMNIST with an image size of 64 × 64, when using linear probing. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: Performance comparison for zero-padding and scaling on DermaMNIST with an image size of 128 × 128, when using linear probing. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: Accuracy of DenseNet-121 on DermaMNIST when training model with a differ￾ent number of data for each class. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_25.png]
Figure 26
Figure 26. Figure 26: Accuracy of ResNet-18 on DermaMNIST when training model with a different number of data for each class [PITH_FULL_IMAGE:figures/full_fig_p031_26.png]
Figure 27
Figure 27. Figure 27: Accuracy of EfficientNet-B4 on DermaMNIST when training model with a dif￾ferent number of data for each class. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_27.png]
Figure 28
Figure 28. Figure 28: Accuracy of ViT-B/16 on DermaMNIST when training model with a different number of data for each class [PITH_FULL_IMAGE:figures/full_fig_p032_28.png]
Figure 29
Figure 29. Figure 29: Accuracy of CLIP ViT-B/16 on DermaMNIST when training model with a dif￾ferent number of data for each class. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_29.png]
Figure 30
Figure 30. Figure 30: Accuracy of SAM-C ViT-B/16 on DermaMNIST when training model with a different number of data for each class [PITH_FULL_IMAGE:figures/full_fig_p033_30.png]
Figure 31
Figure 31. Figure 31: Accuracy of SAM ViT-B/16 on DermaMNIST when training model with a dif￾ferent number of data for each class. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_31.png]
Figure 32
Figure 32. Figure 32: Accuracy of EVA-02 ViT-B/14 on DermaMNIST when training model with a different number of data for each class [PITH_FULL_IMAGE:figures/full_fig_p034_32.png]
Figure 33
Figure 33. Figure 33: Accuracy of DINO OpenAI ViT-B/16 on DermaMNIST when training model with a different number of data for each class. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_33.png]
Figure 34
Figure 34. Figure 34: Accuracy of DINOv2 ViT-B/14 on DermaMNIST when training model with a different number of data for each class. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_34.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 15 canonical work pages

  1. [1]

    On biases in a UK B iobank-based retinal image classification model

    Anissa Alloula, Rima Mustafa, Daniel R McGowan, and Bart omiej W Papie \.z . On biases in a UK B iobank-based retinal image classification model. In MICCAI Workshop on Fairness of AI in Medical Imaging, pages 140--150. Springer, 2024

    work page 2024

  2. [2]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  3. [3]

    Foundational models defining a new era in vision: A survey and outlook

    Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721, 2023

    arXiv 2023

  4. [4]

    Towards general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks

    Mohammed Baharoon, Waseem Qureshi, Jiahong Ouyang, Yanwu Xu, Kilian Phol, Abdulrhman Aljouie, and Wei Peng. Towards general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. arXiv preprint arXiv:2312.02366, 2023

    arXiv 2023

  5. [5]

    Universeg: Universal medical image segmentation

    Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21438--21451, 2023

    2023

  6. [6]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv 'e J 'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650--9660, 2021

    work page 2021

  7. [7]

    When vision transformers outperform resnets without pre-training or strong data augmentations

    Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. arXiv preprint arXiv:2106.01548, 2021

    arXiv 2021

  8. [8]

    Chexagent: Towards a foundation model for chest x-ray interpretation

    Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208, 2024

    arXiv 2024

Show all 38 references
  1. [9]

    Rethinking model prototyping through the medmnist+ dataset collection

    Sebastian Doerrich, Francesco Di Salvo, Julius Brockmann, and Christian Ledig. Rethinking model prototyping through the medmnist+ dataset collection. arXiv preprint arXiv:2404.15786, 2024

    2024 arXiv

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  3. [11]

    Eva-02: A visual representation for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149: 0 105171, 2024

    2024

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

    2016

  5. [13]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700--4708, 2017

    2017

  6. [14]

    Audiogpt: Understanding and generating speech, music, sound, and talking head

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intell...

    2024

  7. [15]

    Joana Pal \'e s Huix, Adithya Raju Ganeshan, Johan Fredin Haslum, Magnus S \"o derberg, Christos Matsoukas, and Kevin Smith. Are natural domain foundation models useful for medical image classification? In Proceedings of the IEEE/CVF Winter Conference on Applications of Comput...

    2024

  8. [16]

    Openclip, July 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/zenodo.5143773. I...

    2021 doi

  9. [17]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015--4026, 2023

    2023

  10. [18]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  11. [19]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, pages 1--13, 2022

    2022

  12. [20]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15: 0 654, 2024

    2024

  13. [21]

    Segment anything model for medical image analysis: an experimental study

    Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89: 0 102918, 2023

    2023

  14. [22]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  15. [23]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 0 211--252, 2015

    2015

  16. [24]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  17. [25]

    How to train your vit? data, augmentation, and regularization in vision transformers

    Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270, 2021

    2021 arXiv

  18. [26]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946, pages 6105--6114, 2019

    1905 arXiv

  19. [27]

    Xraygpt: Chest radiographs summarization using large medical vision-language models

    Omkar Chakradhar Thawakar, Abdelrahman M Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Khan. Xraygpt: Chest radiographs summarization using large medical vision-language models. In Proceedings of the 23rd Worksho...

  20. [28]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  21. [29]

    A real-world dataset and benchmark for foundation model adaptation in medical image classification

    Dequan Wang, Xiaosong Wang, Lilong Wang, Mengzhang Li, Qian Da, Xiaoqiang Liu, Xiangyu Gao, Jun Shen, Junjun He, Tian Shen, et al. A real-world dataset and benchmark for foundation model adaptation in medical image classification. Scientific Data, 10 0 (1): 0 574, 2023

    2023

  22. [30]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/huggingface/pytorch-image-models, 2019

    2019

  24. [32]

    Navigating data scarcity using foundation models: A benchmark of few-shot and zero-shot learning approaches in medical imaging

    Stefano Woerner and Christian F Baumgartner. Navigating data scarcity using foundation models: A benchmark of few-shot and zero-shot learning approaches in medical imaging. In International Workshop on Foundation Models for General Medical AI, pages 30--39. Springer, 2024

    2024

  25. [33]

    Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10 0 (1): 0 41, 2023

    2023

  26. [34]

    A generalist vision--language foundation model for diverse biomedical tasks

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision--language foundation model for diverse biomedical tasks. Nature Medicine, pages 1--13, 2024

    2024

  27. [35]

    On the challenges and perspectives of foundation models for medical image analysis

    Shaoting Zhang and Dimitris Metaxas. On the challenges and perspectives of foundation models for medical image analysis. Medical image analysis, 91: 0 102996, 2024

    2024

  28. [36]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023

    2023 arXiv

  29. [37]

    Recommender systems in the era of large language models (llms)

    Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, et al. Recommender systems in the era of large language models (llms). IEEE Transactions on Knowledge and Data Engineering, 2024

    2024

  30. [38]

    A comprehensive survey on pretrained foundation models: A history from bert to chatgpt

    Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.