REVIEW 4 major objections 5 minor 20 references
Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Frozen PanDerm features rival fine-tuned Swin Transformer on skin-lesion classification, and fusing their predictions improves accuracy further.
desk verdict Modest but competent benchmark of PanDerm vs fine-tuned ViTs; the central comparison hinges on undisclosed pretraining data and needs error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the frozen vision encoder of PanDerm, which converts each dermatoscopic image into an embedding, with nonlinear probing classifiers (MLP, XGBoost, TabNet) trained on top of those fixed features. The comparison arm is the ViT base and Swin Transformer V2 base, both fine-tuned end to end after ImageNet pretraining. The fusion mechanism combines the soft predictions of the best PanDerm probe (MLP) and the best ViT baseline (Swin), exploiting the complementarity between the two model families rather than merging their feature spaces.
What would settle it
Check PanDerm's pretraining data against the official HAM10000 and MSKCC test splits; any overlap of individual images would invalidate the central comparison. Alternatively, rerun the PanDerm-MLP probe on a new dermatoscopic dataset released after PanDerm's training cutoff; if accuracy drops far below the Swin baseline, the frozen-feature advantage is not general.
Extended reading notes
Core claim
Using frozen PanDerm embeddings with an MLP probe reaches 76.32% accuracy (64.40% balanced) on MSKCC and 92.69% (79.57% balanced) on HAM10000, nearly matching or exceeding the fully fine-tuned Swin Transformer V2 base (75.30% and 64.52% balanced on MSKCC; 91.80% and 81.43% balanced on HAM10000). Fusing PanDerm-MLP and Swin predictions improves accuracy further to 76.80% (64.36% balanced) on MSKCC and 93.59% (83.25% balanced) on HAM10000. The paper concludes that frozen dermatology foundation features are competitive with fine-tuned ViTs and that late fusion of the two gives the best overall accuracy.
Load-bearing premise
The comparison assumes PanDerm's training corpus did not include the HAM10000 or MSKCC test images; if it did, the reported accuracies reflect memorization rather than generalization.
Editorial extensions
If this is right
- A clinician-oriented system can use frozen PanDerm embeddings with a simple MLP head and reach accuracy comparable to a fully fine-tuned Swin Transformer, avoiding costly per-task fine-tuning of large models.
- Fusing PanDerm and Swin predictions gives the best accuracy on both datasets, so the two model families encode partially complementary lesion information that a late-fusion step can exploit.
- On MSKCC, the fusion's balanced accuracy (64.36%) is slightly below PanDerm-MLP (64.40%) and Swin (64.52%), so accuracy gains from fusion do not automatically translate into better class-balanced performance.
- Across PanDerm probes, MLP outperforms XGBoost and TabNet on both datasets, showing classifier choice materially affects frozen-feature performance.
Reading between the lines
- If PanDerm's training data is later shown to exclude both benchmarks, this result is a strong argument for caching frozen dermatology embeddings once and training only lightweight probes, which would make model updates in clinical workflows substantially cheaper.
- A natural next test is per-class analysis of the fusion: since HAM10000 is heavily class-imbalanced, fusion likely concentrates its gain on rare malignant categories, a claim the paper does not test directly.
- The same frozen-feature plus late-fusion approach could be tested on other dermatology tasks such as segmentation or dermoscopic attribute detection, where foundation-model embeddings may carry useful spatial information.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This short paper compares a dermatology-specific foundation model, PanDerm, with two ImageNet-pretrained vision transformers (ViT base and Swin Transformer V2 base) for skin lesion classification. PanDerm is used as a frozen feature extractor and probed with three non-linear classifiers (MLP, XGBoost, TabNet), while the ViT models are fine-tuned end to end. Experiments on HAM10000 and MSKCC use the official train/validation/test splits from reference [20]. The main claims are that the PanDerm-MLP model performs comparably to the fine-tuned Swin model, and that fusing PanDerm-MLP and Swin predictions improves accuracy. Results are reported in Table 1 as accuracy and balanced accuracy.
Significance. If the comparison is methodologically clean, the result is practically useful: it suggests that probing a dermatology-specific foundation model can match fine-tuning a general-purpose vision transformer, and that simple score fusion gives a further gain. The paper has several strengths: it uses the official splits from the PanDerm paper, reports both accuracy and balanced accuracy, and covers two public datasets. However, the significance is currently limited by the lack of statistical uncertainty quantification, by incomplete methodological disclosure (fusion procedure and hyperparameters), and by an unresolved question about whether PanDerm's pretraining data overlaps the evaluation test splits. The balanced-accuracy results also complicate the headline accuracy-based claims.
major comments (4)
- [Section 2 (Datasets and PanDerm representation)] The central comparison is fair only if PanDerm's pretraining corpus is disjoint from the HAM10000 and MSKCC test splits. The paper states that the official splits from reference [20] are used, but it never discloses what PanDerm was trained on. Because HAM10000 and MSKCC are public ISIC-era datasets and PanDerm is a dermatology-specific foundation model, overlap is plausible. The authors should state explicitly whether the pretraining data in [20] includes any of these images and, ideally, verify that the test images are not in the training set. Without this disclosure, the reported PanDerm-vs-Swin comparison could reflect memorization rather than generalization.
- [Table 1 and Section 3] The claim that PanDerm-MLP performs comparably to Swin is metric-dependent. On HAM10000, accuracy favors PanDerm-MLP (92.69% vs 91.80%), but balanced accuracy favors Swin (81.43% vs 79.57%). On MSKCC, the balanced-accuracy values are essentially tied (64.40% vs 64.52%). No confidence intervals or significance tests are provided, so it is unclear whether the reported differences are meaningful. Given the large gap between accuracy and balanced accuracy on HAM10000, the paper should justify why accuracy is treated as the primary metric and should report uncertainty or error bars for both metrics.
- [Section 3 and Table 1 (fusion row)] The fusion method is not specified. The last row of Table 1 is described only as combining predictions from the best ViT model and the best PanDerm-based model, but no equation, weighting scheme, or averaging procedure is given. Without this detail, the fusion result cannot be reproduced, and it is unclear whether the fusion gain is robust or an artifact of a particular combination rule.
- [Section 2 (training and probing details)] Hyperparameters for the MLP hidden layer and dropout, XGBoost, TabNet, and the fine-tuning schedules for ViT and Swin are omitted. No random seeds or multiple runs are reported. These omissions prevent reproducibility and make it impossible to assess whether the small differences in Table 1 are stable across runs.
minor comments (5)
- [Table 1] The entry 'Panderm_TabNet' has inconsistent capitalization; it should be 'PanDerm_TabNet' for consistency with the other rows.
- [Section 2] The preprocessing and input resolution for PanDerm embeddings are not described; if PanDerm uses a different input size or normalization than the ViT models, this should be stated.
- [Figure 1] The example images are not discussed in the text; a sentence describing what they illustrate would help the reader.
- [Section 2 (splits)] The manuscript says it follows the official splits from [20], but it does not specify which exact split indices or preprocessing steps were used; this should be clarified for reproducibility.
- [Title and Section 3] The title emphasizes fusion, but the fusion details appear only in the results section; moving a precise description of the fusion procedure to the Method section would improve the paper's structure.
Circularity Check
No circularity: the paper is an empirical benchmark comparing external model checkpoints on standard splits; the undisclosed PanDerm pretraining corpus is a data-leakage risk, not a circular derivation.
full rationale
The paper's claims are supported by direct measurements reported in Table 1: frozen PanDerm features are classified with MLP, XGBoost, and TabNet, and ViT/Swin baselines are fine-tuned on the same data. There is no fitted parameter renamed as a prediction, no theoretical derivation that reduces to its own inputs, and no uniqueness claim imported from the authors' prior work. The authors cite Yan et al. [20] for the model and the official splits, but [20] is an external source, not a self-citation. The self-citations present (e.g., [8], [12]-[15]) concern methodological context and are not load-bearing. The only substantive concern is that PanDerm's pretraining data is not disclosed, so one cannot rule out overlap with the HAM10000 or MSKCC test sets. If such overlap existed, the comparison would measure memorization rather than generalization. However, this is a data-hygiene and external-validity risk, not an internal circularity that can be exhibited from the paper's equations or definitions. Per the specified rules, speculation about undisclosed pretraining data cannot raise the circularity score. The benchmark is self-contained with respect to the stated evaluation protocol, so the appropriate score is 0.
Assumptions & free parameters
free parameters (5)
- MLP hidden layer size
- MLP dropout rate
- XGBoost hyperparameters
- TabNet hyperparameters
- Fine-tuning schedule for ViT and Swin
assumptions (4)
- domain assumption PanDerm was not pre-trained on the HAM10000 or MSKCC test images.
- domain assumption The official train/validation/test splits from [20] are valid and were not used to fit the classifiers in a way that leaks test information.
- domain assumption ImageNet statistics are appropriate for normalizing dermoscopic images for all models.
- domain assumption Full fine-tuning of the ViT models yields better performance than probing, as cited from [11].
Cite this review
Pith. "Pith review of Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification." pith.science (2026). https://pith.science/paper/2BRC3P4L
@misc{pith2026250516338,
author = {Pith},
title = {Pith review of: Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/2BRC3P4L}},
note = {Machine review of arXiv:2505.16338}
}
read the original abstract
Accurate classification of skin lesions from dermatoscopic images is essential for diagnosis and treatment of skin cancer. In this study, we investigate the utility of a dermatology-specific foundation model, PanDerm, in comparison with two Vision Transformer (ViT) architectures (ViT base and Swin Transformer V2 base) for the task of skin lesion classification. Using frozen features extracted from PanDerm, we apply non-linear probing with three different classifiers, namely, multi-layer perceptron (MLP), XGBoost, and TabNet. For the ViT-based models, we perform full fine-tuning to optimize classification performance. Our experiments on the HAM10000 and MSKCC datasets demonstrate that the PanDerm-based MLP model performs comparably to the fine-tuned Swin transformer model, while fusion of PanDerm and Swin Transformer predictions leads to further performance improvements. Future work will explore additional foundation models, fine-tuning strategies, and advanced fusion techniques.
Figures
Reference graph
Works this paper leans on
-
[20]
arXiv preprint arXiv:2410.15038 (2024)
Yan, S., Yu, Z., Primiero, C., Vico-Alonso, C., Wang, Z., Yang, L., Tschandl, P., Hu, M., Tan, G., Tang, V., et al.: A general-purpose multimodal foundation model for dermatology. arXiv preprint arXiv:2410.15038 (2024)
arXiv 2024
-
[1]
Adebiyi, A., Abdalnabi, N., Simoes, E.J., Becevic, M., Smith, E.H., Rao, P.: Trans- formers in skin lesion classification and diagnosis: A systematic review. medRxiv (2025). https://doi.org/https://doi.org/10.1101/2024.09.19.24314004
-
[2]
Proceed- ings of the AAAI Conference on Artificial Intelligence35(8), 6679–6687 (2021)
Arik, S.O., Pfister, T.: TabNet: Attentive interpretable tabular learning. Proceed- ings of the AAAI Conference on Artificial Intelligence35(8), 6679–6687 (2021). https://doi.org/https://doi.org/10.1609/aaai.v35i8.16826
-
[3]
Neural Computing and Applications35(9), 6713–6722 (2023)
Ayas, S.: Multiclass skin lesion classification in dermoscopic images using swin transformer model. Neural Computing and Applications35(9), 6713–6722 (2023). https://doi.org/https://doi.org/10.1007/s00521-022-08053-z Title Suppressed Due to Excessive Length 5
-
[4]
Nature Medicine30(3), 850–862 (2024).https://doi.org/https://doi.org/ 10.1038/s41591-024-02857-3
Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F.K., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L.L., Wang, J.J., Vaidya, A., Le, L.P., Gerber, G., Sahai, S., Williams, W., Mah- mood, F.: Towards a general-purpose foundation model for computational pathol- ogy. Nature Medicine30(3), 850–862 (20...
-
[5]
Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 785–794. KDD ’16, Association for Computing Machinery (2016). https://doi.org/https://doi.org/10.1145/2939672.2939785
arXiv 2016
-
[6]
arXiv preprint arXiv:1710.05006 (2017)
Codella, N.C.F., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H.: Skin lesion analysis to- ward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collabora- tion (ISIC). arXiv preprint arXiv:1710.050...
arXiv 2017
-
[7]
In: IEEE Conference on Computer Vision and PatternRecognition.pp.248–255(2009)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large- scale hierarchical image database. In: IEEE Conference on Computer Vision and PatternRecognition.pp.248–255(2009). https://doi.org/https://doi.org/10. 1109/CVPR.2009.5206848
arXiv 2009
Show all 20 references
-
[8]
medRxiv (2025)
Dwivedi, K., Mahbod, A., Ecker, R.C., Janjić, K.: A fusion-based multiomics classification approach for enhanced gene discovery in non-small cell lung can- cer. medRxiv (2025). https://doi.org/https://doi.org/10.1101/2025.05.02. 25326847
2025 doi
-
[9]
BMJ357 (2017)
Elmore, J.G., Barnhill, R.L., Elder, D.E., Longton, G.M., Pepe, M.S., Reisch, L.M., Carney, P.A., Titus, L.J., Nelson, H.D., Onega, T., Tosteson, A.N.A., Wein- stock, M.A., Knezevich, S.R., Piepkorn, M.W.: Pathologists’ diagnosis of invasive melanoma and melanocytic proliferat...
2017 doi
-
[10]
Frontiers in Medicine V olume 10 - 2023 (2024)
Furriel, B.C.R.S., Oliveira, B.D., Prôa, R., Paiva, J.Q., Loureiro, R.M., Calixto, W.P., Reis, M.R.C., Giavina-Bianchi, M.: Artificial intelligence for skin cancer detection and classification for clinical environment: a systematic review. Frontiers in Medicine V olume 10 - 20...
2024
-
[11]
(eds.) Bildverar- beitung für die Medizin 2025
Ganz, J., Ammeling, J., Rosbach, E., Lausser, L., Bertram, C.A., Breininger, K., Aubreville, M.: Is self-supervision enough? In: Palm, C., Breininger, K., Deserno, T., Handels, H., Maier, A., Maier-Hein, K.H., Tolxdorff, T.M. (eds.) Bildverar- beitung für die Medizin 2025. pp....
2025 doi
-
[12]
Computer Methods and Programs in Biomedicine193, 105475 (2020)
Mahbod, A., Schaefer, G., Wang, C., Dorffner, G., Ecker, R., Ellinger, I.: Trans- fer learning using a multi-scale and multi-network ensemble for skin lesion clas- sification. Computer Methods and Programs in Biomedicine193, 105475 (2020). https://doi.org/https://doi.org/10.10...
2020
-
[13]
In: International Conference on Pattern Recognition
Mahbod,A.,Schaefer,G.,Wang,C.,Ecker,R.,Dorffner,G.,Ellinger,I.:Investigat- ing and exploiting image resolution for transfer learning-based skin lesion classifi- cation. In: International Conference on Pattern Recognition. pp. 4047–4053 (2021). https://doi.org/https://doi.org/1...
2021
-
[14]
In: International Conference on Acous- tics, Speech and Signal Processing
Mahbod, A., Schaefer, G., Wang, C., Ecker, R., Ellinger, I.: Skin lesion classifica- tion using hybrid deep neural networks. In: International Conference on Acous- tics, Speech and Signal Processing. pp. 1229–1233. IEEE (May 2019). https: //doi.org/https://doi.org/10.1109/ICAS...
2019
-
[15]
Computer Methods and Programs in Biomedicine 197, 105725 (2020)
Mahbod, A., Tschandl, P., Langs, G., Ecker, R., Ellinger, I.: The effects of skin lesion segmentation on the performance of dermatoscopic image classifi- cation. Computer Methods and Programs in Biomedicine 197, 105725 (2020). https://doi.org/https://doi.org/10.1016/j.cmpb.2020.105725
2020
-
[16]
Nature Machine Intelligence6(3), 354–367 (2024)
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T.L., Bernatz, S., Hosny, A., Mak, R.H., Birkbak, N.J., Aerts, H.J.W.L.: Foundation model for cancer imaging biomarkers. Nature Machine Intelligence6(3), 354–367 (2024). https://doi.org/https://doi.org/10.10...
2024 doi
-
[17]
arXiv preprint arXiv:2503.23958 (2025)
Torbati, N., Meshcheryakova, A., Mechtcheriakova, D., Mahbod, A.: A multi-stage auto-context deep learning framework for tissue and nuclei segmentation and clas- sification in h&e-stained histological images of advanced melanoma. arXiv preprint arXiv:2503.23958 (2025)
2025 arXiv
-
[18]
Scien- tific Data 5, 180161 (2018).https://doi.org/https://doi.org/10.1038/sdata
Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scien- tific Data 5, 180161 (2018).https://doi.org/https://doi.org/10.1038/sdata. 2018.161
2018 doi
-
[19]
JAMA Dermatology157(12), 1425–1436 (2021)
Watts, C.G., McLoughlin, K., Goumas, C., van Kemenade, C.H., Aitken, J.F., Soyer,H.P.,FernandezPeñas,P.,Guitera,P.,Scolyer,R.A.,Morton,R.L.,Menzies, S.W., Caruana, M., Kang, Y.J., Mann, G.J., Chakera, A.H., Madronio, C.M., Arm- strong, B.K., Thompson, J.F., Cust, A.E.: Associa...
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.