Pith. sign in

REVIEW 4 major objections 5 minor 20 references

Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Frozen PanDerm features rival fine-tuned Swin Transformer on skin-lesion classification, and fusing their predictions improves accuracy further.

desk verdict Modest but competent benchmark of PanDerm vs fine-tuned ViTs; the central comparison hinges on undisclosed pretraining data and needs error bars. read the letter →

arxiv 2505.16338 v1 pith:2BRC3P4L submitted 2025-05-22 cs.CV

classification cs.CV
keywords SkinLesionClassificationFoundationModelMedicalImageAnalysisDeepLearningFusionDermatoscopyVisionTransformerSwin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a dermatology-specific foundation model's frozen image features can compete with fully fine-tuned Vision Transformers on skin-lesion classification. The authors compare PanDerm embeddings probed with three nonlinear classifiers (MLP, XGBoost, TabNet) against fine-tuned ViT and Swin Transformer V2 base models on the HAM10000 and MSKCC datasets. They find that PanDerm-MLP performs comparably to the fine-tuned Swin model, and that late fusion of PanDerm and Swin predictions yields the top accuracy on both datasets (93.59% on HAM10000, 76.80% on MSKCC). The point matters because it suggests strong dermatoscopic classification can be achieved without expensive fine-tuning of large transformers, and that complementary information in the two model families can be combined.

What carries the argument

The central object is the frozen vision encoder of PanDerm, which converts each dermatoscopic image into an embedding, with nonlinear probing classifiers (MLP, XGBoost, TabNet) trained on top of those fixed features. The comparison arm is the ViT base and Swin Transformer V2 base, both fine-tuned end to end after ImageNet pretraining. The fusion mechanism combines the soft predictions of the best PanDerm probe (MLP) and the best ViT baseline (Swin), exploiting the complementarity between the two model families rather than merging their feature spaces.

What would settle it

Check PanDerm's pretraining data against the official HAM10000 and MSKCC test splits; any overlap of individual images would invalidate the central comparison. Alternatively, rerun the PanDerm-MLP probe on a new dermatoscopic dataset released after PanDerm's training cutoff; if accuracy drops far below the Swin baseline, the frozen-feature advantage is not general.

Watch

Extended reading notes

Core claim

Using frozen PanDerm embeddings with an MLP probe reaches 76.32% accuracy (64.40% balanced) on MSKCC and 92.69% (79.57% balanced) on HAM10000, nearly matching or exceeding the fully fine-tuned Swin Transformer V2 base (75.30% and 64.52% balanced on MSKCC; 91.80% and 81.43% balanced on HAM10000). Fusing PanDerm-MLP and Swin predictions improves accuracy further to 76.80% (64.36% balanced) on MSKCC and 93.59% (83.25% balanced) on HAM10000. The paper concludes that frozen dermatology foundation features are competitive with fine-tuned ViTs and that late fusion of the two gives the best overall accuracy.

Load-bearing premise

The comparison assumes PanDerm's training corpus did not include the HAM10000 or MSKCC test images; if it did, the reported accuracies reflect memorization rather than generalization.

Editorial extensions

If this is right

  • A clinician-oriented system can use frozen PanDerm embeddings with a simple MLP head and reach accuracy comparable to a fully fine-tuned Swin Transformer, avoiding costly per-task fine-tuning of large models.
  • Fusing PanDerm and Swin predictions gives the best accuracy on both datasets, so the two model families encode partially complementary lesion information that a late-fusion step can exploit.
  • On MSKCC, the fusion's balanced accuracy (64.36%) is slightly below PanDerm-MLP (64.40%) and Swin (64.52%), so accuracy gains from fusion do not automatically translate into better class-balanced performance.
  • Across PanDerm probes, MLP outperforms XGBoost and TabNet on both datasets, showing classifier choice materially affects frozen-feature performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If PanDerm's training data is later shown to exclude both benchmarks, this result is a strong argument for caching frozen dermatology embeddings once and training only lightweight probes, which would make model updates in clinical workflows substantially cheaper.
  • A natural next test is per-class analysis of the fusion: since HAM10000 is heavily class-imbalanced, fusion likely concentrates its gain on rare malignant categories, a claim the paper does not test directly.
  • The same frozen-feature plus late-fusion approach could be tested on other dermatology tasks such as segmentation or dermoscopic attribute detection, where foundation-model embeddings may carry useful spatial information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This short paper compares a dermatology-specific foundation model, PanDerm, with two ImageNet-pretrained vision transformers (ViT base and Swin Transformer V2 base) for skin lesion classification. PanDerm is used as a frozen feature extractor and probed with three non-linear classifiers (MLP, XGBoost, TabNet), while the ViT models are fine-tuned end to end. Experiments on HAM10000 and MSKCC use the official train/validation/test splits from reference [20]. The main claims are that the PanDerm-MLP model performs comparably to the fine-tuned Swin model, and that fusing PanDerm-MLP and Swin predictions improves accuracy. Results are reported in Table 1 as accuracy and balanced accuracy.

Significance. If the comparison is methodologically clean, the result is practically useful: it suggests that probing a dermatology-specific foundation model can match fine-tuning a general-purpose vision transformer, and that simple score fusion gives a further gain. The paper has several strengths: it uses the official splits from the PanDerm paper, reports both accuracy and balanced accuracy, and covers two public datasets. However, the significance is currently limited by the lack of statistical uncertainty quantification, by incomplete methodological disclosure (fusion procedure and hyperparameters), and by an unresolved question about whether PanDerm's pretraining data overlaps the evaluation test splits. The balanced-accuracy results also complicate the headline accuracy-based claims.

major comments (4)
  1. [Section 2 (Datasets and PanDerm representation)] The central comparison is fair only if PanDerm's pretraining corpus is disjoint from the HAM10000 and MSKCC test splits. The paper states that the official splits from reference [20] are used, but it never discloses what PanDerm was trained on. Because HAM10000 and MSKCC are public ISIC-era datasets and PanDerm is a dermatology-specific foundation model, overlap is plausible. The authors should state explicitly whether the pretraining data in [20] includes any of these images and, ideally, verify that the test images are not in the training set. Without this disclosure, the reported PanDerm-vs-Swin comparison could reflect memorization rather than generalization.
  2. [Table 1 and Section 3] The claim that PanDerm-MLP performs comparably to Swin is metric-dependent. On HAM10000, accuracy favors PanDerm-MLP (92.69% vs 91.80%), but balanced accuracy favors Swin (81.43% vs 79.57%). On MSKCC, the balanced-accuracy values are essentially tied (64.40% vs 64.52%). No confidence intervals or significance tests are provided, so it is unclear whether the reported differences are meaningful. Given the large gap between accuracy and balanced accuracy on HAM10000, the paper should justify why accuracy is treated as the primary metric and should report uncertainty or error bars for both metrics.
  3. [Section 3 and Table 1 (fusion row)] The fusion method is not specified. The last row of Table 1 is described only as combining predictions from the best ViT model and the best PanDerm-based model, but no equation, weighting scheme, or averaging procedure is given. Without this detail, the fusion result cannot be reproduced, and it is unclear whether the fusion gain is robust or an artifact of a particular combination rule.
  4. [Section 2 (training and probing details)] Hyperparameters for the MLP hidden layer and dropout, XGBoost, TabNet, and the fine-tuning schedules for ViT and Swin are omitted. No random seeds or multiple runs are reported. These omissions prevent reproducibility and make it impossible to assess whether the small differences in Table 1 are stable across runs.
minor comments (5)
  1. [Table 1] The entry 'Panderm_TabNet' has inconsistent capitalization; it should be 'PanDerm_TabNet' for consistency with the other rows.
  2. [Section 2] The preprocessing and input resolution for PanDerm embeddings are not described; if PanDerm uses a different input size or normalization than the ViT models, this should be stated.
  3. [Figure 1] The example images are not discussed in the text; a sentence describing what they illustrate would help the reader.
  4. [Section 2 (splits)] The manuscript says it follows the official splits from [20], but it does not specify which exact split indices or preprocessing steps were used; this should be clarified for reproducibility.
  5. [Title and Section 3] The title emphasizes fusion, but the fusion details appear only in the results section; moving a precise description of the fusion procedure to the Method section would improve the paper's structure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark comparing external model checkpoints on standard splits; the undisclosed PanDerm pretraining corpus is a data-leakage risk, not a circular derivation.

full rationale

The paper's claims are supported by direct measurements reported in Table 1: frozen PanDerm features are classified with MLP, XGBoost, and TabNet, and ViT/Swin baselines are fine-tuned on the same data. There is no fitted parameter renamed as a prediction, no theoretical derivation that reduces to its own inputs, and no uniqueness claim imported from the authors' prior work. The authors cite Yan et al. [20] for the model and the official splits, but [20] is an external source, not a self-citation. The self-citations present (e.g., [8], [12]-[15]) concern methodological context and are not load-bearing. The only substantive concern is that PanDerm's pretraining data is not disclosed, so one cannot rule out overlap with the HAM10000 or MSKCC test sets. If such overlap existed, the comparison would measure memorization rather than generalization. However, this is a data-hygiene and external-validity risk, not an internal circularity that can be exhibited from the paper's equations or definitions. Per the specified rules, speculation about undisclosed pretraining data cannot raise the circularity score. The benchmark is self-contained with respect to the stated evaluation protocol, so the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new model components, particles, or theoretical entities. All assumptions are standard empirical ML choices, except the unstated data-hygiene assumption about PanDerm's pre-training corpus, which is the most consequential.

free parameters (5)
  • MLP hidden layer size
    Not specified in Section 2; directly affects PanDerm-MLP results.
  • MLP dropout rate
    Not specified; affects regularization and test accuracy.
  • XGBoost hyperparameters
    Number of trees, tree depth, and learning rate are not reported.
  • TabNet hyperparameters
    TabNet settings are not reported.
  • Fine-tuning schedule for ViT and Swin
    Learning rate, epochs, optimizer, and augmentation magnitudes are not reported.
assumptions (4)
  • domain assumption PanDerm was not pre-trained on the HAM10000 or MSKCC test images.
    Required for a fair comparison; not stated in the paper. Section 2 uses the frozen PanDerm encoder and the official splits from [20].
  • domain assumption The official train/validation/test splits from [20] are valid and were not used to fit the classifiers in a way that leaks test information.
    The paper trusts splits from the PanDerm paper [20] without independent verification.
  • domain assumption ImageNet statistics are appropriate for normalizing dermoscopic images for all models.
    Section 2 states that all images are resized to 224x224 and normalized with ImageNet mean and standard deviation, even for the dermatology-specific PanDerm.
  • domain assumption Full fine-tuning of the ViT models yields better performance than probing, as cited from [11].
    Used to justify fine-tuning rather than probing for the ViT baseline models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification." pith.science (2026). https://pith.science/paper/2BRC3P4L

@misc{pith2026250516338,
  author       = {Pith},
  title        = {Pith review of: Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BRC3P4L}},
  note         = {Machine review of arXiv:2505.16338}
}
read the original abstract

Accurate classification of skin lesions from dermatoscopic images is essential for diagnosis and treatment of skin cancer. In this study, we investigate the utility of a dermatology-specific foundation model, PanDerm, in comparison with two Vision Transformer (ViT) architectures (ViT base and Swin Transformer V2 base) for the task of skin lesion classification. Using frozen features extracted from PanDerm, we apply non-linear probing with three different classifiers, namely, multi-layer perceptron (MLP), XGBoost, and TabNet. For the ViT-based models, we perform full fine-tuning to optimize classification performance. Our experiments on the HAM10000 and MSKCC datasets demonstrate that the PanDerm-based MLP model performs comparably to the fine-tuned Swin transformer model, while fusion of PanDerm and Swin Transformer predictions leads to further performance improvements. Future work will explore additional foundation models, fine-tuning strategies, and advanced fusion techniques.

Figures

Figures reproduced from arXiv: 2505.16338 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages

  1. [20]

    arXiv preprint arXiv:2410.15038 (2024)

    Yan, S., Yu, Z., Primiero, C., Vico-Alonso, C., Wang, Z., Yang, L., Tschandl, P., Hu, M., Tan, G., Tang, V., et al.: A general-purpose multimodal foundation model for dermatology. arXiv preprint arXiv:2410.15038 (2024)

  2. [1]

    medRxiv (2025)

    Adebiyi, A., Abdalnabi, N., Simoes, E.J., Becevic, M., Smith, E.H., Rao, P.: Trans- formers in skin lesion classification and diagnosis: A systematic review. medRxiv (2025). https://doi.org/https://doi.org/10.1101/2024.09.19.24314004

  3. [2]

    Proceed- ings of the AAAI Conference on Artificial Intelligence35(8), 6679–6687 (2021)

    Arik, S.O., Pfister, T.: TabNet: Attentive interpretable tabular learning. Proceed- ings of the AAAI Conference on Artificial Intelligence35(8), 6679–6687 (2021). https://doi.org/https://doi.org/10.1609/aaai.v35i8.16826

  4. [3]

    Neural Computing and Applications35(9), 6713–6722 (2023)

    Ayas, S.: Multiclass skin lesion classification in dermoscopic images using swin transformer model. Neural Computing and Applications35(9), 6713–6722 (2023). https://doi.org/https://doi.org/10.1007/s00521-022-08053-z Title Suppressed Due to Excessive Length 5

  5. [4]

    Nature Medicine30(3), 850–862 (2024).https://doi.org/https://doi.org/ 10.1038/s41591-024-02857-3

    Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F.K., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L.L., Wang, J.J., Vaidya, A., Le, L.P., Gerber, G., Sahai, S., Williams, W., Mah- mood, F.: Towards a general-purpose foundation model for computational pathol- ogy. Nature Medicine30(3), 850–862 (20...

  6. [5]

    In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 785–794. KDD ’16, Association for Computing Machinery (2016). https://doi.org/https://doi.org/10.1145/2939672.2939785

  7. [6]

    arXiv preprint arXiv:1710.05006 (2017)

    Codella, N.C.F., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H.: Skin lesion analysis to- ward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collabora- tion (ISIC). arXiv preprint arXiv:1710.050...

  8. [7]

    In: IEEE Conference on Computer Vision and PatternRecognition.pp.248–255(2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large- scale hierarchical image database. In: IEEE Conference on Computer Vision and PatternRecognition.pp.248–255(2009). https://doi.org/https://doi.org/10. 1109/CVPR.2009.5206848

Show all 20 references
  1. [8]

    medRxiv (2025)

    Dwivedi, K., Mahbod, A., Ecker, R.C., Janjić, K.: A fusion-based multiomics classification approach for enhanced gene discovery in non-small cell lung can- cer. medRxiv (2025). https://doi.org/https://doi.org/10.1101/2025.05.02. 25326847

  2. [9]

    BMJ357 (2017)

    Elmore, J.G., Barnhill, R.L., Elder, D.E., Longton, G.M., Pepe, M.S., Reisch, L.M., Carney, P.A., Titus, L.J., Nelson, H.D., Onega, T., Tosteson, A.N.A., Wein- stock, M.A., Knezevich, S.R., Piepkorn, M.W.: Pathologists’ diagnosis of invasive melanoma and melanocytic proliferat...

  3. [10]

    Frontiers in Medicine V olume 10 - 2023 (2024)

    Furriel, B.C.R.S., Oliveira, B.D., Prôa, R., Paiva, J.Q., Loureiro, R.M., Calixto, W.P., Reis, M.R.C., Giavina-Bianchi, M.: Artificial intelligence for skin cancer detection and classification for clinical environment: a systematic review. Frontiers in Medicine V olume 10 - 20...

  4. [11]

    (eds.) Bildverar- beitung für die Medizin 2025

    Ganz, J., Ammeling, J., Rosbach, E., Lausser, L., Bertram, C.A., Breininger, K., Aubreville, M.: Is self-supervision enough? In: Palm, C., Breininger, K., Deserno, T., Handels, H., Maier, A., Maier-Hein, K.H., Tolxdorff, T.M. (eds.) Bildverar- beitung für die Medizin 2025. pp....

  5. [12]

    Computer Methods and Programs in Biomedicine193, 105475 (2020)

    Mahbod, A., Schaefer, G., Wang, C., Dorffner, G., Ecker, R., Ellinger, I.: Trans- fer learning using a multi-scale and multi-network ensemble for skin lesion clas- sification. Computer Methods and Programs in Biomedicine193, 105475 (2020). https://doi.org/https://doi.org/10.10...

  6. [13]

    In: International Conference on Pattern Recognition

    Mahbod,A.,Schaefer,G.,Wang,C.,Ecker,R.,Dorffner,G.,Ellinger,I.:Investigat- ing and exploiting image resolution for transfer learning-based skin lesion classifi- cation. In: International Conference on Pattern Recognition. pp. 4047–4053 (2021). https://doi.org/https://doi.org/1...

  7. [14]

    In: International Conference on Acous- tics, Speech and Signal Processing

    Mahbod, A., Schaefer, G., Wang, C., Ecker, R., Ellinger, I.: Skin lesion classifica- tion using hybrid deep neural networks. In: International Conference on Acous- tics, Speech and Signal Processing. pp. 1229–1233. IEEE (May 2019). https: //doi.org/https://doi.org/10.1109/ICAS...

  8. [15]

    Computer Methods and Programs in Biomedicine 197, 105725 (2020)

    Mahbod, A., Tschandl, P., Langs, G., Ecker, R., Ellinger, I.: The effects of skin lesion segmentation on the performance of dermatoscopic image classifi- cation. Computer Methods and Programs in Biomedicine 197, 105725 (2020). https://doi.org/https://doi.org/10.1016/j.cmpb.2020.105725

  9. [16]

    Nature Machine Intelligence6(3), 354–367 (2024)

    Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T.L., Bernatz, S., Hosny, A., Mak, R.H., Birkbak, N.J., Aerts, H.J.W.L.: Foundation model for cancer imaging biomarkers. Nature Machine Intelligence6(3), 354–367 (2024). https://doi.org/https://doi.org/10.10...

  10. [17]

    arXiv preprint arXiv:2503.23958 (2025)

    Torbati, N., Meshcheryakova, A., Mechtcheriakova, D., Mahbod, A.: A multi-stage auto-context deep learning framework for tissue and nuclei segmentation and clas- sification in h&e-stained histological images of advanced melanoma. arXiv preprint arXiv:2503.23958 (2025)

  11. [18]

    Scien- tific Data 5, 180161 (2018).https://doi.org/https://doi.org/10.1038/sdata

    Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scien- tific Data 5, 180161 (2018).https://doi.org/https://doi.org/10.1038/sdata. 2018.161

  12. [19]

    JAMA Dermatology157(12), 1425–1436 (2021)

    Watts, C.G., McLoughlin, K., Goumas, C., van Kemenade, C.H., Aitken, J.F., Soyer,H.P.,FernandezPeñas,P.,Guitera,P.,Scolyer,R.A.,Morton,R.L.,Menzies, S.W., Caruana, M., Kang, Y.J., Mann, G.J., Chakera, A.H., Madronio, C.M., Arm- strong, B.K., Thompson, J.F., Cust, A.E.: Associa...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.