REVIEW 4 major objections 5 minor 23 references
Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A transfer-learned EfficientNetV2L reaches 0.8490 accuracy and 0.9768 AUC on the DermaMNIST-C skin-lesion benchmark, matching or beating earlier published classifiers.
desk verdict A careful but very incremental transfer-learning benchmark on DermaMNIST-C whose 'match or surpass' claim rests on a 0.002 accuracy gap with no error bars or code; plausible but unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the result is transfer learning: an ImageNet-pretrained backbone with frozen weights, topped by a custom classification head that flattens the features, applies batch normalization, a dense layer of 64 or 128 units, dropout, and a final seven-unit softmax. The winning Effv3 variant uses the 64-unit dense head. Training employs categorical cross-entropy with Adam at learning rate 1e-4, early stopping, and model checkpointing, and the base model is never fine-tuned. The DermaMNIST-C dataset, a 224x224 reorganized version of DermaMNIST, supplies the images and split that make the higher accuracy possible.
What would settle it
Re-run the Effv3 configuration and the ResNet-50 baseline from [14] on the same DermaMNIST-C split across 10 random seeds and compare the distributions of accuracy and AUC; if the confidence intervals for the two models overlap around the 0.8490 and 0.851 values, the claim that the configuration matches or surpasses existing methods is not supported.
Extended reading notes
Core claim
The paper's central claim, stated in its own terms, is that EfficientNetV2L with frozen ImageNet weights and a simple custom head (the Effv3 configuration) achieves an accuracy of 0.8490 on DermaMNIST-C, with precision 0.8935, AUC 0.9768, and recall 0.8239, and that this matches or surpasses previously published classifiers on this benchmark. In the comparison table the accuracy sits slightly below the 0.851 of the ResNet-50 baseline reported in the DermaMNIST-C paper, but above every other listed method. The authors argue that accuracy alone does not capture precision, AUC, or recall, where their configuration is competitive, and they read the result as evidence that a strong pretrained feature extractor plus a small trainable head is a viable path to improving diagnostic accuracy for pigmented skin lesions.
Load-bearing premise
The load-bearing assumption is that one training run on the fixed official split gives a stable, trustworthy accuracy estimate, so differences like 0.8490 versus 0.851 can be treated as real even though no repeated seeds, confidence intervals, or significance tests are reported.
Editorial extensions
If this is right
- A transfer-learned EfficientNetV2L with a lightweight head is enough to reach 0.8490 accuracy and 0.9768 AUC on DermaMNIST-C, so complex bespoke architectures are not required for this benchmark.
- The same model on the original 28x28 DermaMNIST reaches only 0.7017 accuracy, making the 224x224 DermaMNIST-C version the more meaningful testbed for lesion classification.
- Class imbalance continues to matter: adding class weights in the Effv4 configuration did not improve accuracy or recall over the unweighted Effv3 configuration.
- The high AUC of 0.9768 suggests the model ranks lesion classes nearly correctly even where exact hard-label accuracy is imperfect, which is useful for a screening or triage tool.
Reading between the lines
- Extrapolating beyond the paper: the 0.8490-versus-0.851 accuracy gap is far smaller than typical seed-to-seed variation for CNNs of this size, so the defensible reading of the headline result is “statistically indistinguishable from the leading baseline” until variances are reported.
- The paper freezes the backbone throughout; an untested corollary is that unfreezing the final blocks with a lower learning rate could push accuracy above 0.851, but the paper does not claim this.
- The same transfer-learning recipe could be applied to full-resolution HAM10000 images or other dermatology benchmarks; the paper's own logic predicts that resolution and label cleaning matter more than model choice, and that is a testable extension it does not perform.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates transfer learning with ResNet-50 and EfficientNetV2L on the DermaMNIST and DermaMNIST-C datasets. In Experiment 1, both models are trained on DermaMNIST with an identical convolutional head; in Experiment 2, five configurations are compared on DermaMNIST-C, including a from-scratch simple model and four EfficientNetV2L variants. The best reported configuration, Effv3 e, achieves accuracy 0.8490, precision 0.8935, AUC 0.9768, and recall 0.8239 on DermaMNIST-C. The authors compare this against published methods and claim that one configuration matches or surpasses existing methods.
Significance. If the central benchmark result is substantiated, a simple frozen-backbone EfficientNetV2L with a small dense head would be a useful, low-cost baseline for DermaMNIST-C, and the comparison with reference [14] is a falsifiable empirical claim. However, the paper does not release code, random seeds, repeated-run statistics, or a validation-based model selection protocol, so the current evidence is insufficient to establish the match-or-surpass claim. The significance is therefore moderate and mainly empirical; the manuscript would be strengthened substantially by multi-seed evaluation and explicit metric definitions.
major comments (4)
- [§6, Tables 2–3, Abstract] The central claim that one configuration "matches or surpasses existing methods" is not supported by the reported evidence. Effv3 e's accuracy is 0.8490 against 0.851 for reference [14] (Table 3), a gap of 0.002, and Tables 1–2 report a single run per configuration with no variance, confidence intervals, or significance testing. Moreover, the paper's argument that "higher accuracy does not reflect to higher values in other metrics" cannot be checked because precision, AUC, and recall for the [14] comparator are not reported. Please report multi-seed mean±std or confidence intervals, and either report the comparator's other metrics or restrict the claim to accuracy comparability.
- [§5.2, Table 2] Effv3 e is presented as the best of five configurations, but no validation-based selection protocol is described. If the test set was used to choose among SM, Effv1 e, Effv2 e, Effv3 e, and Effv4 e, the reported 0.8490 is a best-of-five test statistic and is optimistically biased relative to a single model selected before seeing the test set. The paper should state explicitly which split was used for configuration selection, or evaluate the selected model on a separate held-out test fold.
- [§4, Eq. (4) and Eqs. (1)–(3), Tables 1–2] AUC is computed with a binary rank-sum formula in Eq. (4), yet the task has seven classes; the reported AUC of 0.9768 is undefined without specifying the multi-class protocol (e.g., one-vs-rest macro-averaging). Similarly, precision and recall in Eqs. (1)–(3) are binary definitions, and the reported values depend on whether macro-, micro-, or class-weighted averaging is used. Please specify these protocols; otherwise the precision, AUC, and recall columns in Tables 1–2 cannot be reproduced or compared meaningfully.
- [Table 3, §6] Table 3 mixes models evaluated on DermaMNIST (rows [11], [17]–[23]) with models evaluated on DermaMNIST-C (row [14] and this work). The "match or surpass existing methods" claim is therefore overbroad: on DermaMNIST-C, only reference [14] is a direct comparator, and the difference is within sampling noise as argued above. The claim should be scoped to the DermaMNIST-C benchmark and to that specific comparator.
minor comments (5)
- [§2] The sentence "Images are split into training, validation, and test sets in the range of 70:10:20%" is vague; specify exact sample counts for DermaMNIST and DermaMNIST-C.
- [§4] In the sentence before Eq. (4), "AUC is calculated by math Eq. (4)" contains a stray "math"; please correct the typo.
- [§5.1] The discussion states that overfitting occurred, but Table 1 alone does not demonstrate overfitting; clarify whether the Loss column is test-set loss and, if possible, report training/validation curves or the train-test gap to support the claim.
- [Table 3] The row citing [17] attributes GAutoML Vision to a tuberculosis-screening paper; please verify the citation, as it appears to be mismatched.
- [General] No code or data availability statement is provided; for a benchmark comparison, releasing training code and random seeds is important for reproducibility.
Circularity Check
No significant circularity: all reported metrics are measured on an external benchmark and none of the central claims reduce to a fitted parameter or self-citation.
full rationale
The paper's claims are empirical evaluations of CNN classifiers on the public DermaMNIST and DermaMNIST-C datasets, compared against published results. The central claim that EfficientNetV2L 'match or surpass existing methods' is a comparison of measured test-set accuracy (0.8490) against an external published number (0.851 from Ref. [14]); this is not a derivation from fitted inputs. The metrics in Eqs. (1)-(4) are standard definitions, and although Eq. (1)-(3) cite a paper co-authored by one of the present authors (Ref. [15]), this citation is only for the textbook precision/accuracy/recall formulas and is not load-bearing for any prediction. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity; no uniqueness theorem is imported; no ansatz is smuggled in via self-citation. The skeptical concerns about single-run variance, lack of error bars, and test-set-based model selection are legitimate threats to the strength of the empirical comparison, but they are correctness/robustness issues, not circularity. The paper's own limitations (e.g., class imbalance, overfitting) are acknowledged. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Learning rate for Adam optimizer =
1e-4
- Dense head width =
64 or 128 units
- Conv2D filter counts in Experiment 1 head =
32, 64, 128
- Class weights in Effv4 =
not specified
- Dropout rate and early stopping patience =
not specified
assumptions (3)
- domain assumption ImageNet pretrained weights provide useful features for dermatoscopic images at 28x28 and 224x224 resolution.
- ad hoc to paper The binary AUC formula in Eq. (4) can be used for a seven-class problem.
- domain assumption The DermaMNIST-C labels and fixed splits are correct and suitable for comparison.
Cite this review
Pith. "Pith review of Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset." pith.science (2026). https://pith.science/paper/APILOLS7
@misc{pith2026250712961,
author = {Pith},
title = {Pith review of: Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/APILOLS7}},
note = {Machine review of arXiv:2507.12961}
}
read the original abstract
Pigmented skin lesions represent localized areas of increased melanin and can indicate serious conditions like melanoma, a major contributor to skin cancer mortality. The MedMNIST v2 dataset, inspired by MNIST, was recently introduced to advance research in biomedical imaging and includes DermaMNIST, a dataset for classifying pigmented lesions based on the HAM10000 dataset. This study assesses ResNet-50 and EfficientNetV2L models for multi-class classification using DermaMNIST, employing transfer learning and various layer configurations. One configuration achieves results that match or surpass existing methods. This study suggests that convolutional neural networks (CNNs) can drive progress in biomedical image analysis, significantly enhancing diagnostic accuracy.
Figures
Reference graph
Works this paper leans on
-
[14]
Investigating the quality of dermamnist and fitzpatrick17k dermatological image datasets,
Kumar Abhishek, Aditi Jain, and Ghassan Hamarneh, “Investigating the quality of dermamnist and fitzpatrick17k dermatological image datasets,” 2024
work page 2024
-
[11]
Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis,
Jiancheng Yang, Rui Shi, and Bingbing Ni, “Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis,” in IEEE 18th Inter- national Symposium on Biomedical Imaging (ISBI) , 2021, pp. 191–195
work page 2021
-
[17]
Khin Yadanar Win and et al., “Hybrid learning of hand-crafted and deep-activated features using particle swarm optimization and optimized support vector machine for tuberculosis screening,” Applied Sciences, vol. 10, no. 17, pp. 5749, 2020. Title Suppressed Due to Excessive Length 9
work page 2020
-
[23]
Peng Jiang, Juan Liu, Lang Wang, Zhihui Ynag, Hongyu Dong, and Jing Feng, “Deeply supervised layer selective attention network: Towards label-efficient learn- ing for medical image classification,” arXiv preprint arXiv:2209.13844 , 2022
work page Pith review arXiv 2022
-
[1]
Christina Ring, Nathan Cox, and Jason B Lee, “Dermatoscopy,” Clinics in Der- matology, vol. 39, no. 4, pp. 635–642, 2021
work page 2021
-
[2]
Josep Malvehy, Ralph P Braun, Susana Puig, Ashfaq A Marghoob, and Alfred W Kopf, Handbook of dermoscopy, CRC Press, 2006
work page 2006
-
[3]
Rona M MacKie, Skin cancer, CRC Press, 1996
work page 1996
-
[4]
Taxiarchis Konstantinos Nikolouzakis, Luca Falzone, Konstantinos Lasithiotakis, Sabine Kr¨ uger-Krasagakis, Alexandra Kalogeraki, Maria Sifaki, Demetrios A Span- didos, Emmanuel Chrysos, Aristidis Tsatsakis, and John Tsiaoussis, “Current and future trends in molecular biomarkers for diagnostic, prognostic, and predictive purposes in non-melanoma skin canc...
work page 2020
Show all 23 references
-
[5]
Ashfaq A Marghoob, Ralph Peter Braun, and Alfred W Kopf, Atlas of dermoscopy, Taylor & Francis London, 2005
2005
-
[6]
Karen L Agnew, Christopher B Bunker, and Sarah T Arron, Fast Facts: Skin Cancer, Karger Medical and Scientific Publishers, 2015
2015
-
[7]
A gan-based image synthe- sis method for skin lesion classification,
Zhiwei Qin, Zhao Liu, Ping Zhu, and Yongbo Xue, “A gan-based image synthe- sis method for skin lesion classification,” Computer Methods and Programs in Biomedicine, vol. 195, pp. 105568, 2020
2020
-
[8]
Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni, “Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,” Scientific Data , vol. 10, no. 1, pp. 41, 2023
2023
-
[9]
The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,
Philipp Tschandl, “The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” 2018
2018
-
[10]
Applications of support vector machine (svm) learning in cancer genomics,
Shujun Huang, Nianguang Cai, Pedro Penzuti Pacheco, Shavira Narrandes, Yang Wang, and Wayne Xu, “Applications of support vector machine (svm) learning in cancer genomics,” Cancer genomics & proteomics , vol. 15, no. 1, pp. 41–51, 2018
2018
-
[12]
The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific data , p. 180161, 2018
2018
-
[13]
Skin lesion analysis toward melanoma detection 2018: A chal- lenge hosted by the international skin imaging collaboration (isic),
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al., “Skin lesion analysis toward melanoma detection 2018: A chal- lenge hosted by the international skin imag...
2018 arXiv
-
[15]
Unsupervised method for wildfire flame segmen- tation and detection,
Emir Buza and Amila Akagic, “Unsupervised method for wildfire flame segmen- tation and detection,” IEEE Access, vol. 10, pp. 55213–55225, 2022
2022
-
[16]
A review on evaluation metrics for data classification evaluations,
Mohammad Hossin and Md Nasir Sulaiman, “A review on evaluation metrics for data classification evaluations,” International journal of data mining & knowledge management process, vol. 5, no. 2, pp. 1, 2015
2015
-
[18]
Attentive octave convolutional capsule network for medical image classification,
Hong Zhang, Zhengzhen Li, Hao Zhao, Zan Li, and Yanping Zhang, “Attentive octave convolutional capsule network for medical image classification,” Applied Sciences, vol. 12, no. 5, 2022
2022
-
[19]
Extreme learning machine for biomedical image classification: A multi- case study,
Francesco Mercaldo, Luca Brunese, Antonella Santone, Fabio Martinelli, and Mario Cesarelli, “Extreme learning machine for biomedical image classification: A multi- case study,” EAI Endorsed Transactions on Pervasive Health and Technology , vol. 10, 2024
2024
-
[20]
Skin lesion clas- sification based on hybrid self-supervised pretext task,
Dedong Yang, Jianwen Zhang, Yangyang Li, and Zhiquan Ling, “Skin lesion clas- sification based on hybrid self-supervised pretext task,” International Journal of Imaging Systems and Technology , vol. 34, no. 2, pp. e23058, 2024
2024
-
[21]
Medrdf: a robust and retrain-less diagnostic framework for medical pretrained models against adversarial attack,
Mengting Xu, Tao Zhang, and Daoqiang Zhang, “Medrdf: a robust and retrain-less diagnostic framework for medical pretrained models against adversarial attack,” IEEE Transactions on Medical Imaging , vol. 41, no. 8, pp. 2130–2143, 2022
2022
-
[22]
Failure detection in deep neural networks for medical imaging,
Sabeen Ahmed, Dimah Dera, Saud Ul Hassan, Nidhal Bouaynaya, and Ghulam Rasool, “Failure detection in deep neural networks for medical imaging,” Frontiers in Medical Technology, vol. 4, pp. 919046, 2022
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.