Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A transfer-learned EfficientNetV2L reaches 0.8490 accuracy and 0.9768 AUC on the DermaMNIST-C skin-lesion benchmark, matching or beating earlier published classifiers.

desk verdict A careful but very incremental transfer-learning benchmark on DermaMNIST-C whose 'match or surpass' claim rests on a 0.002 accuracy gap with no error bars or code; plausible but unproven. read the letter →

arxiv 2507.12961 v1 pith:APILOLS7 submitted 2025-07-17 eess.IV cs.AIcs.CV

classification eess.IVcs.AIcs.CV
keywords DermaMNISTMNIST-CskinlesionclassificationtransferlearningEfficientNetV2LResNet-50medicalimageconvolutionalneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a transfer-learned convolutional network can classify pigmented skin lesions accurately on the DermaMNIST and DermaMNIST-C benchmarks, with one configuration matching or beating previously reported accuracy. The best configuration is an ImageNet-pretrained EfficientNetV2L with a small dense classification head, which the paper reports at 0.8490 accuracy, 0.9768 AUC, and 0.8239 recall on DermaMNIST-C. If true, it would mean reliable automated screening of dermatoscopic images is achievable with a standard pretrained backbone and a lightweight head rather than a purpose-built architecture. The study also indicates that dataset quality and resolution are decisive: the same approach reaches only 0.7017 accuracy on the original 28x28 DermaMNIST.

What carries the argument

The mechanism that carries the result is transfer learning: an ImageNet-pretrained backbone with frozen weights, topped by a custom classification head that flattens the features, applies batch normalization, a dense layer of 64 or 128 units, dropout, and a final seven-unit softmax. The winning Effv3 variant uses the 64-unit dense head. Training employs categorical cross-entropy with Adam at learning rate 1e-4, early stopping, and model checkpointing, and the base model is never fine-tuned. The DermaMNIST-C dataset, a 224x224 reorganized version of DermaMNIST, supplies the images and split that make the higher accuracy possible.

What would settle it

Re-run the Effv3 configuration and the ResNet-50 baseline from [14] on the same DermaMNIST-C split across 10 random seeds and compare the distributions of accuracy and AUC; if the confidence intervals for the two models overlap around the 0.8490 and 0.851 values, the claim that the configuration matches or surpasses existing methods is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its own terms, is that EfficientNetV2L with frozen ImageNet weights and a simple custom head (the Effv3 configuration) achieves an accuracy of 0.8490 on DermaMNIST-C, with precision 0.8935, AUC 0.9768, and recall 0.8239, and that this matches or surpasses previously published classifiers on this benchmark. In the comparison table the accuracy sits slightly below the 0.851 of the ResNet-50 baseline reported in the DermaMNIST-C paper, but above every other listed method. The authors argue that accuracy alone does not capture precision, AUC, or recall, where their configuration is competitive, and they read the result as evidence that a strong pretrained feature extractor plus a small trainable head is a viable path to improving diagnostic accuracy for pigmented skin lesions.

Load-bearing premise

The load-bearing assumption is that one training run on the fixed official split gives a stable, trustworthy accuracy estimate, so differences like 0.8490 versus 0.851 can be treated as real even though no repeated seeds, confidence intervals, or significance tests are reported.

Editorial extensions

If this is right

  • A transfer-learned EfficientNetV2L with a lightweight head is enough to reach 0.8490 accuracy and 0.9768 AUC on DermaMNIST-C, so complex bespoke architectures are not required for this benchmark.
  • The same model on the original 28x28 DermaMNIST reaches only 0.7017 accuracy, making the 224x224 DermaMNIST-C version the more meaningful testbed for lesion classification.
  • Class imbalance continues to matter: adding class weights in the Effv4 configuration did not improve accuracy or recall over the unweighted Effv3 configuration.
  • The high AUC of 0.9768 suggests the model ranks lesion classes nearly correctly even where exact hard-label accuracy is imperfect, which is useful for a screening or triage tool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extrapolating beyond the paper: the 0.8490-versus-0.851 accuracy gap is far smaller than typical seed-to-seed variation for CNNs of this size, so the defensible reading of the headline result is “statistically indistinguishable from the leading baseline” until variances are reported.
  • The paper freezes the backbone throughout; an untested corollary is that unfreezing the final blocks with a lower learning rate could push accuracy above 0.851, but the paper does not claim this.
  • The same transfer-learning recipe could be applied to full-resolution HAM10000 images or other dermatology benchmarks; the paper's own logic predicts that resolution and label cleaning matter more than model choice, and that is a testable extension it does not perform.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates transfer learning with ResNet-50 and EfficientNetV2L on the DermaMNIST and DermaMNIST-C datasets. In Experiment 1, both models are trained on DermaMNIST with an identical convolutional head; in Experiment 2, five configurations are compared on DermaMNIST-C, including a from-scratch simple model and four EfficientNetV2L variants. The best reported configuration, Effv3 e, achieves accuracy 0.8490, precision 0.8935, AUC 0.9768, and recall 0.8239 on DermaMNIST-C. The authors compare this against published methods and claim that one configuration matches or surpasses existing methods.

Significance. If the central benchmark result is substantiated, a simple frozen-backbone EfficientNetV2L with a small dense head would be a useful, low-cost baseline for DermaMNIST-C, and the comparison with reference [14] is a falsifiable empirical claim. However, the paper does not release code, random seeds, repeated-run statistics, or a validation-based model selection protocol, so the current evidence is insufficient to establish the match-or-surpass claim. The significance is therefore moderate and mainly empirical; the manuscript would be strengthened substantially by multi-seed evaluation and explicit metric definitions.

major comments (4)
  1. [§6, Tables 2–3, Abstract] The central claim that one configuration "matches or surpasses existing methods" is not supported by the reported evidence. Effv3 e's accuracy is 0.8490 against 0.851 for reference [14] (Table 3), a gap of 0.002, and Tables 1–2 report a single run per configuration with no variance, confidence intervals, or significance testing. Moreover, the paper's argument that "higher accuracy does not reflect to higher values in other metrics" cannot be checked because precision, AUC, and recall for the [14] comparator are not reported. Please report multi-seed mean±std or confidence intervals, and either report the comparator's other metrics or restrict the claim to accuracy comparability.
  2. [§5.2, Table 2] Effv3 e is presented as the best of five configurations, but no validation-based selection protocol is described. If the test set was used to choose among SM, Effv1 e, Effv2 e, Effv3 e, and Effv4 e, the reported 0.8490 is a best-of-five test statistic and is optimistically biased relative to a single model selected before seeing the test set. The paper should state explicitly which split was used for configuration selection, or evaluate the selected model on a separate held-out test fold.
  3. [§4, Eq. (4) and Eqs. (1)–(3), Tables 1–2] AUC is computed with a binary rank-sum formula in Eq. (4), yet the task has seven classes; the reported AUC of 0.9768 is undefined without specifying the multi-class protocol (e.g., one-vs-rest macro-averaging). Similarly, precision and recall in Eqs. (1)–(3) are binary definitions, and the reported values depend on whether macro-, micro-, or class-weighted averaging is used. Please specify these protocols; otherwise the precision, AUC, and recall columns in Tables 1–2 cannot be reproduced or compared meaningfully.
  4. [Table 3, §6] Table 3 mixes models evaluated on DermaMNIST (rows [11], [17]–[23]) with models evaluated on DermaMNIST-C (row [14] and this work). The "match or surpass existing methods" claim is therefore overbroad: on DermaMNIST-C, only reference [14] is a direct comparator, and the difference is within sampling noise as argued above. The claim should be scoped to the DermaMNIST-C benchmark and to that specific comparator.
minor comments (5)
  1. [§2] The sentence "Images are split into training, validation, and test sets in the range of 70:10:20%" is vague; specify exact sample counts for DermaMNIST and DermaMNIST-C.
  2. [§4] In the sentence before Eq. (4), "AUC is calculated by math Eq. (4)" contains a stray "math"; please correct the typo.
  3. [§5.1] The discussion states that overfitting occurred, but Table 1 alone does not demonstrate overfitting; clarify whether the Loss column is test-set loss and, if possible, report training/validation curves or the train-test gap to support the claim.
  4. [Table 3] The row citing [17] attributes GAutoML Vision to a tuberculosis-screening paper; please verify the citation, as it appears to be mismatched.
  5. [General] No code or data availability statement is provided; for a benchmark comparison, releasing training code and random seeds is important for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all reported metrics are measured on an external benchmark and none of the central claims reduce to a fitted parameter or self-citation.

full rationale

The paper's claims are empirical evaluations of CNN classifiers on the public DermaMNIST and DermaMNIST-C datasets, compared against published results. The central claim that EfficientNetV2L 'match or surpass existing methods' is a comparison of measured test-set accuracy (0.8490) against an external published number (0.851 from Ref. [14]); this is not a derivation from fitted inputs. The metrics in Eqs. (1)-(4) are standard definitions, and although Eq. (1)-(3) cite a paper co-authored by one of the present authors (Ref. [15]), this citation is only for the textbook precision/accuracy/recall formulas and is not load-bearing for any prediction. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity; no uniqueness theorem is imported; no ansatz is smuggled in via self-citation. The skeptical concerns about single-run variance, lack of error bars, and test-set-based model selection are legitimate threats to the strength of the empirical comparison, but they are correctness/robustness issues, not circularity. The paper's own limitations (e.g., class imbalance, overfitting) are acknowledged. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on standard transfer-learning assumptions and on hand-chosen hyperparameters, not on new theoretical entities. The most fragile inputs are the metric definition for AUC and the undocumented training hyperparameters.

free parameters (5)
  • Learning rate for Adam optimizer = 1e-4
    Chosen in Section 3; directly affects convergence and final metrics.
  • Dense head width = 64 or 128 units
    Selected by comparing Effv2 versus Effv3 in Table 2; Effv3 with 64 units is the best result.
  • Conv2D filter counts in Experiment 1 head = 32, 64, 128
    Hand-specified in Section 5.1; only one configuration is tested.
  • Class weights in Effv4 = not specified
    Introduced in Section 5.2 to handle imbalance; values are not given.
  • Dropout rate and early stopping patience = not specified
    Mentioned in Section 3 but values are omitted; chosen by hand.
assumptions (3)
  • domain assumption ImageNet pretrained weights provide useful features for dermatoscopic images at 28x28 and 224x224 resolution.
    Central to the transfer learning design; Section 3 states the base models are pretrained on ImageNet and then frozen.
  • ad hoc to paper The binary AUC formula in Eq. (4) can be used for a seven-class problem.
    Section 4 defines AUC with positive and negative example counts, but the experiments are multi-class; no one-vs-rest or averaging scheme is stated.
  • domain assumption The DermaMNIST-C labels and fixed splits are correct and suitable for comparison.
    The paper relies on [14] for the improved split and class labels; all accuracy numbers depend on this external curation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset." pith.science (2026). https://pith.science/paper/APILOLS7

@misc{pith2026250712961,
  author       = {Pith},
  title        = {Pith review of: Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/APILOLS7}},
  note         = {Machine review of arXiv:2507.12961}
}
read the original abstract

Pigmented skin lesions represent localized areas of increased melanin and can indicate serious conditions like melanoma, a major contributor to skin cancer mortality. The MedMNIST v2 dataset, inspired by MNIST, was recently introduced to advance research in biomedical imaging and includes DermaMNIST, a dataset for classifying pigmented lesions based on the HAM10000 dataset. This study assesses ResNet-50 and EfficientNetV2L models for multi-class classification using DermaMNIST, employing transfer learning and various layer configurations. One configuration achieves results that match or surpass existing methods. This study suggests that convolutional neural networks (CNNs) can drive progress in biomedical image analysis, significantly enhancing diagnostic accuracy.

Figures

Figures reproduced from arXiv: 2507.12961 by the authors.

Figure 1
Figure 1. The first row represents the original DermaMNIST images, while the second row represents the DermaMNIST-C images. improved organization. This version includes 8,208 training images, 1,232 test images, and 575 validation images, addressing issues of duplication and resolu￾tion to enhance classification outcomes [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Confusion matrix of Efficient￾Net model for the Experiment 1. In [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Confusion matrix of EfficientNetV2L v3 model for the Experiment 2. 7 Conclusion This study explored application of advanced deep learning techniques for clas￾sifying dermatological images. Given high prevalence of skin conditions includ￾ing melanoma aim was to enhance classification accuracy and efficiency using advanced machine learning models. The research underscores importance of uti￾lizing sophisticated deep le… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages

  1. [14]

    Investigating the quality of dermamnist and fitzpatrick17k dermatological image datasets,

    Kumar Abhishek, Aditi Jain, and Ghassan Hamarneh, “Investigating the quality of dermamnist and fitzpatrick17k dermatological image datasets,” 2024

  2. [11]

    Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis,

    Jiancheng Yang, Rui Shi, and Bingbing Ni, “Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis,” in IEEE 18th Inter- national Symposium on Biomedical Imaging (ISBI) , 2021, pp. 191–195

  3. [17]

    Hybrid learning of hand-crafted and deep-activated features using particle swarm optimization and optimized support vector machine for tuberculosis screening,

    Khin Yadanar Win and et al., “Hybrid learning of hand-crafted and deep-activated features using particle swarm optimization and optimized support vector machine for tuberculosis screening,” Applied Sciences, vol. 10, no. 17, pp. 5749, 2020. Title Suppressed Due to Excessive Length 9

  4. [23]

    Deeply Supervised Layer Selective Attention Network: Towards Label-Efficient Learning for Medical Image Classification

    Peng Jiang, Juan Liu, Lang Wang, Zhihui Ynag, Hongyu Dong, and Jing Feng, “Deeply supervised layer selective attention network: Towards label-efficient learn- ing for medical image classification,” arXiv preprint arXiv:2209.13844 , 2022

  5. [1]

    Dermatoscopy,

    Christina Ring, Nathan Cox, and Jason B Lee, “Dermatoscopy,” Clinics in Der- matology, vol. 39, no. 4, pp. 635–642, 2021

  6. [2]

    Josep Malvehy, Ralph P Braun, Susana Puig, Ashfaq A Marghoob, and Alfred W Kopf, Handbook of dermoscopy, CRC Press, 2006

  7. [3]

    Rona M MacKie, Skin cancer, CRC Press, 1996

  8. [4]

    Current and future trends in molecular biomarkers for diagnostic, prognostic, and predictive purposes in non-melanoma skin cancer,

    Taxiarchis Konstantinos Nikolouzakis, Luca Falzone, Konstantinos Lasithiotakis, Sabine Kr¨ uger-Krasagakis, Alexandra Kalogeraki, Maria Sifaki, Demetrios A Span- didos, Emmanuel Chrysos, Aristidis Tsatsakis, and John Tsiaoussis, “Current and future trends in molecular biomarkers for diagnostic, prognostic, and predictive purposes in non-melanoma skin canc...

Show all 23 references
  1. [5]

    Ashfaq A Marghoob, Ralph Peter Braun, and Alfred W Kopf, Atlas of dermoscopy, Taylor & Francis London, 2005

  2. [6]

    Karen L Agnew, Christopher B Bunker, and Sarah T Arron, Fast Facts: Skin Cancer, Karger Medical and Scientific Publishers, 2015

  3. [7]

    A gan-based image synthe- sis method for skin lesion classification,

    Zhiwei Qin, Zhao Liu, Ping Zhu, and Yongbo Xue, “A gan-based image synthe- sis method for skin lesion classification,” Computer Methods and Programs in Biomedicine, vol. 195, pp. 105568, 2020

  4. [8]

    Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni, “Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,” Scientific Data , vol. 10, no. 1, pp. 41, 2023

  5. [9]

    The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,

    Philipp Tschandl, “The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” 2018

  6. [10]

    Applications of support vector machine (svm) learning in cancer genomics,

    Shujun Huang, Nianguang Cai, Pedro Penzuti Pacheco, Shavira Narrandes, Yang Wang, and Wayne Xu, “Applications of support vector machine (svm) learning in cancer genomics,” Cancer genomics & proteomics , vol. 15, no. 1, pp. 41–51, 2018

  7. [12]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific data , p. 180161, 2018

  8. [13]

    Skin lesion analysis toward melanoma detection 2018: A chal- lenge hosted by the international skin imaging collaboration (isic),

    Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al., “Skin lesion analysis toward melanoma detection 2018: A chal- lenge hosted by the international skin imag...

  9. [15]

    Unsupervised method for wildfire flame segmen- tation and detection,

    Emir Buza and Amila Akagic, “Unsupervised method for wildfire flame segmen- tation and detection,” IEEE Access, vol. 10, pp. 55213–55225, 2022

  10. [16]

    A review on evaluation metrics for data classification evaluations,

    Mohammad Hossin and Md Nasir Sulaiman, “A review on evaluation metrics for data classification evaluations,” International journal of data mining & knowledge management process, vol. 5, no. 2, pp. 1, 2015

  11. [18]

    Attentive octave convolutional capsule network for medical image classification,

    Hong Zhang, Zhengzhen Li, Hao Zhao, Zan Li, and Yanping Zhang, “Attentive octave convolutional capsule network for medical image classification,” Applied Sciences, vol. 12, no. 5, 2022

  12. [19]

    Extreme learning machine for biomedical image classification: A multi- case study,

    Francesco Mercaldo, Luca Brunese, Antonella Santone, Fabio Martinelli, and Mario Cesarelli, “Extreme learning machine for biomedical image classification: A multi- case study,” EAI Endorsed Transactions on Pervasive Health and Technology , vol. 10, 2024

  13. [20]

    Skin lesion clas- sification based on hybrid self-supervised pretext task,

    Dedong Yang, Jianwen Zhang, Yangyang Li, and Zhiquan Ling, “Skin lesion clas- sification based on hybrid self-supervised pretext task,” International Journal of Imaging Systems and Technology , vol. 34, no. 2, pp. e23058, 2024

  14. [21]

    Medrdf: a robust and retrain-less diagnostic framework for medical pretrained models against adversarial attack,

    Mengting Xu, Tao Zhang, and Daoqiang Zhang, “Medrdf: a robust and retrain-less diagnostic framework for medical pretrained models against adversarial attack,” IEEE Transactions on Medical Imaging , vol. 41, no. 8, pp. 2130–2143, 2022

  15. [22]

    Failure detection in deep neural networks for medical imaging,

    Sabeen Ahmed, Dimah Dera, Saud Ul Hassan, Nidhal Bouaynaya, and Ghulam Rasool, “Failure detection in deep neural networks for medical imaging,” Frontiers in Medical Technology, vol. 4, pp. 919046, 2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.