Pith. sign in

REVIEW 3 major objections 6 minor 32 references

Melanoma Detection with Uncertainty Quantification

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By training on merged public datasets and rejecting uncertain predictions, this paper pushes melanoma detection accuracy from 93.2% to 97.8% while cutting misdiagnoses by over 40.5%.

desk verdict A useful benchmark and public toolkit, but the misdiagnosis-reduction claim lacks the random-rejection control needed to support it. read the letter →

arxiv 2411.10322 v1 pith:4XVYG3X2 submitted 2024-11-15 cs.CV

SangHyuk Kim , Edward Gaibor , Brian Matejek , Daniel Haehn This is my paper · ORCID
classification cs.CV
keywords melanomadetectionuncertaintyquantificationentropy-basedrejectionmodelcalibrationdatasetmergingdeepneuralnetworksmedicalimagingskinlesionclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that melanoma detection improves when classifiers are trained on a pool of merged public skin-lesion datasets and then filtered with an entropy-based uncertainty score. On four benchmark test sets, the authors report accuracy rising from up to 93.2% before rejection to 97.8% after rejection, with misdiagnoses falling by over 40.5%. The practical point is that a detector can be made safer by explicitly labeling low-confidence cases as 'Uncertain' and referring them to a human expert instead of forcing a binary yes/no answer.

What carries the argument

The central mechanism is Shannon entropy of the softmax probability vector, $H(X) = -\sum_i p_i \log p_i$, used as an uncertainty score. Predictions whose entropy exceeds a per-model threshold are rejected as 'Uncertain' and referred to a human expert; the thresholds are chosen on the validation set by minimizing the Expected Calibration Error and Brier score, and only the remaining confident predictions are evaluated. Empirically, rejecting up to 20% of low-confidence predictions improves accuracy.

What would settle it

Run the same pipeline on an independent clinical cohort with a different acquisition protocol, and check whether the validation-tuned threshold rejects a large fraction of true melanomas or whether expert review of rejected images fails to recover the classifier's missed melanomas; either observation would refute the claim that uncertainty rejection reduces misdiagnoses.

Watch

Extended reading notes

Core claim

The central claim is that uncertainty-based rejection, applied to softmax classifiers trained on combined public datasets, substantially improves melanoma detection while also improving calibration. After calibrating models and rejecting the most uncertain predictions, accuracy on the ISIC2017, ISIC2018, 7-point criteria, and Kaggle test sets reaches as high as 97.8%, up from 93.2%, with large gains in precision and sensitivity and a reduction in misdiagnoses of over 40.5%. The paper also finds that combining multiple datasets improves both classification and calibration, and that shallower networks are often better calibrated than deeper ones.

Load-bearing premise

The reported gains assume that a rejection threshold tuned on a validation set transfers to the test distribution, and that measuring accuracy only on the retained, non-rejected samples—with rejected cases referred to an expert—is a fair measure of real diagnostic improvement.

Editorial extensions

If this is right

  • Any softmax classifier can be augmented with a third 'Uncertain' output without retraining, by thresholding the entropy of its probability vector.
  • Training on merged datasets improves calibration as well as accuracy, so confidence scores from multi-source training can be taken more literally.
  • Shallower networks such as EfficientNet and VGG often show better calibration than deeper networks, which matters for deployment where trust in confidence is critical.
  • Entropy-based rejection reduces both false positives and false negatives, including most false negatives on the Kaggle test set, which is the clinically dangerous error direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The threshold-selection strategy could be stress-tested under distribution shift by tuning on one dataset and applying the fixed threshold to a dataset collected with a different camera or population; the paper does not report this experiment.
  • A prospective study that tracks whether dermatologists confirm the 'Uncertain' referrals would be the natural validation step for clinical adoption.
  • The same merge-and-reject pipeline could transfer to other imbalanced medical imaging tasks such as mammography, a direction the authors list as future work.
  • The reported 40.5% misdiagnosis reduction is computed on retained samples only; a full accounting that treats rejected samples as diagnosed-by-expert could change the headline number.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes a framework for melanoma detection that merges up to ten public dermoscopic datasets, trains 24 CNN architectures under a unified protocol (1,296 dataset/model combinations), calibrates classifiers using ECE and Brier score, and rejects predictions whose Shannon entropy exceeds a model-specific threshold tuned on validation data. On four external test sets (ISIC2017, ISIC2018, 7-point criteria, Kaggle), the selected top models achieve accuracy up to 93.2% before rejection and 97.8% after rejection, with a claimed reduction in misdiagnoses of over 40.5%. The authors also release code, data, and a client-side web interface.

Significance. If the improvements are real, the paper provides a useful and reproducible empirical study of uncertainty-based deferral for melanoma classification, with several commendable strengths: the 1,296-experiment sweep over public datasets, external test-set evaluation, calibration metrics, a public repository, and a deployed web interface. However, the central quantitative claims currently rest on (i) post hoc selection of best models, (ii) retained-only metrics without coverage or a random-rejection control, and (iii) single-seed results with no intervals. These are fixable, and the work would be valuable after those controls are added.

major comments (3)
  1. [Section 3.3, Table 1, Fig. 5] The central claim that uncertainty-based rejection causes the reported improvements is not yet established. Thresholds are tuned on validation data, but Table 1 reports 'after rejection' precision, sensitivity, specificity, F1, accuracy, and AUC only on the retained test samples, with no count or fraction of rejected samples given for any row. Since removing difficult samples mechanically raises accuracy on the remainder, the gains (e.g., 93.2% to 97.8% on Kaggle) are also fully consistent with a selection effect that would occur even for random rejection. The authors should report the retained-sample size/coverage for every configuration and compare against a random-rejection baseline at matched coverage, with confidence intervals. They should also specify how rejected cases are counted in the misdiagnosis-reduction claim in Fig. 5: if rejected cases are referred to an expert, the end-to-end misdiagnosis rate depends on referral accuracy, which is not modeled.
  2. [Section 4 and Table 1] The paper's headline numbers are selected post hoc. Figure 1 states that 1,296 experiments were run, and Table 1 shows only 'top-performing models' per test set, apparently chosen by precision on the same test sets. This selection invalidates the strongest point estimates as estimates of framework performance. The authors should report the full distribution (mean, median, range) of metrics across all 1,296 runs, or use a fixed model set, or split test sets so that model selection and evaluation are separate. Without this, the claim that the framework 'achieves' 97.8% accuracy is an optimistic maximum, not an expected performance.
  3. [Sections 3.2–4] No statistical uncertainty is quantified. All experiments appear to be single runs; no random seeds, standard deviations, or confidence intervals are reported. Given stochastic SGD training and small test sets (7-point criteria has a few hundred images), differences of several percentage points between before/after rejection may not be significant. Please report at least three training seeds (or bootstrap confidence intervals) for the key metrics in Table 1 and Fig. 5.
minor comments (6)
  1. [Section 2.3, Eq. (1)] Please specify the base of the logarithm and the normalization. The claim that H(X) lies in [0,1] holds for binary classification only if log base 2 is used, but the equation uses an unspecified log.
  2. [Table 3, ISIC2018 row] The entry 'VGG19 [19]' should cite reference [21] (VGG), not [19] (DenseNet).
  3. [Section 4] The sentence 'ResNet152 improves precision by at least 10% points' is contradicted by Table 1, e.g., the ISIC2017 row [A–D,I,J] shows 82.0% to 87.0%, a gain of only 5 percentage points. Please rephrase to 'up to more than 10 percentage points' or report per-row gains.
  4. [Figure 5] The caption says 'A plot comparing false diagnoses before and after applying uncertainty-based rejection across benchmarks,' but the y-axis label and legend are difficult to read. Please clarify what quantity is plotted (absolute counts or normalized rates) and for which model(s) and test sets.
  5. [Section 2.4] The description of the 'Uncertain' class is brief. Please clarify whether the system outputs a three-way decision ('Melanoma,' 'Non-Melanoma,' 'Don't Know') and how that interacts with the binary softmax probabilities.
  6. [Table 1] Several cells have inconsistent spacing around slashes (e.g., '91.5% / 91.8%' versus '84.8% /94.1%') and some numbers are missing spaces. Please reformat for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the uncertainty-rejection thresholds are tuned on validation sets and performance is measured on external test sets, with no load-bearing self-citation or definitional reduction.

full rationale

This paper is an empirical benchmark study rather than a derivation from a first-principles model, so the circularity patterns of self-definition, renamed known results, or imported uniqueness theorems do not apply. The only tunable component in the uncertainty-rejection pipeline is the rejection threshold, and Section 3.3 states that it is selected on validation sets: "We calculate ECE and Brier scores while varying the threshold to minimize their average for each model. After this dynamic rejection, we re-evaluate the remaining samples and compare them to the originals." The post-rejection accuracy in Table 1 is then reported on held-out external test sets (ISIC2017, ISIC2018, 7-point criteria, Kaggle), so the improvement is not forced by construction from the threshold choice. The entropy formula (Eq. 1), ECE (Eq. 2), and Brier score (Eq. 3) are standard external definitions, not redefinitions of the outcome. Reference [11] is used only for the web-application architecture and is not load-bearing for the central claim. The absence of a random-rejection baseline is a legitimate experimental-design concern, since the reported gains could partly reflect a selection effect rather than the informativeness of the entropy score, but that is a correctness risk, not circularity under the stated criteria.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim depends on three domain assumptions about the dataset merge, the validity of entropy as an uncertainty signal, and threshold transfer, plus a per-model threshold free parameter. No new physical or methodological entities are introduced.

free parameters (1)
  • uncertainty rejection threshold per model = Ranges from 0.08 to 0.2 across reported models (Table 1)
    Selected per model by minimizing ECE and Brier score on the validation set (Section 3.3). The post-rejection accuracy and misdiagnosis reduction directly depend on this choice, so it is a fitted hyperparameter in the evaluation.
assumptions (4)
  • standard math Shannon entropy H(X) = -sum p_i log p_i is an appropriate measure of predictive uncertainty for a softmax classifier.
    Used in Section 2.3 to compute uncertainty scores; this is a standard information-theoretic definition.
  • domain assumption Resizing all images to 224x224 pixels and mapping all non-melanoma labels to a single class preserves enough information for binary melanoma detection.
    Invoked in Section 3.1 during preprocessing and binary label conversion; if this assumption fails, the merged dataset is not a valid training signal.
  • domain assumption High-entropy softmax predictions are more likely to be misclassifications, so rejecting them reduces errors.
    This is the core premise of the uncertainty rejection module in Section 2.3. The paper provides no calibration of entropy against actual error rates beyond the reported aggregate gains.
  • ad hoc to paper A rejection threshold tuned on the validation set generalizes to the test distribution for each model.
    Section 3.3 selects thresholds by minimizing ECE/Brier on validation sets; test-set metrics then depend on this generalization without an explicit check of distribution shift.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Melanoma Detection with Uncertainty Quantification." pith.science (2026). https://pith.science/paper/4XVYG3X2

@misc{pith2026241110322,
  author       = {Pith},
  title        = {Pith review of: Melanoma Detection with Uncertainty Quantification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XVYG3X2}},
  note         = {Machine review of arXiv:2411.10322}
}
read the original abstract

Early detection of melanoma is crucial for improving survival rates. Current detection tools often utilize data-driven machine learning methods but often overlook the full integration of multiple datasets. We combine publicly available datasets to enhance data diversity, allowing numerous experiments to train and evaluate various classifiers. We then calibrate them to minimize misdiagnoses by incorporating uncertainty quantification. Our experiments on benchmark datasets show accuracies of up to 93.2% before and 97.8% after applying uncertainty-based rejection, leading to a reduction in misdiagnoses by over 40.5%. Our code and data are publicly available, and a web-based interface for quick melanoma detection of user-supplied images is also provided.

Figures

Figures reproduced from arXiv: 2411.10322 by the authors.

Figure 3
Figure 3. We calculate ECE and Brier scores while varying the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 23 canonical work pages

  1. [1]

    Deep Neural Networks (DNNs) are effective in melanoma detection, but their evaluation lacks standardization, complicating comparisons

    INTRODUCTION Melanoma is a severe skin cancer responsible for around 55,500 deaths annually [1]. Deep Neural Networks (DNNs) are effective in melanoma detection, but their evaluation lacks standardization, complicating comparisons. For exam- ple, DNN-based melanoma classification using segmented features [2] has shown promising results on the ISIC’16 [3] ...

    work page

  2. [2]

    Melanoma Detection with Uncertainty Quantification

    METHODS 2.1. Input Most public melanoma datasets include images and associ- ated metadata about skin lesions. Image sizes range from 147x147 to 3096x3096 pixels, with class counts from 2 to 9, consistently including melanoma labels. These datasets vary in structure—some use a single folder with a separate CSV for labels and splits, while others organize b...

    work page Pith review arXiv 2024

  3. [3]

    Melanoma

    EXPERIMENTAL SETUP 3.1. Datasets and Preprocessing We utilize publicly available skin disease datasets [3, 4, 6, 12, 13, 14, 15, 16, 17, 18] and evaluate our experiments using pro- vided test sets [4, 6, 15, 18]. When a validation set is not pro- vided, we split the training set into an 80:20 ratio for creating a validation set. These datasets include a “...

    work page

  4. [4]

    ) * ' " ) * % '

    RESULTS Performance Before Rejection. Classification performance is ranked by precision, following the ISIC challenge [3]. DenseNet201 and ResNet152 achieve the best results, with Fig. 4. Calibration curves of ResNet50 as a demonstration. Left: Single dataset (ISIC’20). Right: Combined datasets (ISIC’16, ISIC’17, ISIC’18, MEDNODE, Kaggle). Table 2. Leader...

    work page arXiv 2017

  5. [5]

    Future plans include collecting real-time diagnosis data from dermatologists and using Generative Adversarial Networks (GAN) for iterative retraining

    CONCLUSIONS We propose a melanoma framework integrating unified train- ing and testing with uncertainty-based rejection to enhance classification and calibration while reducing misdiagnoses. Future plans include collecting real-time diagnosis data from dermatologists and using Generative Adversarial Networks (GAN) for iterative retraining. We also aim to ...

    work page

  6. [6]

    Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DLA or ARPA-H

    ACKNOWLEDGMENTS This material is based, in part, upon work supported by the Defense Logistics Agency (DLA) and the Advanced Re- search Projects Agency for Health (ARPA-H) under Contract Number SP4701-23-C-0073. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the...

    work page

  7. [7]

    COMPLIANCE WITH ETHICAL STANDARDS All datasets used in this study were anonymized and publicly available, for which no ethical approval was required

    work page

  8. [8]

    Melanoma,

    Dirk Schadendorf, Alexander CJ Van Akkooi, Car- ola Berking, Klaus G Griewank, Ralf Gutzmer, Axel Hauschild, Andreas Stang, Alexander Roesch, and Selma Ugurel, “Melanoma,” The Lancet, vol. 392, no. 10151, pp. 971–984, 2018

    work page 2018

Show all 32 references
  1. [9]

    Hy- brid fully convolutional networks-based skin lesion seg- mentation and melanoma detection using deep feature,

    Kalyanakumar Jayapriya and Israel Jeena Jacob, “Hy- brid fully convolutional networks-based skin lesion seg- mentation and melanoma detection using deep feature,” International Journal of Imaging Systems and Technol- ogy, vol. 30, no. 2, pp. 348–357, 2020

    2020

  2. [10]

    David Gutman, Noel CF Codella, Emre Celebi, Brian Helba, Michael Marchetti, Nabin Mishra, and Allan Halpern, “Skin lesion analysis toward melanoma de- tection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the interna- tional skin imag...

    2016 arXiv

  3. [11]

    IEEE, 2018, pp

    Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al., “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international sym- posium on biom...

    2017

  4. [12]

    A new approach for classifica- tion skin lesion based on transfer learning, deep learn- ing, and iot system,

    Douglas de A Rodrigues, Roberto F Ivo, Suresh Chan- dra Satapathy, Shuihua Wang, Jude Hemanth, and Pe- dro P Reboucas Filho, “A new approach for classifica- tion skin lesion based on transfer learning, deep learn- ing, and iot system,” Pattern Recognition Letters, vol. 136, pp...

    2020

  5. [13]

    Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the interna- tional skin imaging collaboration (isic),

    Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al., “Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the interna- tional skin imag...

    2018 arXiv

  6. [14]

    Comparative study of deep learning models in melanoma detection,

    Farnaz Haghshenas, Adam Krzy ˙zak, and Stanislaw Os- owski, “Comparative study of deep learning models in melanoma detection,” in IAPR Workshop on Artifi- cial Neural Networks in Pattern Recognition . Springer, 2024, pp. 121–131

    2024

  7. [15]

    On calibration of modern neural networks,

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger, “On calibration of modern neural networks,” in International conference on machine learning . PMLR, 2017, pp. 1321–1330

    2017

  8. [16]

    Verification of forecasts expressed in terms of probability,

    Glenn W Brier, “Verification of forecasts expressed in terms of probability,” Monthly weather review, vol. 78, no. 1, pp. 1–3, 1950

    1950

  9. [17]

    Shannon entropy as a measure of un- certainty in positions and momenta,

    Lukasz Rudnicki, “Shannon entropy as a measure of un- certainty in positions and momenta,”Journal of Russian Laser Research, vol. 32, pp. 393–399, 2011

    2011

  10. [18]

    End- to-end learning of brain tissue segmentation from imper- fect labeling,

    Alex Fedorov, Jeremy Johnson, Eswar Damaraju, Alexei Ozerin, Vince Calhoun, and Sergey Plis, “End- to-end learning of brain tissue segmentation from imper- fect labeling,” in 2017 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2017, pp. 3785– 3792

    2017

  11. [19]

    Bcn20000: Dermoscopic lesions in the wild,

    Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al., “Bcn20000: Dermoscopic lesions in the wild,” arXiv preprint arXiv:1908.02288, 2019

    1908 arXiv

  12. [20]

    A patient-centric dataset of images and metadata for identifying melanomas using clinical context,

    Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz- Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Marc Combalia, Stephen Dusza, Pascale Guit- era, David Gutman, et al., “A patient-centric dataset of images and metadata for identifying melanomas using clinical con...

    2021

  13. [21]

    Ph 2-a der- moscopic image database for research and benchmark- ing,

    Teresa Mendonc ¸a, Pedro M Ferreira, Jorge S Marques, Andr´e RS Marcal, and Jorge Rozeira, “Ph 2-a der- moscopic image database for research and benchmark- ing,” in 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, ...

    2013

  14. [22]

    Seven-point check- list and skin lesion classification using multitask mul- timodal neural nets,

    Jeremy Kawahara, Sara Daneshvar, Giuseppe Argen- ziano, and Ghassan Hamarneh, “Seven-point check- list and skin lesion classification using multitask mul- timodal neural nets,” IEEE journal of biomedical and health informatics, vol. 23, no. 2, pp. 538–546, 2018

    2018

  15. [23]

    Pad-ufes-20: A skin lesion dataset composed of patient data and clinical im- ages collected from smartphones,

    Andre GC Pacheco, Gustavo R Lima, Amanda S Salo- mao, Breno Krohling, Igor P Biral, Gabriel G de An- gelo, F ´abio CR Alves Jr, Jos ´e GM Esgario, Alana C Simora, Pedro BC Castro, et al., “Pad-ufes-20: A skin lesion dataset composed of patient data and clinical im- ages collec...

    2020

  16. [24]

    Med- node: A computer-assisted melanoma diagnosis system using non-dermoscopic images,

    Ioannis Giotis, Nynke Molders, Sander Land, Michael Biehl, Marcel F Jonkman, and Nicolai Petkov, “Med- node: A computer-assisted melanoma diagnosis system using non-dermoscopic images,” Expert systems with applications, vol. 42, no. 19, pp. 6578–6585, 2015

    2015

  17. [25]

    Skin cancer: Malignant vs. benign,

    CLAUDIO FANCONI, “Skin cancer: Malignant vs. benign,” https://www. kaggle.com/datasets/fanconic/ skin-cancer-malignant-vs-benign

  18. [26]

    Densely connected convolu- tional networks,

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger, “Densely connected convolu- tional networks,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition, 2017, pp. 4700–4708

    2017

  19. [27]

    Deep residual learning for image recognition,

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  20. [28]

    Very deep convolutional networks for large-scale image recogni- tion,

    Karen Simonyan and Andrew Zisserman, “Very deep convolutional networks for large-scale image recogni- tion,” arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  21. [29]

    Efficientnet: Rethinking model scaling for convolutional neural networks,

    Mingxing Tan and Quoc Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International conference on machine learning . PMLR, 2019, pp. 6105–6114

    2019

  22. [30]

    Imagenet: A large-scale hierarchical image database,

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  23. [31]

    Large-scale machine learning with stochastic gradient descent,

    L ´eon Bottou, “Large-scale machine learning with stochastic gradient descent,” in Proceedings of COMP- STAT’2010: 19th International Conference on Com- putational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers . Springer, 2010, pp. 177–186

    2010

  24. [32]

    Siim-isic melanoma classification,

    Anna Zawacki, “Siim-isic melanoma classification,” 2020

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.