Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Deep neural network or dermatologist?

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Melanoma-classifying neural networks can score high accuracy while relying on image artifacts rather than lesion features.

desk verdict A reproducible, honest empirical study showing that high-AUC melanoma classifiers can rely on spurious cues like image corners, but the saliency-map evidence for that claim has a faithfulness gap; still worth a serious referee. read the letter →

arxiv 1908.06612 v1 pith:LKUBBX56 submitted 2019-08-19 cs.LG eess.IVstat.ML

classification cs.LGeess.IVstat.ML
keywords melanomadetectioninterpretabilityGradCAMKernelSHAPsaliencymapsdeeplearningdermoscopyimagesmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether two interpretability tools, GradCAM and Kernel SHAP, can show why a deep network calls a skin lesion melanoma. The authors train 30 convolutional networks on the public HAM10000 dataset to distinguish melanoma from benign moles, then inspect the resulting saliency maps. They find that accuracy is not the whole story: models with the same AUC sometimes make decisions using different image features, and some models weight the classification using irrelevant regions such as dark corners of the image. The upshot is that reporting AUC alone can hide what a model has actually learned, and saliency maps need further validation before they can guide clinical use.

What carries the argument

The load-bearing tools are the two local interpretability maps. GradCAM weights the final convolutional feature maps by the gradient of the class score, producing a heatmap of pixels that drove the melanoma decision. Kernel SHAP approximates Shapley values from cooperative game theory, attributing to each pixel region its fair contribution to the prediction. The paper uses these maps on 30 Inception-style transfer-learned models, and uses randomization tests—shuffling progressively shallower layer weights and checking that the maps change—to confirm the maps carry model-specific information. The maps are what connect the observed AUC differences to claims about spurious correlations.

What would settle it

Take one of the models whose saliency map highlights the dark corners of a dermoscopy image, ablate or mask that corner region, and re-run the prediction. If removing the highlighted region leaves the model's output essentially unchanged, the map did not reveal a causally important feature, and the claim that the model weights the decision using spurious correlations is undermined. Conversely, if a same-AUC model whose map highlights the lesion keeps its prediction after the mask while the corner-fixated model flips, the paper's reading is supported.

Watch

Extended reading notes

Core claim

The authors establish that, within this training setup, interpretability maps expose failure modes that accuracy metrics miss: 17 test images were misclassified by at least 25 of 30 models, and saliency maps from models with identical AUCs of 85% differed by an average structural similarity (SSIM) of 20%, with one model relying on dark image corners to call a lesion melanoma. They also show that GradCAM and Kernel SHAP pass basic sanity checks—reproducible output and dependence on model weights—so the differing explanations are not merely noise. The conclusion is that similar accuracy does not imply similar reasoning, and interpretability screening should accompany accuracy reporting for skin-cancer classifiers.

Load-bearing premise

The argument assumes that the saliency maps produced by GradCAM and Kernel SHAP faithfully reveal which image features actually drove each prediction, not just that the maps change when model weights change.

Editorial extensions

If this is right

  • Reporting AUC alone is insufficient to characterize a melanoma-detection model; equal-AUC models can rely on different features.
  • Saliency maps located on clinically irrelevant regions, such as the image border, can flag predictions that should not be trusted even when the model is confident.
  • Seventeen consistently misclassified test images, mostly also missed by primary care physicians, identify a concrete hard subset for future training and evaluation.
  • Interpretability methods that pass weight-randomization sanity checks are not guaranteed to be clinically meaningful; further validation is required before deployment.
  • Because the suite of 30 models and the training code are publicly released, the result is reproducible and can be extended to other architectures and datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dark-corner reliance suggests a testable fix: cropping or masking image corners before training could measure how much of the reported AUC depends on artifacts.
  • Models with identical AUC but divergent saliency maps could be combined into ensembles to reduce the influence of any single spurious cue, an extension the paper mentions only for future accuracy gains.
  • A stricter validation of these maps would require causal intervention, such as ablating highlighted regions and checking that predictions change accordingly; the sanity checks here establish model dependence, not causal faithfulness.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper trains 30 Inception-based CNNs on the public HAM10000 dermoscopic dataset for binary melanoma versus naevus classification, reporting a mean AUC of 85%. It then applies two local interpretability methods, GradCAM and Kernel SHAP, to the trained models and performs three sanity checks: reproducibility, model-weight dependence via layer randomization, and sensitivity across three same-AUC models. The authors present example saliency maps, including one case where the highlighted region lies in the dark corners of an image, and argue that evaluating models by AUC alone gives limited insight because models of similar accuracy can produce different explanations. Code and trained models are released.

Significance. If the central interpretive claim is established, the paper makes a useful empirical contribution to medical AI evaluation: it provides a reproducible model suite, applies standard sanity checks to interpretability methods in a clinically relevant domain, and shows that AUC-matched models can produce visually different saliency maps. The release of code and trained models is a concrete strength that supports reproducibility. However, the paper's main conclusion goes beyond raw map differences to claims about latent decision rationales, and that step requires evidence the current experiments do not fully supply.

major comments (3)
  1. [Section 2.5, Fig. 3] The inference that the model "has learned to weight the class decision using a spurious correlation" assumes that the GradCAM and Kernel SHAP maps faithfully reflect the features that drive the classifier's output. The sanity checks in Section 2.4 demonstrate reproducibility and weight dependence, but not fidelity; a map can depend on weights and still fail to track decision-relevant features. The manuscript should either add a faithfulness check (for example, occluding the highlighted regions and measuring the change in prediction probability) or consistently restrict the conclusion to the weaker, directly observable claim that the maps highlight clinically irrelevant regions.
  2. [Section 2.4, sensitivity experiment] The sensitivity test compares maps from three same-AUC models and reports an average SSIM difference of 20%, but it does not establish that the differences exceed the intrinsic stochasticity of the explanation methods. In Section 2.4(i), Kernel SHAP maps have mean SSIM 0.92 with standard deviation 0.028 when run twice on the same model and image, so part of the observed 20% variation could be method noise rather than a difference in model rationale. The authors should report the sensitivity statistic relative to the reproducibility baseline and, ideally, perform repeated runs of each method on each model.
  3. [Section 2.5 and Section 2.6] The spurious-correlation analysis is based on visual inspection of a small number of selected images, and the claimed frequency ("occasionally") is never quantified across the 30 models and the test set. Because the Limitations section correctly warns that visual inspection will become impractical at scale, the paper should specify how many model-image pairs produced maps on clinically irrelevant regions, ideally with a lesion-segmentation or background mask and a simple quantitative overlap criterion.
minor comments (5)
  1. [Section 2.2] The statement that the mean AUC has "a variance of 1.8%" is ambiguous; please report the standard deviation or a confidence interval with explicit units.
  2. [Section 2.3] The description of Kernel SHAP as needing to account for all 2^{d-1} subsets is correct for exact Shapley values but potentially misleading for the approximate Kernel SHAP estimator actually used; consider clarifying the distinction.
  3. [Fig. 3 caption] Using "confidence" for softmax probabilities is imprecise, since modern CNNs are often miscalibrated; consider saying "prediction probability."
  4. [Title and Introduction] The title asks "Deep neural network or dermatologist?" but the paper does not include a head-to-head comparison with dermatologists; the human evaluation in Section 2.2 involves three primary care physicians on 17 images. Consider either adding such a comparison or retitling to reflect the interpretability focus.
  5. [Section 2.4] The text contains a typo in the phrase "for both GradCShrapenlAM and kernel SHAP"; this should read "GradCAM."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical, externally benchmarked study whose claims are not constructed from fitted inputs or self-citation chains.

full rationale

This is an empirical study rather than a derivation. The central claims are (i) that GradCAM and Kernel SHAP maps pass basic sanity checks in this domain, (ii) that models occasionally assign importance to clinically irrelevant image features, and (iii) that models of similar AUC produce different saliency maps. None of these claims is defined in terms of another claim that is being predicted, and no fitted parameter is renamed as a prediction. The interpretability methods are existing external methods: GradCAM is justified via the reference to Adebayo et al. [2] and Selvaraju et al. [19], and Kernel SHAP is justified via Lundberg and Lee [12], both external to the present paper. The authors also perform their own reproducibility, model-dependence, and sensitivity checks, comparing maps to externally defined SSIM baselines. The only self-citation is reference [21] (GradMask, by co-author Becks Simpson et al.), appearing only in the Limitations section as a future-work suggestion for regularization; it is not load-bearing for any of the paper's conclusions. The skeptical concern that saliency maps may not faithfully reflect true decision-relevant features is a validity assumption, not a circularity: the paper explicitly acknowledges in Section 2.6 that 'a saliency map located on the lesion can not yet be viewed as justification that clinically meaningful correlations have been learned,' and the central empirical observations (same-AUC models give different maps; some maps highlight dark corners) do not reduce to their inputs by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. The hyperparameters are model-defining and searched, not fixed to force any result. The load-bearing assumptions are the reliability of the public dataset labels and the faithfulness of the interpretability methods.

free parameters (1)
  • CNN training hyperparameters (learning rate, dropout, momentum, beta1, beta2, dense nodes, epochs, optimizer) = Not reported per model; selected via Bayesian hyperparameter search across 30 models
    These define the population of models studied. The central claim about models of similar accuracy depends on this population, but not on any specific hyperparameter value.
assumptions (3)
  • domain assumption HAM10000 ground-truth labels for naevus and melanoma are sufficiently accurate.
    Used in Section 2.1 as the training and test labels. If labels are systematically wrong, the spurious-correlation findings may be artifacts of mislabeled data.
  • domain assumption GradCAM and Kernel SHAP saliency maps are meaningful proxies for the model's decision process.
    Invoked throughout Sections 2.3 to 2.5. The paper performs sanity checks, but the maps are not validated against ground-truth feature attribution.
  • domain assumption The 30 trained models are representative enough to support statements about models in general.
    The paper trains 30 models on subsamples of HAM10000 with one architecture; conclusions may not generalize to other architectures or datasets, a limitation noted in Section 2.6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep neural network or dermatologist?." pith.science (2026). https://pith.science/paper/LKUBBX56

@misc{pith2026190806612,
  author       = {Pith},
  title        = {Pith review of: Deep neural network or dermatologist?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LKUBBX56}},
  note         = {Machine review of arXiv:1908.06612}
}
read the original abstract

Deep learning techniques have proven high accuracy for identifying melanoma in digitised dermoscopic images. A strength is that these methods are not constrained by features that are pre-defined by human semantics. A down-side is that it is difficult to understand the rationale of the model predictions and to identify potential failure modes. This is a major barrier to adoption of deep learning in clinical practice. In this paper we ask if two existing local interpretability methods, Grad-CAM and Kernel SHAP, can shed light on convolutional neural networks trained in the context of melanoma detection. Our contributions are (i) we first explore the domain space via a reproducible, end-to-end learning framework that creates a suite of 30 models, all trained on a publicly available data set (HAM10000), (ii) we next explore the reliability of GradCAM and Kernel SHAP in this context via some basic sanity check experiments (iii) finally, we investigate a random selection of models from our suite using GradCAM and Kernel SHAP. We show that despite high accuracy, the models will occasionally assign importance to features that are not relevant to the diagnostic task. We also show that models of similar accuracy will produce different explanations as measured by these methods. This work represents first steps in bridging the gap between model accuracy and interpretability in the domain of skin cancer classification.

Figures

Figures reproduced from arXiv: 1908.06612 by the authors.

Figure 1
Figure 1. Naevus mis-classified by all 30 models as a melanoma, with GradCAM and kernel SHAP saliency maps. Note there is more than one type of network within the lesion, a feature which can be a marker for melanoma. The GradCAM map (centre image) highlights a key deficiency of the method in this context: almost all of the lesion is obscured by the saliency map, rendering the “explanation” clinically meaningless. inputs to ob… view at source ↗
Figure 2
Figure 2. Explanations following randomization of selected layers in the model. Changes demonstrate dependence of explanation on model weights. SSIM scores averaged over all images for GradCAM degraded across layers by 23%, 4%, 3%, 2%, 4%. Differences were also seen for kernel SHAP: 17%, 3%, 5%, 3%, 7%. Green signifies areas of positive contribution to a diagnosis of melanoma, red signifies negative. 2.4 Sanity checks We perf… view at source ↗
Figure 3
Figure 3. GradCAM and kernel SHAP from two models with AUC 85%. Model 1 cor￾rectly predicted melanoma with 0.999 confidence (first two images). Model 2 incorrectly predicted naevus with 0.996 confidence (second two images). The saliency maps indi￾cate model 2 has learned to weight the class decision using a spurious correlation: the dark corners of the image. 2.6 Limitations There are a number of limitations of this study. Th… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages

  1. [1]

    arXiv preprint arXiv:1903.10464 (2019)

    Aas, K., Jullum, M., Løland, A.: Explaining individual predictions when features are dependent: More accurate approximations to shapley values. arXiv preprint arXiv:1903.10464 (2019)

  2. [2]

    In: Advances in Neural Information Processing Systems

    Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: Advances in Neural Information Processing Systems. pp. 9505–9515 (2018)

  3. [3]

    European Journal of Cancer113, 47–54 (2019)

    Brinker, T.J., Hekler, A., Enk, A.H., Klode, J., Hauschild, A., Berking, C., Schilling, B., Haferkamp, S., Schadendorf, D., Holland-Letz, T., et al.: Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. European Journal of Cancer113, 47–54 (2019)

  4. [4]

    J Med Internet Res 20(10), e11936 (Oct 2018)

    Brinker, T.J., Hekler, A., Utikal, J.S., Grabe, N., Schadendorf, D., Klode, J., Berk- ing, C., Steeb, T., Enk, A.H., von Kalle, C.: Skin cancer classification using con- volutional neural networks: Systematic review. J Med Internet Res 20(10), e11936 (Oct 2018)

  5. [5]

    arXiv preprint arXiv:1808.02610 (2018)

    Chen, J., Song, L., Wainwright, M.J., Jordan, M.I.: L-shapley and c-shapley: Ef- ficient model interpretation for structured data. arXiv preprint arXiv:1808.02610 (2018)

  6. [6]

    BMJ 357 (2017)

    Elmore, J.G.: Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and reproducibility study. BMJ 357 (2017). https://doi.org/10.1136/bmj.j2813

  7. [7]

    Nature 542(7639), 115 (2017)

    Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115 (2017)

  8. [8]

    computer-aided classifier developed with a small dataset of clinical images sur- passes board-certified dermatologists in skin tumour diagnosis

    Fujisawa, Y., Otomo, Y., Ogata, Y., Nakamura, Y., Fujita, R., Ishitsuka, Y., Watanabe, R., Okiyama, N., Ohara, K., Fujimoto, M.: Deep-learning-based, 8 K.Young et al. computer-aided classifier developed with a small dataset of clinical images sur- passes board-certified dermatologists in skin tumour diagnosis. British Journal of Dermatology 180(2), 373–381 (2019)

Show all 23 references
  1. [9]

    Annals of Oncology 29(8), 1836–1842 (2018)

    Haenssle, H., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A.B.H., Thomas, L., Enk, A., et al.: Man against machine: diagnos- tic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in compariso...

  2. [10]

    Janda, M., Soyer, H.: Can clinical decision making be enhanced by artificial intel- ligence? British Journal of Dermatology 180(2), 247–248 (2019)

  3. [11]

    British Journal of Dermatology 166(5), 1069–1080 (2012)

    Lomas, A., Leonardi-Bee, J., Bath-Hextall, F.: A systematic review of worldwide incidence of nonmelanoma skin cancer. British Journal of Dermatology 166(5), 1069–1080 (2012)

  4. [12]

    In: Advances in Neural Information Processing Systems

    Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems. pp. 4765–4774 (2017)

  5. [13]

    Computerized Medical Imaging and Graphics 71, 19 – 29 (2019)

    Mahbod, A., Schaefer, G., Ellinger, I., Ecker, R., Pitiot, A., Wang, C.: Fusing fine- tuned deep features for skin lesion classification. Computerized Medical Imaging and Graphics 71, 19 – 29 (2019)

  6. [14]

    Mar, V., Soyer, H.: Artificial intelligence for melanoma diagnosis: How can we deliver on the promise? (2018)

  7. [15]

    (2019), christophm.github.io/interpretable-ml-book/

    Molnar, C.: Interpretable Machine Learning - A Guide for Making Black Box Mod- els explainable. (2019), christophm.github.io/interpretable-ml-book/

  8. [16]

    Australian family physi- cian 28(5), 466–471 (1999)

    Raasch, B.: Suspicious skin lesions and their management. Australian family physi- cian 28(5), 466–471 (1999)

  9. [17]

    Archives of dermatology 146(3), 283–287 (2010)

    Rogers, H.W., Weinstock, M.A., Harris, A.R., Hinckley, M.R., Feldman, S.R., Fleis- cher, A.B., Coldiron, B.M.: Incidence estimate of nonmelanoma skin cancer in the united states, 2006. Archives of dermatology 146(3), 283–287 (2010)

  10. [18]

    The Lancet 392(10151), 971–984 (2018)

    Schadendorf, D., van Akkooi, A.C., Berking, C., Griewank, K.G., Gutzmer, R., Hauschild, A., Stang, A., Roesch, A., Ugurel, S.: Melanoma. The Lancet 392(10151), 971–984 (2018)

  11. [19]

    In: Proceedings of the IEEE International Conference on Computer Vision

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 618– 626 (2017)

  12. [20]

    Shapley, L.S.: A value for n-person games 2(28), 307–317 (1953)

  13. [21]

    arXiv preprint arXiv:1904.07478 (2019)

    Simpson, B., Dutil, F., Bengio, Y., Cohen, J.P.: Gradmask: Reduce overfitting by regularizing saliency. arXiv preprint arXiv:1904.07478 (2019)

  14. [22]

    Scientific data 5, 180161 (2018)

    Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5, 180161 (2018)

  15. [23]

    IEEE transactions on image processing 13(4), 600–612 (2004)

    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality as- sessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.