{"id":"e36170fc-53d0-4a44-8558-cf25687b7867","arxiv_id":"1908.06612","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"Convolutional networks trained on dermoscopic images can be accurate yet rely on spurious cues such as dark image corners, and models with equal accuracy can produce very different explanations.","lead":"This paper trained 30 neural networks to detect melanoma from skin images and used two explanation tools, GradCAM and Kernel SHAP, to see how the models made decisions. It found that accurate models sometimes rely on irrelevant image features such as dark corners, and that models with the same accuracy can explain themselves very differently.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim assumes saliency maps faithfully reflect model decision-relevance; without a perturbation/faithfulness check, map differences across same-AUC models do not establish different rationales.","rationale":"The reader's weakest assumption identifies exactly the load-bearing condition: the interpretability maps must be faithful enough for human observers to infer which image features drove the classification. My stress-test agrees with that diagnosis. The paper's empirical observations about map differences are solid, and the authors are appropriately cautious in their Limitations section. However, the conclusion that models 'make predictions using completely different rationales' is stronger than what the sanity checks support. The missing step is a quantitative link between the maps and the models' actual predictive behavior. A perturbation-based fidelity test would provide that link: if deleting map-highlighted regions does not change predictions more than deleting random regions, then the maps are not trustworthy evidence about model reasoning, and the central interpretive claims would need to be downgraded to claims about the explanation methods rather than about the models. Since the reader's verdict is already CONDITIONAL and explicitly notes the need for lesion masks or expert annotations, this concern reinforces the stated condition rather than moving the verdict. No change to the verdict is needed.","tokens_in":6044,"tokens_out":4969,"duration_ms":56281,"concrete_test":"Run a deletion/insertion fidelity test on the 200-image balanced test set for the three AUC-85% models and for the Fig. 3 images. For each image and explanation method, order pixels/superpixels by attribution; progressively mask the top 10%, 20%, ..., 100% and record the model's predicted melanoma probability. Compare against random masking and against lesion-mask masking (manually annotate lesion borders for the specific examples or use ISIC 2017 segmentations where available). The spurious-correlation claim is confirmed only if masking the highlighted dark corner degrades model 2's probability substantially more than masking an equal-area random region; the different-rationales claim is confirmed only if deletion curves differ significantly across the same-AUC models and each is steeper than its random baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference, made in Sections 2.3-2.5 and reiterated in the conclusions, is that GradCAM and Kernel SHAP maps reveal the features that drive each model's predictions. The sanity checks in Section 2.4 show that the maps are reproducible, depend on model weights, and vary across same-AUC models. They do not show that the maps are faithful: a map can be sensitive to weights and still not track the features that actually move the prediction. The spurious-correlation example in Fig. 3 is interpreted by visual inspection of dark corners without a lesion-mask or perturbation baseline. The Limitations section appropriately notes that a map on a lesion cannot justify clinically meaningful correlations, but it still treats a map on a clinically irrelevant region as evidence of spurious weighting; that step requires a fidelity guarantee the sanity checks do not provide. Similarly, different maps across same-AUC models could reflect instability or noise in the explanation methods rather than different decision rationales. The claim that AUC alone is insufficient would still stand if maps are unreliable, but the specific evidence offered for it would not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains 30 Inception-based CNNs on the public HAM10000 dermoscopic dataset for binary melanoma versus naevus classification, reporting a mean AUC of 85%. It then applies two local interpretability methods, GradCAM and Kernel SHAP, to the trained models and performs three sanity checks: reproducibility, model-weight dependence via layer randomization, and sensitivity across three same-AUC models. The authors present example saliency maps, including one case where the highlighted region lies in the dark corners of an image, and argue that evaluating models by AUC alone gives limited insight because models of similar accuracy can produce different explanations. Code and trained models are released.","tokens_in":6258,"tokens_out":4076,"duration_ms":38727,"significance":"If the central interpretive claim is established, the paper makes a useful empirical contribution to medical AI evaluation: it provides a reproducible model suite, applies standard sanity checks to interpretability methods in a clinically relevant domain, and shows that AUC-matched models can produce visually different saliency maps. The release of code and trained models is a concrete strength that supports reproducibility. However, the paper's main conclusion goes beyond raw map differences to claims about latent decision rationales, and that step requires evidence the current experiments do not fully supply.","major_comments":[{"comment":"The inference that the model \"has learned to weight the class decision using a spurious correlation\" assumes that the GradCAM and Kernel SHAP maps faithfully reflect the features that drive the classifier's output. The sanity checks in Section 2.4 demonstrate reproducibility and weight dependence, but not fidelity; a map can depend on weights and still fail to track decision-relevant features. The manuscript should either add a faithfulness check (for example, occluding the highlighted regions and measuring the change in prediction probability) or consistently restrict the conclusion to the weaker, directly observable claim that the maps highlight clinically irrelevant regions.","section":"Section 2.5, Fig. 3"},{"comment":"The sensitivity test compares maps from three same-AUC models and reports an average SSIM difference of 20%, but it does not establish that the differences exceed the intrinsic stochasticity of the explanation methods. In Section 2.4(i), Kernel SHAP maps have mean SSIM 0.92 with standard deviation 0.028 when run twice on the same model and image, so part of the observed 20% variation could be method noise rather than a difference in model rationale. The authors should report the sensitivity statistic relative to the reproducibility baseline and, ideally, perform repeated runs of each method on each model.","section":"Section 2.4, sensitivity experiment"},{"comment":"The spurious-correlation analysis is based on visual inspection of a small number of selected images, and the claimed frequency (\"occasionally\") is never quantified across the 30 models and the test set. Because the Limitations section correctly warns that visual inspection will become impractical at scale, the paper should specify how many model-image pairs produced maps on clinically irrelevant regions, ideally with a lesion-segmentation or background mask and a simple quantitative overlap criterion.","section":"Section 2.5 and Section 2.6"}],"minor_comments":[{"comment":"The statement that the mean AUC has \"a variance of 1.8%\" is ambiguous; please report the standard deviation or a confidence interval with explicit units.","section":"Section 2.2"},{"comment":"The description of Kernel SHAP as needing to account for all 2^{d-1} subsets is correct for exact Shapley values but potentially misleading for the approximate Kernel SHAP estimator actually used; consider clarifying the distinction.","section":"Section 2.3"},{"comment":"Using \"confidence\" for softmax probabilities is imprecise, since modern CNNs are often miscalibrated; consider saying \"prediction probability.\"","section":"Fig. 3 caption"},{"comment":"The title asks \"Deep neural network or dermatologist?\" but the paper does not include a head-to-head comparison with dermatologists; the human evaluation in Section 2.2 involves three primary care physicians on 17 images. Consider either adding such a comparison or retitling to reflect the interpretability focus.","section":"Title and Introduction"},{"comment":"The text contains a typo in the phrase \"for both GradCShrapenlAM and kernel SHAP\"; this should read \"GradCAM.\"","section":"Section 2.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read 1908.06612. The thing to know: it's a reproducible, honest empirical study of GradCAM and Kernel SHAP for melanoma detection on HAM10000, and the key finding—that models with the same AUC can produce very different saliency maps, including one that clearly keys on dark image corners—is a useful cautionary result for anyone trying to deploy deep learning in dermatology. The authors trained 30 CNNs, released code and weights, and ran the standard Adebayo sanity checks: reproducibility, model dependence, and sensitivity to same-AUC models. That's real work, and the negative result (AUC alone tells you little about what the model is doing) is credible.\n\nThe soft spot is the leap from 'different saliency maps' to 'different rationales.' The sanity checks show the maps depend on the model weights and are reproducible, but they don't establish fidelity—a map can be sensitive to weights and still not track the features that actually drive the prediction. The dark-corner example in Fig. 3 is visually striking, but there's no lesion-mask or perturbation experiment to confirm the model's prediction actually relies on those corners. The authors' own Limitations section says a map on a lesion can't justify clinically meaningful correlations, yet they still treat a map on a clinically irrelevant region as evidence of spurious weighting. That inference needs a fidelity guarantee the checks don't provide. The conclusion language 'completely different rationales' is stronger than what the data support; different maps could in principle be noise or explainer instability.\n\nThat said, the central cautionary claim holds up. If the maps are unreliable, that's even more reason not to evaluate a model by AUC alone. The paper is appropriately hedged in most places, and the reproducibility is a genuine asset.\n\nFor whom: researchers in XAI for medical imaging, and anyone who has to decide whether a model with a high AUC is safe to deploy. It's a modest but solid contribution, not a breakthrough. I'd send it to a serious referee; it deserves a review rather than a desk reject. The authors should tighten the 'rationales' language and, if possible, add a perturbation test with lesion masks to back up the spurious-correlation claim.","headline":"A reproducible, honest empirical study showing that high-AUC melanoma classifiers can rely on spurious cues like image corners, but the saliency-map evidence for that claim has a faithfulness gap; still worth a serious referee.","tokens_in":6770,"tokens_out":2475,"would_cite":true,"duration_ms":23898,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Melanoma-classifying neural networks can score high accuracy while relying on image artifacts rather than lesion features.","keywords":["melanoma detection","interpretability","GradCAM","Kernel SHAP","saliency maps","deep learning","dermoscopy images","model evaluation"],"falsifier":"Take one of the models whose saliency map highlights the dark corners of a dermoscopy image, ablate or mask that corner region, and re-run the prediction. If removing the highlighted region leaves the model's output essentially unchanged, the map did not reveal a causally important feature, and the claim that the model weights the decision using spurious correlations is undermined. Conversely, if a same-AUC model whose map highlights the lesion keeps its prediction after the mask while the corner-fixated model flips, the paper's reading is supported.","tokens_in":5863,"feed_emoji":"🩺","tokens_out":4226,"duration_ms":41772,"temperature":0.7,"pith_summary":"This paper asks whether two interpretability tools, GradCAM and Kernel SHAP, can show why a deep network calls a skin lesion melanoma. The authors train 30 convolutional networks on the public HAM10000 dataset to distinguish melanoma from benign moles, then inspect the resulting saliency maps. They find that accuracy is not the whole story: models with the same AUC sometimes make decisions using different image features, and some models weight the classification using irrelevant regions such as dark corners of the image. The upshot is that reporting AUC alone can hide what a model has actually learned, and saliency maps need further validation before they can guide clinical use.","feed_headline":"High-accuracy melanoma AI can lean on image artifacts","feed_subtitle":"Same-AUC models used different image features; accuracy alone hides what a diagnostic model really learns.","key_machinery":"The load-bearing tools are the two local interpretability maps. GradCAM weights the final convolutional feature maps by the gradient of the class score, producing a heatmap of pixels that drove the melanoma decision. Kernel SHAP approximates Shapley values from cooperative game theory, attributing to each pixel region its fair contribution to the prediction. The paper uses these maps on 30 Inception-style transfer-learned models, and uses randomization tests—shuffling progressively shallower layer weights and checking that the maps change—to confirm the maps carry model-specific information. The maps are what connect the observed AUC differences to claims about spurious correlations.","core_discovery":"The authors establish that, within this training setup, interpretability maps expose failure modes that accuracy metrics miss: 17 test images were misclassified by at least 25 of 30 models, and saliency maps from models with identical AUCs of 85% differed by an average structural similarity (SSIM) of 20%, with one model relying on dark image corners to call a lesion melanoma. They also show that GradCAM and Kernel SHAP pass basic sanity checks—reproducible output and dependence on model weights—so the differing explanations are not merely noise. The conclusion is that similar accuracy does not imply similar reasoning, and interpretability screening should accompany accuracy reporting for skin-cancer classifiers.","pith_inferences":["The dark-corner reliance suggests a testable fix: cropping or masking image corners before training could measure how much of the reported AUC depends on artifacts.","Models with identical AUC but divergent saliency maps could be combined into ensembles to reduce the influence of any single spurious cue, an extension the paper mentions only for future accuracy gains.","A stricter validation of these maps would require causal intervention, such as ablating highlighted regions and checking that predictions change accordingly; the sanity checks here establish model dependence, not causal faithfulness."],"forward_implications":["Reporting AUC alone is insufficient to characterize a melanoma-detection model; equal-AUC models can rely on different features.","Saliency maps located on clinically irrelevant regions, such as the image border, can flag predictions that should not be trusted even when the model is confident.","Seventeen consistently misclassified test images, mostly also missed by primary care physicians, identify a concrete hard subset for future training and evaluation.","Interpretability methods that pass weight-randomization sanity checks are not guaranteed to be clinically meaningful; further validation is required before deployment.","Because the suite of 30 models and the training code are publicly released, the result is reproducible and can be extended to other architectures and datasets."],"supporting_citations":[{"why":"Supplies the HAM10000 dataset of dermoscopy images from which all models are trained and tested.","marker":"[22]"},{"why":"Defines GradCAM, one of the two interpretability methods whose saliency maps are analyzed.","marker":"[19]"},{"why":"Defines Kernel SHAP, the Shapley-based interpretability method used throughout the paper.","marker":"[12]"},{"why":"Provides the weight-randomization sanity-check methodology that motivates the paper's model dependence tests.","marker":"[2]"},{"why":"Provides the dermatologist AUC baseline of about 79% and a comparison CNN AUC of 86% used to contextualize model performance.","marker":"[9]"},{"why":"Establishes the dermatologist-level classification benchmark that this work builds on and interrogates.","marker":"[7]"},{"why":"Supplies the SSIM metric used to quantify similarity and difference between saliency maps in the sanity checks.","marker":"[23]"}],"fun_headline_variants":["Same accuracy, different reasoning in melanoma AI","Melanoma AI models match accuracy but not logic","AI melanoma detection: similar scores, divergent clues","When melanoma AI is right for the wrong reasons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the saliency maps produced by GradCAM and Kernel SHAP faithfully reveal which image features actually drove each prediction, not just that the maps change when model weights change.","fun_headline_variants_meta":{"raw":{"variants":["Same accuracy, different reasoning in melanoma AI","Melanoma AI models match accuracy but not logic","AI melanoma detection: similar scores, divergent clues","When melanoma AI is right for the wrong reasons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1264,"prompt_tokens":903,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":519,"tokens_out":361,"duration_ms":4560,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:38:53.090924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the models whose saliency map highlights the dark corners of a dermoscopy image, ablate or mask that corner region, and re-run the prediction. If removing the highlighted region leaves the model's output essentially unchanged, the map did not reveal a causally important feature, and the claim that the model weights the decision using spurious correlations is undermined. Conversely, if a same-AUC model whose map highlights the lesion keeps its prediction after the mask while the corner-fixated model flips, the paper's reading is supported.","supporting_citations":[{"cited_title":"Scientific data 5, 180161 (2018)","cited_arxiv_id":null,"evidence_quote":"Supplies the HAM10000 dataset of dermoscopy images from which all models are trained and tested."},{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Defines Kernel SHAP, the Shapley-based interpretability method used throughout the paper."},{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Provides the weight-randomization sanity-check methodology that motivates the paper's model dependence tests."},{"cited_title":"Annals of Oncology 29(8), 1836–1842 (2018)","cited_arxiv_id":null,"evidence_quote":"Provides the dermatologist AUC baseline of about 79% and a comparison CNN AUC of 86% used to contextualize model performance."}],"review_version":1}