{"id":"01da8888-ff4d-4dab-bc92-20d993d88f9c","arxiv_id":"2411.10322","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Merging public datasets and rejecting high-entropy predictions boosts reported melanoma detection accuracy to 97.8% and cuts misdiagnoses by over 40%, but lacks baseline comparisons and error bars.","lead":"This paper combines ten public skin-lesion datasets to train and calibrate 24 neural networks for melanoma detection, then rejects high-uncertainty predictions. After rejection, reported accuracy rises to 97.8% and misdiagnoses drop by over 40%, though these gains are on retained samples only.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-rejection accuracy and the >40.5% misdiagnosis reduction are not tested against a random-rejection baseline; at matched coverage, entropy-based rejection may simply be dropping easy cases.","rationale":"The paper builds a useful benchmark scaffold: merged public datasets, 1,296 configurations, and public code plus a web interface. The uncertainty module is standard Shannon-entropy rejection, and the calibration analysis is reasonable. However, the headline quantitative claim is a selective-prediction claim, and selective prediction has a well-known control: coverage-matched random rejection. Without that control, the improvement from 93.2% to 97.8% and the stated misdiagnosis reduction are not evidence for the uncertainty mechanism; they are only evidence that some retained subset is easier. The reader's verdict already flags threshold transfer and retained-only scoring, and my concern overlaps with that, but it is more fundamental because it challenges the causal attribution, not just the threshold calibration. The proposed test is simple and directly answers whether entropy rejection beats random rejection at matched coverage. Since the reader already returned CONDITIONAL and this concern adds a condition rather than overturning the framework, the verdict should remain UNCHANGED from the reader's evaluation.","tokens_in":7602,"tokens_out":3659,"duration_ms":37130,"concrete_test":"Reproduce the top Kaggle row from Table 1 (DenseNet201, trainsets A–D,I,J, threshold 0.19). On the Kaggle test set, reject samples by (a) the paper's entropy rule, (b) random selection of exactly the same number of samples, repeated 100 times, at rejection fractions of 5%, 10%, 15%, and 20%. Report accuracy, false positives, false negatives, and retained-set ECE/Brier at each coverage level. If entropy-based rejection is not significantly better than the random-rejection envelope, the 40.5% misdiagnosis reduction is not attributable to uncertainty quantification and the abstract should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that uncertainty-based rejection raises accuracy from 93.2% to 97.8% and reduces misdiagnoses by over 40.5%. Section 3.3 reports that thresholds are chosen on validation sets, after which 'we re-evaluate the remaining samples'; Table 1 and Fig. 5 report metrics only on retained samples and omit rejection counts or coverage. The central inference, that uncertainty quantification causes the improvement, requires a control that the same fraction of samples rejected at random does not produce comparable retained-set accuracy. Without such a control, the observed gains are consistent with a pure selection effect: hard or ambiguous cases are removed, so accuracy on the easy remainder rises regardless of whether the entropy score is informative. The claim is also evaluated under strong per-test-set selection (1,296 experiments, top rows shown), but the random-rejection baseline is the more load-bearing gap because it tests the mechanism rather than the ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a framework for melanoma detection that merges up to ten public dermoscopic datasets, trains 24 CNN architectures under a unified protocol (1,296 dataset/model combinations), calibrates classifiers using ECE and Brier score, and rejects predictions whose Shannon entropy exceeds a model-specific threshold tuned on validation data. On four external test sets (ISIC2017, ISIC2018, 7-point criteria, Kaggle), the selected top models achieve accuracy up to 93.2% before rejection and 97.8% after rejection, with a claimed reduction in misdiagnoses of over 40.5%. The authors also release code, data, and a client-side web interface.","tokens_in":7742,"tokens_out":7119,"duration_ms":62932,"significance":"If the improvements are real, the paper provides a useful and reproducible empirical study of uncertainty-based deferral for melanoma classification, with several commendable strengths: the 1,296-experiment sweep over public datasets, external test-set evaluation, calibration metrics, a public repository, and a deployed web interface. However, the central quantitative claims currently rest on (i) post hoc selection of best models, (ii) retained-only metrics without coverage or a random-rejection control, and (iii) single-seed results with no intervals. These are fixable, and the work would be valuable after those controls are added.","major_comments":[{"comment":"The central claim that uncertainty-based rejection causes the reported improvements is not yet established. Thresholds are tuned on validation data, but Table 1 reports 'after rejection' precision, sensitivity, specificity, F1, accuracy, and AUC only on the retained test samples, with no count or fraction of rejected samples given for any row. Since removing difficult samples mechanically raises accuracy on the remainder, the gains (e.g., 93.2% to 97.8% on Kaggle) are also fully consistent with a selection effect that would occur even for random rejection. The authors should report the retained-sample size/coverage for every configuration and compare against a random-rejection baseline at matched coverage, with confidence intervals. They should also specify how rejected cases are counted in the misdiagnosis-reduction claim in Fig. 5: if rejected cases are referred to an expert, the end-to-end misdiagnosis rate depends on referral accuracy, which is not modeled.","section":"Section 3.3, Table 1, Fig. 5"},{"comment":"The paper's headline numbers are selected post hoc. Figure 1 states that 1,296 experiments were run, and Table 1 shows only 'top-performing models' per test set, apparently chosen by precision on the same test sets. This selection invalidates the strongest point estimates as estimates of framework performance. The authors should report the full distribution (mean, median, range) of metrics across all 1,296 runs, or use a fixed model set, or split test sets so that model selection and evaluation are separate. Without this, the claim that the framework 'achieves' 97.8% accuracy is an optimistic maximum, not an expected performance.","section":"Section 4 and Table 1"},{"comment":"No statistical uncertainty is quantified. All experiments appear to be single runs; no random seeds, standard deviations, or confidence intervals are reported. Given stochastic SGD training and small test sets (7-point criteria has a few hundred images), differences of several percentage points between before/after rejection may not be significant. Please report at least three training seeds (or bootstrap confidence intervals) for the key metrics in Table 1 and Fig. 5.","section":"Sections 3.2–4"}],"minor_comments":[{"comment":"Please specify the base of the logarithm and the normalization. The claim that H(X) lies in [0,1] holds for binary classification only if log base 2 is used, but the equation uses an unspecified log.","section":"Section 2.3, Eq. (1)"},{"comment":"The entry 'VGG19 [19]' should cite reference [21] (VGG), not [19] (DenseNet).","section":"Table 3, ISIC2018 row"},{"comment":"The sentence 'ResNet152 improves precision by at least 10% points' is contradicted by Table 1, e.g., the ISIC2017 row [A–D,I,J] shows 82.0% to 87.0%, a gain of only 5 percentage points. Please rephrase to 'up to more than 10 percentage points' or report per-row gains.","section":"Section 4"},{"comment":"The caption says 'A plot comparing false diagnoses before and after applying uncertainty-based rejection across benchmarks,' but the y-axis label and legend are difficult to read. Please clarify what quantity is plotted (absolute counts or normalized rates) and for which model(s) and test sets.","section":"Figure 5"},{"comment":"The description of the 'Uncertain' class is brief. Please clarify whether the system outputs a three-way decision ('Melanoma,' 'Non-Melanoma,' 'Don't Know') and how that interacts with the binary softmax probabilities.","section":"Section 2.4"},{"comment":"Several cells have inconsistent spacing around slashes (e.g., '91.5% / 91.8%' versus '84.8% /94.1%') and some numbers are missing spaces. Please reformat for readability.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears intended for a conference or workshop. The main gap is the missing random-rejection control and the post hoc model selection, both of which are standard expectations for empirical medical-imaging papers. Since the code and data are public, adding these analyses should be feasible. The abstract's quantitative claims currently overstate the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ed — quick take on arXiv:2411.10322. The genuinely useful part is the benchmark: ten public melanoma datasets, 24 CNNs, 1,296 experiments, and code that actually ships. If you work on skin-lesion classification, this saves you a lot of data wrangling. The calibration comparison across single vs. combined datasets is also a nice practical data point.\n\nThe advertised result, that uncertainty-based rejection raises accuracy from 93.2% to 97.8% and cuts misdiagnoses by 40.5%, is where I get off the train. The paper tunes an entropy threshold per model on validation, then reports metrics only on the retained samples. There is no random-rejection baseline, no coverage number, no rejection counts. So the gains could come substantially from simply dropping hard cases. A control that rejects the same fraction at random would tell us how much of the improvement is actually due to entropy being informative. Without that, the 40.5% number in the abstract is not supported as stated.\n\nOther soft spots: the best model per test set is selected post hoc by precision, which is a form of selection bias; there are no confidence intervals or multiple seeds; and there is no comparison with other uncertainty methods like MC dropout or ensembles. These are fixable but they matter.\n\nWhat the paper does well is being an honest empirical scan with open code and a web demo. The observation that deeper networks like ResNet benefit more from rejection than shallower ones is interesting, though not deeply analyzed. The claim that combined datasets improve calibration is supported by their experiments.\n\nFor the right reader — someone building a melanoma-screening prototype or a benchmark for UQ in dermatology — this is a useful resource. But the central mechanistic claim needs more work. I would not desk-reject; I'd send it for review with a request for a random-rejection baseline, coverage reporting, and aggregate statistics across models. As it stands, it's a solid benchmark paper wearing a stronger causal claim on its abstract.","headline":"A useful benchmark and public toolkit, but the misdiagnosis-reduction claim lacks the random-rejection control needed to support it.","tokens_in":8259,"tokens_out":3215,"would_cite":false,"duration_ms":31262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By training on merged public datasets and rejecting uncertain predictions, this paper pushes melanoma detection accuracy from 93.2% to 97.8% while cutting misdiagnoses by over 40.5%.","keywords":["melanoma detection","uncertainty quantification","entropy-based rejection","model calibration","dataset merging","deep neural networks","medical imaging","skin lesion classification"],"falsifier":"Run the same pipeline on an independent clinical cohort with a different acquisition protocol, and check whether the validation-tuned threshold rejects a large fraction of true melanomas or whether expert review of rejected images fails to recover the classifier's missed melanomas; either observation would refute the claim that uncertainty rejection reduces misdiagnoses.","tokens_in":7378,"feed_emoji":"🩺","tokens_out":6397,"duration_ms":50613,"temperature":0.7,"pith_summary":"This paper argues that melanoma detection improves when classifiers are trained on a pool of merged public skin-lesion datasets and then filtered with an entropy-based uncertainty score. On four benchmark test sets, the authors report accuracy rising from up to 93.2% before rejection to 97.8% after rejection, with misdiagnoses falling by over 40.5%. The practical point is that a detector can be made safer by explicitly labeling low-confidence cases as 'Uncertain' and referring them to a human expert instead of forcing a binary yes/no answer.","feed_headline":"Uncertainty filtering lifts melanoma detection to 97.8% accuracy","feed_subtitle":"Rejecting low-confidence predictions from merged-dataset classifiers cuts misdiagnoses by over 40 percent.","key_machinery":"The central mechanism is Shannon entropy of the softmax probability vector, $H(X) = -\\sum_i p_i \\log p_i$, used as an uncertainty score. Predictions whose entropy exceeds a per-model threshold are rejected as 'Uncertain' and referred to a human expert; the thresholds are chosen on the validation set by minimizing the Expected Calibration Error and Brier score, and only the remaining confident predictions are evaluated. Empirically, rejecting up to 20% of low-confidence predictions improves accuracy.","core_discovery":"The central claim is that uncertainty-based rejection, applied to softmax classifiers trained on combined public datasets, substantially improves melanoma detection while also improving calibration. After calibrating models and rejecting the most uncertain predictions, accuracy on the ISIC2017, ISIC2018, 7-point criteria, and Kaggle test sets reaches as high as 97.8%, up from 93.2%, with large gains in precision and sensitivity and a reduction in misdiagnoses of over 40.5%. The paper also finds that combining multiple datasets improves both classification and calibration, and that shallower networks are often better calibrated than deeper ones.","pith_inferences":["The threshold-selection strategy could be stress-tested under distribution shift by tuning on one dataset and applying the fixed threshold to a dataset collected with a different camera or population; the paper does not report this experiment.","A prospective study that tracks whether dermatologists confirm the 'Uncertain' referrals would be the natural validation step for clinical adoption.","The same merge-and-reject pipeline could transfer to other imbalanced medical imaging tasks such as mammography, a direction the authors list as future work.","The reported 40.5% misdiagnosis reduction is computed on retained samples only; a full accounting that treats rejected samples as diagnosed-by-expert could change the headline number."],"forward_implications":["Any softmax classifier can be augmented with a third 'Uncertain' output without retraining, by thresholding the entropy of its probability vector.","Training on merged datasets improves calibration as well as accuracy, so confidence scores from multi-source training can be taken more literally.","Shallower networks such as EfficientNet and VGG often show better calibration than deeper networks, which matters for deployment where trust in confidence is critical.","Entropy-based rejection reduces both false positives and false negatives, including most false negatives on the Kaggle test set, which is the clinically dangerous error direction."],"supporting_citations":[{"why":"Supplies the ISIC2017 benchmark test set on which post-rejection accuracy rises to 96.3%.","marker":"[4]"},{"why":"Supplies the ISIC2018 benchmark test set on which post-rejection accuracy reaches 96.0%.","marker":"[6]"},{"why":"Supplies the 7-point criteria benchmark test set used in the rejection experiments.","marker":"[15]"},{"why":"Supplies the Kaggle benchmark on which the highest post-rejection accuracy of 97.8% is achieved.","marker":"[18]"},{"why":"Defines Expected Calibration Error, the calibration metric used to select per-model rejection thresholds.","marker":"[8]"},{"why":"Provides Shannon entropy, the uncertainty score used to reject low-confidence predictions.","marker":"[10]"},{"why":"Defines DenseNet201, one of the top-performing architectures in the reported benchmarks.","marker":"[19]"},{"why":"Defines ResNet152, the architecture showing the largest precision gains after rejection.","marker":"[20]"}],"fun_headline_variants":["Uncertainty-based rejection lifts melanoma accuracy to 97.8%","Melanoma detection: rejecting uncertain cases cuts misdiagnoses by 40.5%","How uncertainty quantification improves melanoma detection to 97.8%","Combining datasets and uncertainty filtering achieves 97.8% melanoma accuracy","Uncertainty filtering reduces melanoma misdiagnoses by over 40.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that a rejection threshold tuned on a validation set transfers to the test distribution, and that measuring accuracy only on the retained, non-rejected samples—with rejected cases referred to an expert—is a fair measure of real diagnostic improvement.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty-based rejection lifts melanoma accuracy to 97.8%","Melanoma detection: rejecting uncertain cases cuts misdiagnoses by 40.5%","How uncertainty quantification improves melanoma detection to 97.8%","Combining datasets and uncertainty filtering achieves 97.8% melanoma accuracy","Uncertainty filtering reduces melanoma misdiagnoses by over 40.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001443,"raw_usage":{"total_tokens":5725,"prompt_tokens":770,"completion_tokens":4955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":386,"completion_tokens_details":{"reasoning_tokens":4855}},"tokens_in":386,"tokens_out":4955,"duration_ms":29955,"temperature":1.0,"reasoning_tokens":4855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:43:54.507661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on an independent clinical cohort with a different acquisition protocol, and check whether the validation-tuned threshold rejects a large fraction of true melanomas or whether expert review of rejected images fails to recover the classifier's missed melanomas; either observation would refute the claim that uncertainty rejection reduces misdiagnoses.","supporting_citations":[{"cited_title":") * ' \" ) * % '","cited_arxiv_id":null,"evidence_quote":"Supplies the ISIC2017 benchmark test set on which post-rejection accuracy rises to 96.3%."},{"cited_title":"Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DLA or ARPA-H","cited_arxiv_id":null,"evidence_quote":"Supplies the ISIC2018 benchmark test set on which post-rejection accuracy reaches 96.0%."},{"cited_title":"End- to-end learning of brain tissue segmentation from imper- fect labeling,","cited_arxiv_id":null,"evidence_quote":"Supplies the Kaggle benchmark on which the highest post-rejection accuracy of 97.8% is achieved."},{"cited_title":"Melanoma,","cited_arxiv_id":null,"evidence_quote":"Defines Expected Calibration Error, the calibration metric used to select per-model rejection thresholds."},{"cited_title":"A patient-centric dataset of images and metadata for identifying melanomas using clinical context,","cited_arxiv_id":null,"evidence_quote":"Defines ResNet152, the architecture showing the largest precision gains after rejection."}],"review_version":1}