{"id":"3d90424f-053a-48f2-a8b9-0692d366c7c5","arxiv_id":"2412.20007","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Skin lesion segmentation Dice can be estimated from region-specific uncertainty via linear regression, but the reported predictive accuracy is in-sample and relies on ground-truth region masks.","lead":"Skin lesion segmentation models with Monte Carlo dropout and Bayes-by-backprop produce pixel-level uncertainty maps. The authors fit linear regressions that use these uncertainty values to estimate Dice scores, aiming to flag unreliable segmentations without a reference label.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ROI-based Dice prediction is not deployable because Algorithm 1 defines lesion/non-lesion ROIs from ground-truth masks, which are unavailable at inference.","rationale":"I agree with the reader's weakest assumption: the most load-bearing condition is that Algorithm 1's ROI uncertainties are defined via Xtesti-lesion = XGTi. In deployment there is no ground-truth mask, so Equations 3 and 4 cannot be applied to new dermoscopy images. The paper's own out-of-distribution discussion resorts to Eq. 5 overall uncertainty, implicitly acknowledging this limitation while still presenting the ROI-based models as the central contribution. This is more than a missing implementation detail: it creates a circular evaluation design, because when XGT is available Dice is directly computable and an uncertainty-based predictor is unnecessary. Other issues (dataset count inconsistency, no code/data release, potential sign inconsistency in the melanoma theta_b coefficient) reinforce caution but do not replace this principal concern. The appropriate verdict remains conditional: the paper can be salvaged by demonstrating a mask-free ROI estimation route with comparable predictive accuracy, or by carefully restricting the claim. Since my read does not move the reader's conditional verdict, I recommend UNCHANGED.","tokens_in":12313,"tokens_out":3754,"duration_ms":37158,"concrete_test":"Use the 588-image test set (or a new held-out set) and compute X1 and X2 twice: (a) from GT masks as in Algorithm 1, and (b) from the DLM's own predicted binary mask used as a surrogate ROI mask. Fit and evaluate Eq. 3 separately for each definition, reporting RMSE and Spearman rho per class. If version (b) does not achieve comparable RMSE (≤0.16 for nevus/melanoma, ≤0.19 for seborrheic keratosis), the ROI-based Dice prediction cannot be deployed without ground truth, and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central regression claim is unusable at inference because Algorithm 1 Step 2 defines the lesion ROI mask Xtesti-lesion as the ground-truth mask XGTi, and the non-lesion ROI as its complement. X1 and X2 in Eq. 3 are therefore not model-derived features: they require the exact labels that Dice already compares against. On an unlabeled clinical image, XGTi is absent, so neither X1 nor X2 can be computed; the authors' fallback for out-of-distribution data (Eq. 5) drops the ROI decomposition entirely and predicts Dice from global uncertainty only. Consequently, the four 'first-time' ROI-based linear models (Eqs. 3 and 4) are validated only in a regime where ground truth is available and Dice can be computed directly. That makes the claimed low-compute, ground-truth-free quality estimation unsubstantiated. The paper needs either a mask-free route to obtain lesion/non-lesion ROIs (e.g., the DLM's own predicted mask or a coarse localization) or an explicit restriction that ROI-based prediction applies only when labels exist.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains two families of segmentation networks on ISIC-2019 dermoscopy images (trained-from-scratch and transfer learning, each augmented with Monte Carlo dropout or Bayes-by-backprop), extracts pixel-level uncertainty maps, decomposes them into lesion and non-lesion clinical regions of interest using ground-truth masks, and fits four linear regression models that relate these ROI-based uncertainty features to the Dice score of the segmentation. The authors report statistically significant Spearman correlations and low RMSE for nevus, melanoma, and seborrheic keratosis classes, and claim a first-of-its-kind low-compute framework for predicting DLM segmentation performance without needing a reference standard.","tokens_in":12602,"tokens_out":6181,"duration_ms":60117,"significance":"If the central claims were valid, the framework would be a practically useful, lightweight quality-control tool for dermoscopy segmentation: a clinician could estimate Dice-like segmentation quality from uncertainty maps alone. The paper has real strengths: it uses a public benchmark (ISIC 2019), reports uncertainty visualizations, introduces a clinically motivated ROI decomposition of uncertainty, and formulates a falsifiable regression prediction target. However, the current validation does not support the deployment claim. The ROI masks are defined from ground-truth labels, which are unavailable at inference, and the regression models appear to be fit and evaluated on the same images. Both issues are load-bearing for the advertised label-free, out-of-sample Dice prediction.","major_comments":[{"comment":"The lesion and non-lesion ROI masks are defined as Xtesti-lesion = XGTi and its complement, Xtesti-non-lesion. On an unlabeled image at inference there is no ground-truth mask XGTi, so the ROI-specific uncertainty features X1 and X2 in Equations 3 and 4 cannot be computed. The only model that could be applied without labels is Equation 5, which uses overall uncertainty and drops the ROI decomposition. This directly contradicts the abstract's claim of ROI-based Dice prediction 'for the first time' in a low-compute, clinically usable workflow. Please add a mask-free ROI generation step (for example, using the DLM's own predicted segmentation mask or a coarse lesion localizer) and re-validate the regression models, or explicitly restrict the claims of Equations 3 and 4 to settings where labels are available.","section":"§III-C, Algorithm 1 Step 2"},{"comment":"The manuscript states an 80/20 split (about 2,347 training and 588 test images), but then reports regression fits and statistics 'based on the entire test set of 2,935 images' and 'for all 2,935 test images.' Since 20% of 2,935 is approximately 588, the text is internally inconsistent about which images were used. If the regression coefficients, RMSE values, and correlations in Section IV-A and Table III were computed by fitting and evaluating on the same images, the reported RMSE is an in-sample fit statistic and does not demonstrate predictive accuracy on unseen images. Please clarify the exact composition of the regression training and validation subsets, and provide out-of-sample RMSE and correlation values from a proper train/validation split or cross-validation.","section":"§III-B, §III-C, §IV-A"},{"comment":"The conclusion states that 'significant correlations (p < 0.05) between overall and ROI-based uncertainties with Dice scores were established using Spearman's rank correlation,' but Table III reports only Spearman's ρ without p-values or confidence intervals, and several ROI-specific ρ values are near zero (for example, nevus lesion ρ = -0.0243, melanoma non-lesion ρ = 0.0008, seborrhoeic non-lesion ρ = -0.1538). The claim of significant correlation for all four models is therefore not supported by the reported evidence. Please provide p-values and confidence intervals for each ρ, state which of the four models (Equations 3, 4(i), 4(ii), 5) are covered by each claim, and correct the text if not all correlations are significant.","section":"§IV-A, Table III, §VI"},{"comment":"The sentence 'Algorithm 1 estimates model performance unsupervised' is contradicted by Algorithm 1 Step 2, which requires the ground-truth mask XGTi to define the lesion and non-lesion ROIs and thereby to compute X1 and X2. As written, Algorithm 1 is supervised by labels. Please revise the description of the algorithm's inputs, or remove the 'unsupervised' characterization, since the label requirement is precisely the limitation that prevents deployment on unlabeled images.","section":"§VI, Conclusion"}],"minor_comments":[{"comment":"RMSE values are reported as thresholds ('≤ 0.16' or '≤ 0.19') without a statement of what quantity the RMSE is computed on or whether it is an in-sample or out-of-sample error; please state the definition and evaluation protocol.","section":"§IV-A1–3"},{"comment":"Equation (2) defines U nc_j_Mp = max(Mp(Xtesti)) for each MC iteration, while the text says the uncertainty is the difference between the 67th and 33rd percentiles of the sigmoid values; please clarify whether 'max' is applied to a sigmoid probability output or is a typographical error.","section":"§III-B, Eq. (2)"},{"comment":"For the tfs d model, the reported 95% confidence interval (0.8705–0.8802) does not contain the reported median Dice of 0.8809; please recheck the bootstrap CI calculation and the reported digits.","section":"§IV-A, Table I"},{"comment":"The text states that negative values of θb indicate that uncertainty decreases performance, but Table III reports a positive θb for the melanoma overall-uncertainty model (5.7393); this contradiction needs an explanation or a correction.","section":"§IV-A4, Table III"},{"comment":"The conclusion refers to 'whole slide image segmentation,' but the study uses dermoscopy images from ISIC-2019, not whole slide images; please use the correct imaging modality.","section":"§VI, Conclusion"},{"comment":"The section title says all data, code, and models are deposited in a GitHub repository, but the text says they 'will be made available upon request to the corresponding author'; please clarify whether the repository is publicly accessible or restricted.","section":"§VII, Data, Code and Model Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is potentially useful, but the ground-truth requirement in Algorithm 1 and the apparent in-sample regression evaluation are both load-bearing and cannot be resolved by text edits alone; they require additional experiments with mask-free ROI generation and a proper held-out regression validation. The 'for the first time' novelty claims should also be tempered in light of prior work (e.g., Ref. [9]) that already used linear regression with uncertainty metrics for segmentation quality assessment, though the ROI decomposition is a distinguishing element. The inconsistent use of '2,935 test images' after an 80/20 split will need careful clarification during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2412.20007. First, the useful kernel is real: the paper trains four segmentation models (MCD vs BBP crossed with TFS vs TL) on ISIC-19 and reports pixel-level uncertainty maps alongside Dice, AUROC, TPR, and FPR. That part is competent, and the qualitative comparison of where uncertainty lives—boundary versus interior—is worth a look. Second, the central claim—four linear regression models predict Dice from ROI-based uncertainty—does not hold as stated, because Algorithm 1 defines the lesion ROI as the ground-truth mask. At inference on unlabeled images, the lesion and non-lesion uncertainty features cannot be computed, so Equations 3 and 4 cannot be applied. The authors fall back to Equation 5 (overall uncertainty) for out-of-distribution data, but that quietly abandons the ROI decomposition that the abstract sells as the novelty. The stress-test note is right: this is label leakage, not a deployment-ready estimator.\n\nThe in-sample evaluation is the next soft spot. The regression coefficients appear to be fit and 'predicted' on the same 2,935 images; no held-out split for the regression is described. That makes the low RMSE values look better than they are. There are also numerical inconsistencies—the 80/20 split gives roughly 588 test images, yet the paper repeatedly says 2,935 test images, and the class-specific subsets sum to 605. Those need fixing. No code or data are available yet; the GitHub link says 'will be made available upon request,' so independent checking is limited.\n\nNone of these flaws make the paper worthless. The DLM segmentation numbers are plausible, the uncertainty maps look carefully generated, and splitting uncertainty into lesion and non-lesion regions is a reasonable interpretability tool—it just is not a label-free quality estimator in its current form. If the authors re-run the regression with a properly held-out set and replace the ground-truth ROI mask with a model-derived mask, or at least state the limitation clearly, the contribution becomes defensible.\n\nWho is this for? Someone working on uncertainty-based quality assessment in medical image segmentation will want to know this paper, mainly as a cautionary example of how label leakage can sneak into a pipeline. I would bring it to a reading group to talk about the deployment gap, but I would not cite it until the ROI issue is resolved. My recommendation: send it to peer review with a request for major revision—the empirical work is worth referee time, but the main claim needs to be reworked or explicitly scoped.","headline":"ROI-based Dice prediction is not label-free: Algorithm 1 uses ground-truth masks, so the main novelty doesn't survive deployment.","tokens_in":13106,"tokens_out":3767,"would_cite":false,"duration_ms":36708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dice scores can be predicted from pixel-level uncertainty in skin-lesion segmentation.","keywords":["Deep learning","Skin cancer","Uncertainty prediction","Image segmentation","Statistical modeling","Monte Carlo dropout","Bayes-by-backprop","Dermoscopy"],"falsifier":"Take a held-out set of dermoscopy images with ground-truth masks, fit Equation 3 on a training split using true masks, then for each held-out image compute $X_1$ and $X_2$ twice—once with the true mask and once with the model's predicted mask. If the predicted-mask version no longer tracks Dice with significant Spearman correlation and similarly low RMSE, the claim that ROI uncertainty predicts segmentation performance in unlabeled deployment fails. A second check is to re-fit the regression using uncertainty from fewer Monte Carlo iterations; if the correlation collapses, the result depends on the full 50-iteration protocol.","tokens_in":12085,"feed_emoji":"🩺","tokens_out":11958,"duration_ms":101546,"temperature":0.7,"pith_summary":"Deep learning segmentation of skin lesions usually reports one Dice score after the fact, with no indication of where the model is unsure. This paper tries to close that gap by predicting the Dice score from pixel-level uncertainty maps, computed with Monte Carlo dropout or Bayes-by-backprop and split into lesion and non-lesion clinical regions. Four linear regression models—using lesion uncertainty, non-lesion uncertainty, both, or overall uncertainty—are fitted for nevus, melanoma, and seborrheic keratosis classes, with reported Spearman correlations at $p<0.05$ and RMSE between about 0.16 and 0.19. If the relationship holds, a clinician could judge whether a segmentation is trustworthy from its uncertainty heatmap and direct attention toward high-uncertainty pixels.","feed_headline":"Dice scores predicted from skin-lesion segmentation uncertainty","feed_subtitle":"Four regression models tie pixel-level uncertainty in lesion and non-lesion regions to segmentation accuracy.","key_machinery":"The carrying mechanism is Algorithm 1, which converts a per-pixel uncertainty map into two region-level predictors. It takes the Hadamard (elementwise) product of the uncertainty map with the ground-truth lesion mask to get lesion-region uncertainty $X_1$, and with the complement of that mask to get non-lesion uncertainty $X_2$, then averages each product. Those predictors enter the linear models $Y_{\\text{Dice}} = \\alpha_0 + \\alpha_1 X_1 + \\alpha_2 X_2$, with one-predictor versions for each region and an overall-uncertainty version. The underlying uncertainty map is the inter-percentile range of 50 Monte Carlo dropout or Bayes-by-backprop sigmoid outputs, $P_{67} - P_{33}$, and segmentations are finalized by thresholding the mean Monte Carlo output at 0.95.","core_discovery":"The central discovery is a quantitative link between a segmentation network's uncertainty and its Dice score: per-pixel uncertainty, averaged separately over lesion and non-lesion regions, linearly predicts the Dice coefficient. Using 2,935 dermoscopy images from ISIC-19 and VGG-UNet models trained from scratch or by transfer learning, the paper reports four regression models—Equation 3 with both regional uncertainties, Equations 4(i)-4(ii) with each region alone, and Equation 5 with overall uncertainty. For nevus, melanoma, and seborrheic keratosis, most fitted coefficients are negative, indicating that higher regional uncertainty lowers predicted Dice, with Spearman correlations significant at $p<0.05$ and low root-mean-square error. The paper also finds that Monte Carlo dropout models achieved both higher Dice and higher uncertainty than Bayes-by-backprop models, with dropout uncertainty concentrated inside lesions and backpropagation uncertainty at lesion boundaries.","pith_inferences":["In real deployment on unlabeled images, the lesion and non-lesion masks of Algorithm 1 would have to come from the model's own predicted segmentation rather than from ground truth; the paper does not test that substitution, and validating it is the key step to making ROI-based Dice prediction label-free.","The regression coefficients are fitted to ISIC-19 data and to these particular VGG-UNet models, so transferring the framework to another dataset or backbone would likely require refitting, even if the linear functional form survives.","A direct extension would be to train the same regression on uncertainties computed from predicted masks and compare the coefficients and RMSE with the ground-truth-mask results; agreement would support using the framework as an unsupervised quality monitor.","The two-region decomposition applies beyond skin: any foreground/background segmentation task could produce the same lesion/non-lesion uncertainty split, with coefficients re-estimated per application."],"forward_implications":["If Dice is predictable from regional uncertainty, low-quality segmentations could be flagged from uncertainty heatmaps instead of by recomputing Dice against ground truth.","The fitted coefficients reveal which clinical region drives failures: lesion-region uncertainty matters most for melanoma and seborrheic keratosis, while non-lesion uncertainty matters most for nevus.","Monte Carlo dropout models combine higher Dice with higher uncertainty than Bayes-by-backprop models, so uncertainty magnitude alone is not a universal quality score and must be read through class- and architecture-specific coefficients.","A single linear model with class indicator variables predicts Dice across all 2,935 images with RMSE 0.1543, indicating the three class-specific models can be merged into one equation.","Equation 5, using only overall uncertainty, offers a fallback when region masks are unavailable, but it discards the lesion-versus-non-lesion information that the ROI models exploit."],"supporting_citations":[{"why":"Supplies Monte Carlo dropout as the Bayesian approximation used to produce pixel-level uncertainty maps.","marker":"[5]"},{"why":"Frames Bayesian-neural-network segmentation evaluation around Dice scores, the performance target the regression predicts.","marker":"[8]"},{"why":"Shows linear regression with uncertainty metrics for segmentation quality, the direct antecedent this paper extends to skin lesion regions.","marker":"[9]"},{"why":"Demonstrates correlation between Dice and uncertainty in myocardial segmentation, motivating the same link in dermoscopy.","marker":"[10]"},{"why":"Provides the ISIC-19 dermoscopy dataset and ground-truth binary masks used for training, evaluation, and regression fitting.","marker":"[17]"},{"why":"Describes the VGG-UNet training and preprocessing workflow reused for the from-scratch and transfer-learning models.","marker":"[19]"},{"why":"Supplies the VGG-16 architecture on which the modified encoder-decoder segmentation network is based.","marker":"[20]"}],"fun_headline_variants":["Uncertainty slopes predict Dice for skin lesion segmentation","Pixel uncertainty linearly forecasts Dice in dermoscopy","Lesion and tissue uncertainty predict segmentation Dice","Regression ties skin-lesion uncertainty to Dice scores","Low-compute regression predicts Dice from pixel uncertainty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the true lesion mask is available when it computes the lesion and non-lesion uncertainties $X_1$ and $X_2$; on a genuinely new image without ground truth, only the overall-uncertainty fallback remains, and the advertised region-based Dice prediction cannot be run as written.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty slopes predict Dice for skin lesion segmentation","Pixel uncertainty linearly forecasts Dice in dermoscopy","Lesion and tissue uncertainty predict segmentation Dice","Regression ties skin-lesion uncertainty to Dice scores","Low-compute regression predicts Dice from pixel uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1552,"prompt_tokens":982,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":499}},"tokens_in":598,"tokens_out":570,"duration_ms":6240,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:39:51.241185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of dermoscopy images with ground-truth masks, fit Equation 3 on a training split using true masks, then for each held-out image compute $X_1$ and $X_2$ twice—once with the true mask and once with the model's predicted mask. If the predicted-mask version no longer tracks Dice with significant Spearman correlation and similarly low RMSE, the claim that ROI uncertainty predicts segmentation performance in unlabeled deployment fails. A second check is to re-fit the regression using uncertainty from fewer Monte Carlo iterations; if the correlation collapses, the result depends on the full 50-iteration protocol.","supporting_citations":[{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning,","cited_arxiv_id":null,"evidence_quote":"Supplies Monte Carlo dropout as the Bayesian approximation used to produce pixel-level uncertainty maps."},{"cited_title":"Bayesian neural networks for uncertainty estimation of imaging biomarkers,","cited_arxiv_id":null,"evidence_quote":"Frames Bayesian-neural-network segmentation evaluation around Dice scores, the performance target the regression predicts."},{"cited_title":"An exploration of un- certainty information for segmentation quality assessment,","cited_arxiv_id":null,"evidence_quote":"Shows linear regression with uncertainty metrics for segmentation quality, the direct antecedent this paper extends to skin lesion regions."},{"cited_title":"Accuracy, uncertainty, and adaptability of automatic myocardial asl segmentation using deep cnn,","cited_arxiv_id":null,"evidence_quote":"Demonstrates correlation between Dice and uncertainty in myocardial segmentation, motivating the same link in dermoscopy."},{"cited_title":"ISIC Archive homepage,","cited_arxiv_id":null,"evidence_quote":"Provides the ISIC-19 dermoscopy dataset and ground-truth binary masks used for training, evaluation, and regression fitting."},{"cited_title":"A deep-learning toolkit for visualization and interpretation of segmented medical images,","cited_arxiv_id":null,"evidence_quote":"Describes the VGG-UNet training and preprocessing workflow reused for the from-scratch and transfer-learning models."}],"review_version":1}