REVIEW 4 major objections 6 minor 20 references
Uncertainty Quantified Deep Learning and Regression Analysis Framework for Image Segmentation of Skin Cancer Lesions
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Dice scores can be predicted from pixel-level uncertainty in skin-lesion segmentation.
desk verdict ROI-based Dice prediction is not label-free: Algorithm 1 uses ground-truth masks, so the main novelty doesn't survive deployment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is Algorithm 1, which converts a per-pixel uncertainty map into two region-level predictors. It takes the Hadamard (elementwise) product of the uncertainty map with the ground-truth lesion mask to get lesion-region uncertainty $X_1$, and with the complement of that mask to get non-lesion uncertainty $X_2$, then averages each product. Those predictors enter the linear models $Y_{\text{Dice}} = \alpha_0 + \alpha_1 X_1 + \alpha_2 X_2$, with one-predictor versions for each region and an overall-uncertainty version. The underlying uncertainty map is the inter-percentile range of 50 Monte Carlo dropout or Bayes-by-backprop sigmoid outputs, $P_{67} - P_{33}$, and segmentations are finalized by thresholding the mean Monte Carlo output at 0.95.
What would settle it
Take a held-out set of dermoscopy images with ground-truth masks, fit Equation 3 on a training split using true masks, then for each held-out image compute $X_1$ and $X_2$ twice—once with the true mask and once with the model's predicted mask. If the predicted-mask version no longer tracks Dice with significant Spearman correlation and similarly low RMSE, the claim that ROI uncertainty predicts segmentation performance in unlabeled deployment fails. A second check is to re-fit the regression using uncertainty from fewer Monte Carlo iterations; if the correlation collapses, the result depends on the full 50-iteration protocol.
Extended reading notes
Core claim
The central discovery is a quantitative link between a segmentation network's uncertainty and its Dice score: per-pixel uncertainty, averaged separately over lesion and non-lesion regions, linearly predicts the Dice coefficient. Using 2,935 dermoscopy images from ISIC-19 and VGG-UNet models trained from scratch or by transfer learning, the paper reports four regression models—Equation 3 with both regional uncertainties, Equations 4(i)-4(ii) with each region alone, and Equation 5 with overall uncertainty. For nevus, melanoma, and seborrheic keratosis, most fitted coefficients are negative, indicating that higher regional uncertainty lowers predicted Dice, with Spearman correlations significant at $p<0.05$ and low root-mean-square error. The paper also finds that Monte Carlo dropout models achieved both higher Dice and higher uncertainty than Bayes-by-backprop models, with dropout uncertainty concentrated inside lesions and backpropagation uncertainty at lesion boundaries.
Load-bearing premise
The framework assumes the true lesion mask is available when it computes the lesion and non-lesion uncertainties $X_1$ and $X_2$; on a genuinely new image without ground truth, only the overall-uncertainty fallback remains, and the advertised region-based Dice prediction cannot be run as written.
Editorial extensions
If this is right
- If Dice is predictable from regional uncertainty, low-quality segmentations could be flagged from uncertainty heatmaps instead of by recomputing Dice against ground truth.
- The fitted coefficients reveal which clinical region drives failures: lesion-region uncertainty matters most for melanoma and seborrheic keratosis, while non-lesion uncertainty matters most for nevus.
- Monte Carlo dropout models combine higher Dice with higher uncertainty than Bayes-by-backprop models, so uncertainty magnitude alone is not a universal quality score and must be read through class- and architecture-specific coefficients.
- A single linear model with class indicator variables predicts Dice across all 2,935 images with RMSE 0.1543, indicating the three class-specific models can be merged into one equation.
- Equation 5, using only overall uncertainty, offers a fallback when region masks are unavailable, but it discards the lesion-versus-non-lesion information that the ROI models exploit.
Reading between the lines
- In real deployment on unlabeled images, the lesion and non-lesion masks of Algorithm 1 would have to come from the model's own predicted segmentation rather than from ground truth; the paper does not test that substitution, and validating it is the key step to making ROI-based Dice prediction label-free.
- The regression coefficients are fitted to ISIC-19 data and to these particular VGG-UNet models, so transferring the framework to another dataset or backbone would likely require refitting, even if the linear functional form survives.
- A direct extension would be to train the same regression on uncertainties computed from predicted masks and compare the coefficients and RMSE with the ground-truth-mask results; agreement would support using the framework as an unsupervised quality monitor.
- The two-region decomposition applies beyond skin: any foreground/background segmentation task could produce the same lesion/non-lesion uncertainty split, with coefficients re-estimated per application.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper trains two families of segmentation networks on ISIC-2019 dermoscopy images (trained-from-scratch and transfer learning, each augmented with Monte Carlo dropout or Bayes-by-backprop), extracts pixel-level uncertainty maps, decomposes them into lesion and non-lesion clinical regions of interest using ground-truth masks, and fits four linear regression models that relate these ROI-based uncertainty features to the Dice score of the segmentation. The authors report statistically significant Spearman correlations and low RMSE for nevus, melanoma, and seborrheic keratosis classes, and claim a first-of-its-kind low-compute framework for predicting DLM segmentation performance without needing a reference standard.
Significance. If the central claims were valid, the framework would be a practically useful, lightweight quality-control tool for dermoscopy segmentation: a clinician could estimate Dice-like segmentation quality from uncertainty maps alone. The paper has real strengths: it uses a public benchmark (ISIC 2019), reports uncertainty visualizations, introduces a clinically motivated ROI decomposition of uncertainty, and formulates a falsifiable regression prediction target. However, the current validation does not support the deployment claim. The ROI masks are defined from ground-truth labels, which are unavailable at inference, and the regression models appear to be fit and evaluated on the same images. Both issues are load-bearing for the advertised label-free, out-of-sample Dice prediction.
major comments (4)
- [§III-C, Algorithm 1 Step 2] The lesion and non-lesion ROI masks are defined as Xtesti-lesion = XGTi and its complement, Xtesti-non-lesion. On an unlabeled image at inference there is no ground-truth mask XGTi, so the ROI-specific uncertainty features X1 and X2 in Equations 3 and 4 cannot be computed. The only model that could be applied without labels is Equation 5, which uses overall uncertainty and drops the ROI decomposition. This directly contradicts the abstract's claim of ROI-based Dice prediction 'for the first time' in a low-compute, clinically usable workflow. Please add a mask-free ROI generation step (for example, using the DLM's own predicted segmentation mask or a coarse lesion localizer) and re-validate the regression models, or explicitly restrict the claims of Equations 3 and 4 to settings where labels are available.
- [§III-B, §III-C, §IV-A] The manuscript states an 80/20 split (about 2,347 training and 588 test images), but then reports regression fits and statistics 'based on the entire test set of 2,935 images' and 'for all 2,935 test images.' Since 20% of 2,935 is approximately 588, the text is internally inconsistent about which images were used. If the regression coefficients, RMSE values, and correlations in Section IV-A and Table III were computed by fitting and evaluating on the same images, the reported RMSE is an in-sample fit statistic and does not demonstrate predictive accuracy on unseen images. Please clarify the exact composition of the regression training and validation subsets, and provide out-of-sample RMSE and correlation values from a proper train/validation split or cross-validation.
- [§IV-A, Table III, §VI] The conclusion states that 'significant correlations (p < 0.05) between overall and ROI-based uncertainties with Dice scores were established using Spearman's rank correlation,' but Table III reports only Spearman's ρ without p-values or confidence intervals, and several ROI-specific ρ values are near zero (for example, nevus lesion ρ = -0.0243, melanoma non-lesion ρ = 0.0008, seborrhoeic non-lesion ρ = -0.1538). The claim of significant correlation for all four models is therefore not supported by the reported evidence. Please provide p-values and confidence intervals for each ρ, state which of the four models (Equations 3, 4(i), 4(ii), 5) are covered by each claim, and correct the text if not all correlations are significant.
- [§VI, Conclusion] The sentence 'Algorithm 1 estimates model performance unsupervised' is contradicted by Algorithm 1 Step 2, which requires the ground-truth mask XGTi to define the lesion and non-lesion ROIs and thereby to compute X1 and X2. As written, Algorithm 1 is supervised by labels. Please revise the description of the algorithm's inputs, or remove the 'unsupervised' characterization, since the label requirement is precisely the limitation that prevents deployment on unlabeled images.
minor comments (6)
- [§IV-A1–3] RMSE values are reported as thresholds ('≤ 0.16' or '≤ 0.19') without a statement of what quantity the RMSE is computed on or whether it is an in-sample or out-of-sample error; please state the definition and evaluation protocol.
- [§III-B, Eq. (2)] Equation (2) defines U nc_j_Mp = max(Mp(Xtesti)) for each MC iteration, while the text says the uncertainty is the difference between the 67th and 33rd percentiles of the sigmoid values; please clarify whether 'max' is applied to a sigmoid probability output or is a typographical error.
- [§IV-A, Table I] For the tfs d model, the reported 95% confidence interval (0.8705–0.8802) does not contain the reported median Dice of 0.8809; please recheck the bootstrap CI calculation and the reported digits.
- [§IV-A4, Table III] The text states that negative values of θb indicate that uncertainty decreases performance, but Table III reports a positive θb for the melanoma overall-uncertainty model (5.7393); this contradiction needs an explanation or a correction.
- [§VI, Conclusion] The conclusion refers to 'whole slide image segmentation,' but the study uses dermoscopy images from ISIC-2019, not whole slide images; please use the correct imaging modality.
- [§VII, Data, Code and Model Availability] The section title says all data, code, and models are deposited in a GitHub repository, but the text says they 'will be made available upon request to the corresponding author'; please clarify whether the repository is publicly accessible or restricted.
Circularity Check
ROI-based Dice 'prediction' uses ground-truth masks to define its input features, and the regression coefficients are fit on the same test images whose Dice they then report.
-
self definitional
[Algorithm 1, Steps 1-2 (Section III-C)]
"Step 2: Generate ROI Uncertainty: Input: Test image Xtesti and ground-truth mask XGTi. • Xtesti-lesion = XGTi, where '0' represents non-lesion pixels, '1' represents lesion pixels. • Xtesti-non-lesion = Xtesti-lesionC, complement of Xtesti-lesion... Step 1: ... XDicei = Dice(OMi , XGTi)"
The regression inputs X1 and X2 are mean uncertainties masked by Xtesti-lesion = XGTi and its complement, while the dependent variable Y_Dice = Dice(OMi, XGTi) uses the same ground-truth mask. Thus Equation 3 does not predict Dice from model-derived features alone; it regresses Dice on quantities that already require the exact labels Dice is compared against. On an unlabeled image XGTi is absent, so X1 and X2 cannot be computed, and the claimed label-free ROI-based Dice prediction cannot be evaluated. The fallback in Section V, which uses Equation 5 where ground truth is missing, confirms that only overall uncertainty, not the four ROI models, survives without ground truth.
-
fitted input called prediction
[Section IV-A4 and Section III-C (Equations 3-5)]
"Solving for the values of these coefficients based on the entire test set of 2,935 images, α0 = 0.8852, α1 = -7.5594, α2 = -2.6110, ϕ1 = 0.06, ϕ2 = -0.0229 and ϕ3 = -0.0372 were calculated. A low RMSE of 0.1543 was calculated for this linear regression model."
The coefficients of the linear 'prediction' model are solved on the entire test set, and the low RMSE is reported on the same images whose Dice values form the dependent variable. This is in-sample goodness of fit, not a forecast of unseen segmentation quality. The same pattern appears for the class-wise coefficients in Table III, whose RMSE values are reported on the images used to fit them. Presenting this fitting error as 'predictive' performance and as a 'low-compute uncertainty estimation workflow' converts an in-sample regression summary into an out-of-sample capability claim without a held-out evaluation.
1 more flagged steps
-
other
[Section VI (Conclusion) and Section V (Discussion)]
"Algorithm 1 estimates model performance unsupervised, generating uncertainty maps and mean uncertainty values. ... When tested with OoD data lacking true ground-truth, the study employs historical performance metrics such as Dice scores and AUROC (Table I and II) to evaluate model effectiveness, utilizing Equation 5 where appropriate."
Algorithm 1's own Step 2 requires the ground-truth mask XGTi as input, so the conclusion that it 'estimates model performance unsupervised' is contradicted by the algorithm's definition. The Discussion passage concedes that, for OoD data without ground truth, only Equation 5 is used, which is the overall-uncertainty model rather than the four ROI-based models being advertised. This confirms that the central ROI-based Dice prediction does not operate without labels; what remains label-free is only the overall-uncertainty regression, not the claimed first-time ROI-specific predictive framework.
full rationale
The paper's central novelty is four linear regression models that 'predict' Dice segmentation performance from region-specific uncertainties (Equations 3 to 5). The key circularity is definitional: Algorithm 1 Step 2 sets the lesion ROI mask equal to the ground-truth mask XGTi and the non-lesion ROI as its complement, so the features X1 and X2 are computed by averaging the DLM's uncertainty over the same labels used to define the target variable Y_Dice = Dice(OMi, XGTi). At inference on unlabeled images, XGTi is absent, so X1 and X2 cannot be formed; the paper itself restricts ground-truth-free OoD use to Equation 5, which uses only overall uncertainty rather than the four ROI-based models. The claim that Algorithm 1 works 'unsupervised' is therefore not supported by the algorithm's own inputs. Additionally, the regression coefficients for the expanded Equation 3 are solved 'based on the entire test set of 2,935 images' and the reported RMSE is computed on the same set, making the reported accuracy an in-sample fit rather than an independent predictive validation. No load-bearing self-citation chain is present; the problem is not citation reliance but the construction of the predictive features from the target labels and the in-sample evaluation of the fitted models. A mask-free route to lesion and non-lesion ROIs, or an explicit restriction of the ROI models to labeled data, would be needed to make the advertised prediction claim non-circular. For these reasons, the central ROI-based Dice prediction partially reduces by construction, warranting a score of 7 rather than a lower score.
Assumptions & free parameters
free parameters (6)
- Linear regression coefficients for Equation 3 (alpha0, alpha1, alpha2) by class =
Nevus: 0.8449, -1.1932, -1.5108. Melanoma: 0.9644, -8.6780, -3.0881. Seborrheic: 0.8835, -8.4215, -4.7513.
- Linear regression coefficients for Equations 4 and 5 (beta, gamma, theta) =
Reported in Table III for each class and ROI.
- Class dummy coefficients (phi1, phi2, phi3) =
0.06, -0.0229, -0.0372
- Number of Monte Carlo iterations (alpha = 50) =
50
- Binarization threshold for predicted segmentation (0.95) =
0.95
- Percentile pair for uncertainty (33rd and 67th) =
P33 and P67
assumptions (5)
- domain assumption Monte Carlo dropout approximates Bayesian inference for segmentation uncertainty.
- domain assumption ISIC 2019 ground-truth masks are clinically correct and reliable.
- ad hoc to paper Dice score is linearly related to ROI mean uncertainty.
- ad hoc to paper The 67th minus 33rd percentile of per-pixel sigmoid outputs is a valid pixel-level uncertainty measure.
- domain assumption Preprocessing and training details from the cited prior paper [19] are valid and sufficient.
Cite this review
Pith. "Pith review of Uncertainty Quantified Deep Learning and Regression Analysis Framework for Image Segmentation of Skin Cancer Lesions." pith.science (2026). https://pith.science/paper/ALDZW4JN
@misc{pith2026241220007,
author = {Pith},
title = {Pith review of: Uncertainty Quantified Deep Learning and Regression Analysis Framework for Image Segmentation of Skin Cancer Lesions},
year = {2026},
howpublished = {\url{https://pith.science/paper/ALDZW4JN}},
note = {Machine review of arXiv:2412.20007}
}
read the original abstract
Deep learning models (DLMs) frequently achieve accurate segmentation and classification of tumors from medical images. However, DLMs lacking feedback on their image segmentation mechanisms, such as Dice coefficients and confidence in their performance, face challenges when processing previously unseen images in real-world clinical settings. Uncertainty estimates to identify DLM predictions at the cellular or single-pixel level that require clinician review can enhance trust. However, their deployment requires significant computational resources. This study reports two DLMs, one trained from scratch and another based on transfer learning, with Monte Carlo dropout or Bayes-by-backprop uncertainty estimations to segment lesions from the publicly available The International Skin Imaging Collaboration-19 dermoscopy image database with cancerous lesions. A novel approach to compute pixel-by-pixel uncertainty estimations of DLM segmentation performance in multiple clinical regions from a single dermoscopy image with corresponding Dice scores is reported for the first time. Image-level uncertainty maps demonstrated correspondence between imperfect DLM segmentation and high uncertainty levels in specific skin tissue regions, with or without lesions. Four new linear regression models that can predict the Dice performance of DLM segmentation using constants and uncertainty measures, either individually or in combination from lesions, tissue structures, and non-tissue pixel regions critical for clinical diagnosis and prognostication in skin images (Spearman's correlation, p < 0.05), are reported for the first time for low-compute uncertainty estimation workflows.
Figures
Reference graph
Works this paper leans on
-
[1]
Skin Cancer Foundation., “Skin cancer facts page,” https: //www.skincancer.org/skin-cancer-information/skin-cancer-facts/, 2024, accessed February 2024
work page 2024
-
[2]
American Academy of dermatology Assosciation., “Skin cancer facts page,” https://www.aad.org/media/stats-skin-cancer, 2022, accessed 22- April-2022
work page 2022
-
[3]
M. Goyal, T. Knackstedt, S. Yan, and S. Hassanpour, “Artificial intelligence-based image classification methods for diagnosis of skin cancer: Challenges and opportunities,” Computers in biology and medicine, vol. 127, p. 104065, 2020
work page 2020
-
[4]
Current applications and future impact of machine learning in radiology,
G. Choy, O. Khalilzadeh, M. Michalski, S. Do, A. E. Samir, O. S. Pianykh, J. R. Geis, P. V . Pandharipande, J. A. Brink, and K. J. Dreyer, “Current applications and future impact of machine learning in radiology,” Radiology, vol. 288, no. 2, pp. 318–328, 2018
work page 2018
-
[5]
Dropout as a bayesian approximation: Representing model uncertainty in deep learning,
Y . Gal and Z. Ghahramani., “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning . PMLR, 2016, pp. 1050–1059
work page 2016
-
[6]
C. M. Bishop., “Bayesian neural networks,” Journal of the Brazilian Computer Society, vol. 4, pp. 61–68, 1997
work page 1997
-
[7]
The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,
P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific data, vol. 5, no. 1, pp. 1–9, 2018
2018
-
[8]
Bayesian neural networks for uncertainty estimation of imaging biomarkers,
J. Senapati, A. G. Roy, S. P ¨olsterl, D. Gutmann, S. Gatidis, C. Schlett, A. Peters, F. Bamberg, and C. Wachinger, “Bayesian neural networks for uncertainty estimation of imaging biomarkers,” in Machine Learning in Medical Imaging: 11th International Workshop, MLMI 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Proceedings 11. S...
work page 2020
Show all 20 references
-
[9]
An exploration of un- certainty information for segmentation quality assessment,
K. Hoebel, V . Andrearczyk, A. Beers, J. Patel, K. Chang, A. De- peursinge, H. M ¨uller, and J. Kalpathy-Cramer, “An exploration of un- certainty information for segmentation quality assessment,” in Medical Imaging 2020: Image Processing, vol. 11313. SPIE, 2020, pp. 381–390
2020
-
[10]
Accuracy, uncertainty, and adaptability of automatic myocardial asl segmentation using deep cnn,
H. P. Do, Y . Guo, A. J. Yoon, and K. S. Nayak, “Accuracy, uncertainty, and adaptability of automatic myocardial asl segmentation using deep cnn,” Magnetic resonance in medicine , vol. 83, no. 5, pp. 1863–1874, 2020
2020
-
[11]
Quantifying uncer- tainty of deep neural networks in skin lesion classification,
P. Van Molle, T. Verbelen, C. De Boom, B. Vankeirsbilck, J. De Vylder, B. Diricx, T. Kimpe, P. Simoens, and B. Dhoedt, “Quantifying uncer- tainty of deep neural networks in skin lesion classification,” in Uncer- tainty for Safe Utilization of Machine Learning in Medical Imagin...
2019
-
[12]
Un- certainty estimation in deep neural networks for dermoscopic image classification,
M. Combalia, F. Hueto, S. Puig, J. Malvehy, and V . Vilaplana, “Un- certainty estimation in deep neural networks for dermoscopic image classification,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , 2020, pp. 744–745
2020
-
[13]
Multi- class skin cancer classification architecture based on deep convolutional neural network,
M. Shapna Akter, H. Shahriar, S. Sneha, and A. Cuzzocrea, “Multi- class skin cancer classification architecture based on deep convolutional neural network,” arXiv e-prints, pp. arXiv–2303, 2023
2023
-
[14]
Melanoma segmentation using deep learning with test-time augmenta- tions and conditional random fields,
H. Ashraf, A. Waris, M. F. Ghafoor, S. O. Gilani, and I. K. Niazi, “Melanoma segmentation using deep learning with test-time augmenta- tions and conditional random fields,” Scientific Reports, vol. 12, no. 1, p. 3948, 2022
2022
-
[15]
Skin lesion seg- mentation using deep learning for images acquired from smartphones,
G. G. De Angelo, A. G. Pacheco, and R. A. Krohling, “Skin lesion seg- mentation using deep learning for images acquired from smartphones,” in 2019 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2019, pp. 1–8
2019
-
[16]
Melanoma segmentation: A framework of im- proved densenet77 and unet convolutional neural network,
M. Nawaz, T. Nazir, M. Masood, F. Ali, M. A. Khan, U. Tariq, N. Sahar, and R. Dama ˇseviˇcius, “Melanoma segmentation: A framework of im- proved densenet77 and unet convolutional neural network,”International Journal of Imaging Systems and Technology , vol. 32, no. 6, pp. 2137...
2022
-
[17]
ISIC Archive homepage,
“ISIC Archive homepage,” 2019, https://www.isic-archive.com/#! /topWithHeader/wideContentTop/main
2019
-
[18]
Bcn20000: Dermoscopic lesions in the wild,
M. Combalia, N. C. Codella, V . Rotemberg, B. Helba, V . Vila- plana, O. Reiter, C. Carrera, A. Barreiro, A. C. Halpern, S. Puig et al. , “Bcn20000: Dermoscopic lesions in the wild,” arXiv preprint arXiv:1908.02288, 2019
1908 arXiv
-
[19]
A deep-learning toolkit for visualization and interpretation of segmented medical images,
S. Ghosal and P. Shah, “A deep-learning toolkit for visualization and interpretation of segmented medical images,” Cell Reports Methods , vol. 1, no. 7, 2021
2021
-
[20]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman., “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
2014 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.