REVIEW 3 major objections 3 minor
Towards Trustworthy Breast Tumor Segmentation in Ultrasound using Monte Carlo Dropout and Deep Ensembles for Epistemic Uncertainty Estimation
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read An uncertainty-aware U-Net achieves state-of-the-art segmentation of breast ultrasound lesions and provides calibrated confidence signals.
desk verdict A useful but underdocumented benchmark: the BUSI deduplication is the one thing I'd want to see before trusting the SOTA and calibration claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the Residual Encoder U-Net architecture combined with two Bayesian-inspired uncertainty techniques: Monte Carlo dropout, which treats dropout as sampling from the posterior to produce a distribution of masks, and deep ensembles, which average predictions from separately trained models. Both are used to estimate epistemic uncertainty, the part of uncertainty that could be reduced with better data or models. The uncertainty maps are claimed to be calibrated, meaning the model's confidence scores correspond to actual correctness probabilities.
What would settle it
Run an independent audit of the deduplication pipeline—publish the exact similarity threshold and the method (e.g., perceptual hashing versus visual review)—then re-evaluate the same models on the full, unfiltered BUSI dataset and on a different breast ultrasound dataset; if the state-of-the-art accuracy or calibration degrades markedly, the central claim fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a modified Residual Encoder U-Net, when paired with Monte Carlo dropout, deep ensembles, or both, does not sacrifice accuracy for uncertainty awareness: it reaches state-of-the-art segmentation performance on the Breast-Lesion-USG dataset while producing calibrated epistemic uncertainty estimates that concentrate on regions of low model confidence. The authors also contribute a deduplicated version of the BUSI dataset, arguing that prior benchmarks may have been inflated by training and testing on near-duplicate images.
Load-bearing premise
The reported accuracy and calibration numbers rest on the correctness of the BUSI dataset deduplication: if duplicated images were removed incorrectly or near-duplicates remained, the state-of-the-art claim and uncertainty calibration could be misleading.
Editorial extensions
If this is right
- If the central claim holds, breast ultrasound segmentation can be deployed with per-pixel confidence maps, allowing clinicians to focus review on the most uncertain boundaries.
- The deduplicated BUSI subset offers a more realistic benchmark; previous state-of-the-art numbers on the full dataset may have been optimistic due to duplicate leakage.
- The observed accuracy drop and uncertainty increase on out-of-distribution data concretely quantify the cost of domain shift, motivating closer attention to domain adaptation in breast ultrasound.
- Combining Monte Carlo dropout with deep ensembles may become a practical recipe for uncertainty-aware medical segmentation, since it maintains accuracy while adding calibrated confidence.
Reading between the lines
- The paper's deduplication step is load-bearing: without a reproducible, well-defined duplicate criterion, the reported numbers could be an artifact of how the subset was chosen rather than a true model property.
- Calibration is demonstrated on limited data; extending to multi-center, multi-vendor ultrasound would reveal whether the uncertainty signals remain trustworthy in real clinical workflows.
- The observed increase in out-of-distribution uncertainty suggests a possible automatic out-of-distribution detector: thresholding on high predicted uncertainty could screen images for human review or further analysis.
- Because the experiments compare MC dropout, ensembles, and their combination, the paper implicitly addresses whether the extra compute of ensembles is worth the calibration gain—a cost-benefit analysis the authors may not fully spell out.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This abstract-only submission proposes a modified Residual Encoder U-Net for breast ultrasound (BUS) segmentation with epistemic uncertainty quantification via Monte Carlo dropout, deep ensembles, and their combination. The authors state that they identify and correct data duplication in the BUSI dataset, evaluate on in-distribution and out-of-distribution datasets, achieve state-of-the-art segmentation accuracy on the Breast-Lesion-USG dataset with in-distribution validation, and provide calibrated uncertainty estimates that flag low-confidence regions. A public code repository is referenced. No numerical results, methodological details, or baselines are reported in the abstract.
Significance. If the claims are substantiated by rigorous experiments, this work would be practically valuable: trustworthy uncertainty estimates for breast tumor segmentation could support clinical decision-making, and the explicit handling of dataset duplication addresses a known flaw in BUSI. The public code availability is a concrete strength. However, because the central claims are quantitative empirical assertions, the absence of metrics, baseline comparisons, calibration measures, and a description of the deduplication procedure means the significance cannot currently be assessed from the manuscript as presented.
major comments (3)
- [Abstract (claims of SOTA and calibration)] The central claim, 'state-of-the-art segmentation accuracy on the Breast-Lesion-USG dataset' and 'calibrated uncertainty estimates that effectively signal regions of low model confidence,' is stated without any numerical support. No Dice/Jaccard scores, confidence intervals, calibration errors (ECE, Brier score), or comparator baselines are provided. For an empirical benchmarking paper, these numbers are the core evidence; the abstract alone cannot support the claim.
- [Abstract (BUSI deduplication)] The statement 'identify and correct for data duplication in the BUSI dataset' is load-bearing for the reliability of all reported results, yet no details are given. The reader cannot tell whether duplicates are defined by exact hashing or visual similarity, whether deduplication was performed before or after train/test splitting, whether patient-level leakage was eliminated, or how many images were removed. If near-duplicates remain across splits, accuracy and calibration are inflated; if the procedure is aggressive or applied after splitting, the test set may be unrepresentative. This is an essential methodological detail that must be specified.
- [Abstract (OOD evaluation)] The abstract claims that 'performance declines and increased uncertainty observed in out-of-distribution evaluation,' but the OOD datasets are not named, no quantitative decline is reported, and no uncertainty-calibration measure is given for the OOD setting. This makes the generalization claim and the domain-shift conclusion impossible to evaluate.
minor comments (3)
- [Abstract (dataset nomenclature)] The relationship between the 'BUSI dataset' and the 'Breast-Lesion-USG dataset' is unclear. Are these the same benchmark, or two different datasets? Please clarify the nomenclature and the relationship between the deduplicated subset and the evaluation set.
- [Abstract (method naming)] The phrase 'their combination' is vague. Does this mean MC dropout applied to ensemble members, voting of ensemble and MC-dropout uncertainties, or something else? A precise definition would help.
- [Abstract (code availability)] The code link is a positive step, but the abstract should state the license and, for reproducibility, the exact commit/version used for the reported experiments.
Circularity Check
No circularity identified; abstract reports an empirical benchmark with no self-referential derivation.
full rationale
This is an abstract-only review of an empirical deep-learning paper. The claims are (1) a modified Residual Encoder U-Net achieves state-of-the-art segmentation accuracy on Breast-Lesion-USG after BUSI deduplication, and (2) Monte Carlo dropout, deep ensembles, and their combination provide calibrated epistemic uncertainty estimates. There is no derivation chain in the abstract: no equation is fitted to the quantity it then predicts, no parameter is renamed as a prediction, and no uniqueness theorem or load-bearing self-citation is invoked. The deduplication of BUSI is a data-cleaning step that affects the validity and comparability of the reported numbers, but validity concerns are not circularity: the paper does not define the evaluation metric in terms of its own output or fit the test set. Because the full text is unavailable, no specific circular step can be quoted or exhibited, and the rules require concrete evidence rather than speculation. The appropriate finding is therefore no significant circularity (score 0).
Assumptions & free parameters
assumptions (3)
- domain assumption The deduplication of the BUSI dataset is accurate and does not introduce selection bias.
- domain assumption Monte Carlo dropout and deep ensembles provide valid estimates of epistemic uncertainty.
- domain assumption The out-of-distribution datasets are representative of genuinely unseen clinical domains.
Cite this review
Pith. "Pith review of Towards Trustworthy Breast Tumor Segmentation in Ultrasound using Monte Carlo Dropout and Deep Ensembles for Epistemic Uncertainty Estimation." pith.science (2026). https://pith.science/paper/B67DP3UH
@misc{pith2026250817768,
author = {Pith},
title = {Pith review of: Towards Trustworthy Breast Tumor Segmentation in Ultrasound using Monte Carlo Dropout and Deep Ensembles for Epistemic Uncertainty Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/B67DP3UH}},
note = {Machine review of arXiv:2508.17768}
}
read the original abstract
Automated segmentation of BUS images is important for precise lesion delineation and tumor characterization, but is challenged by inherent artifacts and dataset inconsistencies. In this work, we evaluate the use of a modified Residual Encoder U-Net for breast ultrasound segmentation, with a focus on uncertainty quantification. We identify and correct for data duplication in the BUSI dataset, and use a deduplicated subset for more reliable estimates of generalization performance. Epistemic uncertainty is quantified using Monte Carlo dropout, deep ensembles, and their combination. Models are benchmarked on both in-distribution and out-of-distribution datasets to demonstrate how they generalize to unseen cross-domain data. Our approach achieves state-of-the-art segmentation accuracy on the Breast-Lesion-USG dataset with in-distribution validation, and provides calibrated uncertainty estimates that effectively signal regions of low model confidence. Performance declines and increased uncertainty observed in out-of-distribution evaluation highlight the persistent challenge of domain shift in medical imaging, and the importance of integrated uncertainty modeling for trustworthy clinical deployment. \footnote{Code available at: https://github.com/toufiqmusah/nn-uncertainty.git}
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.