{"id":"6808c0da-441d-48af-891d-e249a0274c98","arxiv_id":"2508.17768","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A modified U-Net with MC dropout and deep ensembles achieves state-of-the-art breast ultrasound segmentation and improved uncertainty estimates after deduplication of the BUSI training data.","lead":"This paper evaluates a modified U-Net with uncertainty estimates for breast ultrasound tumor segmentation. It also corrects duplicated data in a standard benchmark and shows that uncertainty rises when the model meets unseen clinical domains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undocumented BUSI deduplication is the linchpin: reported SOTA accuracy and calibration cannot be assessed without its definition and leak-free split.","rationale":"The reader's weakest_assumption identifies the BUSI deduplication as the critical unknown. My stress-test agrees and sharpens why it matters: the deduplication procedure directly affects the validity of both the accuracy and the uncertainty calibration claims, because any residual near-duplication between training and test sets would inflate performance and make the model appear artificially well-calibrated on familiar patterns. This is not a disagreement with the paper's internal logic; it is a missing external verification step. Since the full text and code were not available for this pass, the appropriate verdict remains UNVERDICTED, exactly as the reader concluded. I see no reason to move to ACCEPT or REJECT based on the abstract alone. The concrete test is a single reproduction that would settle whether the deduplication-induced bias is real; if the results are stable under a strict alternative deduplication, the concern is successfully addressed.","tokens_in":747,"tokens_out":1924,"duration_ms":25139,"concrete_test":"Clone the provided repository (https://github.com/toufiqmusah/nn-uncertainty.git) and inspect the deduplication script. Then reproduce the BUSI evaluation under two alternative deduplication regimes: (a) the authors' original method, and (b) a strict patient-level split using all available patient identifiers or perceptual hashing to remove near-duplicates before any train/test division. If the Dice score difference between (a) and (b) exceeds ~2 points, or if the expected calibration error changes substantially, the reported state-of-the-art accuracy and calibration are not robust to the deduplication choice, and the central claim would need to be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that the model achieves state-of-the-art segmentation accuracy on Breast-Lesion-USG with in-distribution validation and provides calibrated uncertainty estimates that signal low-confidence regions. Both sub-claims depend on the evaluation protocol being unbiased. The only methodological correction mentioned is 'identify and correct for data duplication in the BUSI dataset,' but no details are given: how duplicates are defined (exact hashing vs. visual similarity), whether deduplication was performed before or after splitting, and whether patient-level leakage was eliminated. If near-duplicate images remain across train/test splits, the model's performance is inflated because it has effectively seen test-like examples during training, and calibration is biased toward overconfidence on familiar cases. Conversely, if deduplication is too aggressive or applied after splitting, the test set may become non-representative, and the reported 'state-of-the-art' comparison is unfair or meaningless. The abstract itself flags deduplication as a deliberate correction, so the reliability of every reported number rests on this undocumented procedure. Without the full text or code, this is a missing essential support rather than an internal contradiction, but it is load-bearing: if the deduplication is flawed, both the accuracy and the uncertainty claims collapse.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This abstract-only submission proposes a modified Residual Encoder U-Net for breast ultrasound (BUS) segmentation with epistemic uncertainty quantification via Monte Carlo dropout, deep ensembles, and their combination. The authors state that they identify and correct data duplication in the BUSI dataset, evaluate on in-distribution and out-of-distribution datasets, achieve state-of-the-art segmentation accuracy on the Breast-Lesion-USG dataset with in-distribution validation, and provide calibrated uncertainty estimates that flag low-confidence regions. A public code repository is referenced. No numerical results, methodological details, or baselines are reported in the abstract.","tokens_in":1025,"tokens_out":1676,"duration_ms":20569,"significance":"If the claims are substantiated by rigorous experiments, this work would be practically valuable: trustworthy uncertainty estimates for breast tumor segmentation could support clinical decision-making, and the explicit handling of dataset duplication addresses a known flaw in BUSI. The public code availability is a concrete strength. However, because the central claims are quantitative empirical assertions, the absence of metrics, baseline comparisons, calibration measures, and a description of the deduplication procedure means the significance cannot currently be assessed from the manuscript as presented.","major_comments":[{"comment":"The central claim, 'state-of-the-art segmentation accuracy on the Breast-Lesion-USG dataset' and 'calibrated uncertainty estimates that effectively signal regions of low model confidence,' is stated without any numerical support. No Dice/Jaccard scores, confidence intervals, calibration errors (ECE, Brier score), or comparator baselines are provided. For an empirical benchmarking paper, these numbers are the core evidence; the abstract alone cannot support the claim.","section":"Abstract (claims of SOTA and calibration)"},{"comment":"The statement 'identify and correct for data duplication in the BUSI dataset' is load-bearing for the reliability of all reported results, yet no details are given. The reader cannot tell whether duplicates are defined by exact hashing or visual similarity, whether deduplication was performed before or after train/test splitting, whether patient-level leakage was eliminated, or how many images were removed. If near-duplicates remain across splits, accuracy and calibration are inflated; if the procedure is aggressive or applied after splitting, the test set may be unrepresentative. This is an essential methodological detail that must be specified.","section":"Abstract (BUSI deduplication)"},{"comment":"The abstract claims that 'performance declines and increased uncertainty observed in out-of-distribution evaluation,' but the OOD datasets are not named, no quantitative decline is reported, and no uncertainty-calibration measure is given for the OOD setting. This makes the generalization claim and the domain-shift conclusion impossible to evaluate.","section":"Abstract (OOD evaluation)"}],"minor_comments":[{"comment":"The relationship between the 'BUSI dataset' and the 'Breast-Lesion-USG dataset' is unclear. Are these the same benchmark, or two different datasets? Please clarify the nomenclature and the relationship between the deduplicated subset and the evaluation set.","section":"Abstract (dataset nomenclature)"},{"comment":"The phrase 'their combination' is vague. Does this mean MC dropout applied to ensemble members, voting of ensemble and MC-dropout uncertainties, or something else? A precise definition would help.","section":"Abstract (method naming)"},{"comment":"The code link is a positive step, but the abstract should state the license and, for reproducibility, the exact commit/version used for the reported experiments.","section":"Abstract (code availability)"}],"recommendation":"major_revision","confidential_remarks":"This review is based solely on the abstract because the full text was not available. The abstract makes strong quantitative claims without reporting any metrics or methodological details, most critically the BUSI deduplication procedure. These are missing essential supports, not irreparable errors, so I recommend major revision: the authors should provide the full experimental details, numbers, and baselines. I would not reject on the current evidence, but the paper cannot be accepted in this form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The abstract describes a reasonable empirical study: comparing MC dropout, deep ensembles, and their combination for breast ultrasound segmentation, with the added step of deduplicating the BUSI dataset. That deduplication is potentially the most useful contribution, because BUSI is known to have duplicate images and that can inflate results. The in/out-of-distribution evaluation is also the right thing to do for clinical trust. So there is a solid core here.\n\nThe soft spots are exactly where the reader and stress-test point. The abstract makes strong claims about state-of-the-art accuracy and calibrated uncertainty but gives no numbers, no description of how duplicates were identified (hash? visual similarity? patient-level?), no split details, and no definition of the OOD sets. If the deduplication was done after splitting, or left near-duplicates across train and test, the results could be substantially inflated. If it was done too aggressively, the test set may not be representative. The stress-test call is fair: this is load-bearing, not a side detail.\n\nThat said, this is an abstract-only review. The paper may well contain all the necessary details; the code link suggests the authors plan to share the implementation. I would not reject the paper or call it flawed based on the abstract alone, but I also would not rely on its claims until I see the full methods. The lack of metrics in the abstract is not a problem by itself, but it makes the claims impossible to check here.\n\nMy recommendation: send this to peer review. A careful referee should ask for the deduplication script, the exact splitting procedure, and calibration results. If those hold up, the paper is a useful, if incremental, contribution. I would not cite it myself until I have seen the full text, but it is worth a reading group discussion.\n\nFor the record, the reader's low confidence is appropriate, and the stress-test pinned the right weakness.","headline":"A useful but underdocumented benchmark: the BUSI deduplication is the one thing I'd want to see before trusting the SOTA and calibration claims.","tokens_in":1491,"tokens_out":1973,"would_cite":false,"duration_ms":25262,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An uncertainty-aware U-Net achieves state-of-the-art segmentation of breast ultrasound lesions and provides calibrated confidence signals.","keywords":["breast ultrasound segmentation","epistemic uncertainty","Monte Carlo dropout","deep ensembles","domain shift","BUSI dataset","medical image segmentation","uncertainty calibration"],"falsifier":"Run an independent audit of the deduplication pipeline—publish the exact similarity threshold and the method (e.g., perceptual hashing versus visual review)—then re-evaluate the same models on the full, unfiltered BUSI dataset and on a different breast ultrasound dataset; if the state-of-the-art accuracy or calibration degrades markedly, the central claim fails.","tokens_in":709,"feed_emoji":"🩺","tokens_out":4458,"duration_ms":41620,"temperature":0.7,"pith_summary":"The paper argues that adding explicit epistemic uncertainty estimation to a modified Residual Encoder U-Net yields top-tier segmentation accuracy on the Breast-Lesion-USG benchmark alongside calibrated uncertainty maps that flag low-confidence regions. The authors clean the commonly used BUSI dataset by removing duplicated images, claiming this gives more honest generalization estimates. They benchmark in-distribution and out-of-distribution, finding that performance drops and uncertainty rises under domain shift. The practical payoff is a model that can tell a clinician when it is out of its depth, not just output a mask.","feed_headline":"Segmentation model hits state-of-the-art and flags its own errors","feed_subtitle":"MC dropout and deep ensembles make the model signal exactly where it is unsure","key_machinery":"The central mechanism is the Residual Encoder U-Net architecture combined with two Bayesian-inspired uncertainty techniques: Monte Carlo dropout, which treats dropout as sampling from the posterior to produce a distribution of masks, and deep ensembles, which average predictions from separately trained models. Both are used to estimate epistemic uncertainty, the part of uncertainty that could be reduced with better data or models. The uncertainty maps are claimed to be calibrated, meaning the model's confidence scores correspond to actual correctness probabilities.","core_discovery":"On the paper's own terms, the central discovery is that a modified Residual Encoder U-Net, when paired with Monte Carlo dropout, deep ensembles, or both, does not sacrifice accuracy for uncertainty awareness: it reaches state-of-the-art segmentation performance on the Breast-Lesion-USG dataset while producing calibrated epistemic uncertainty estimates that concentrate on regions of low model confidence. The authors also contribute a deduplicated version of the BUSI dataset, arguing that prior benchmarks may have been inflated by training and testing on near-duplicate images.","pith_inferences":["The paper's deduplication step is load-bearing: without a reproducible, well-defined duplicate criterion, the reported numbers could be an artifact of how the subset was chosen rather than a true model property.","Calibration is demonstrated on limited data; extending to multi-center, multi-vendor ultrasound would reveal whether the uncertainty signals remain trustworthy in real clinical workflows.","The observed increase in out-of-distribution uncertainty suggests a possible automatic out-of-distribution detector: thresholding on high predicted uncertainty could screen images for human review or further analysis.","Because the experiments compare MC dropout, ensembles, and their combination, the paper implicitly addresses whether the extra compute of ensembles is worth the calibration gain—a cost-benefit analysis the authors may not fully spell out."],"forward_implications":["If the central claim holds, breast ultrasound segmentation can be deployed with per-pixel confidence maps, allowing clinicians to focus review on the most uncertain boundaries.","The deduplicated BUSI subset offers a more realistic benchmark; previous state-of-the-art numbers on the full dataset may have been optimistic due to duplicate leakage.","The observed accuracy drop and uncertainty increase on out-of-distribution data concretely quantify the cost of domain shift, motivating closer attention to domain adaptation in breast ultrasound.","Combining Monte Carlo dropout with deep ensembles may become a practical recipe for uncertainty-aware medical segmentation, since it maintains accuracy while adding calibrated confidence."],"supporting_citations":[],"fun_headline_variants":["Breast tumor segmentation that knows its limits","Uncertainty-aware AI pinpoints where it's unsure","U-Net with MC dropout hits SOTA and flags low confidence","Deduplicated data reveal true segmentation performance","Segmentation model calibrated to signal its own errors"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported accuracy and calibration numbers rest on the correctness of the BUSI dataset deduplication: if duplicated images were removed incorrectly or near-duplicates remained, the state-of-the-art claim and uncertainty calibration could be misleading.","fun_headline_variants_meta":{"raw":{"variants":["Breast tumor segmentation that knows its limits","Uncertainty-aware AI pinpoints where it's unsure","U-Net with MC dropout hits SOTA and flags low confidence","Deduplicated data reveal true segmentation performance","Segmentation model calibrated to signal its own errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2378,"prompt_tokens":695,"completion_tokens":1683,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1608}},"tokens_in":439,"tokens_out":1683,"duration_ms":15424,"temperature":1.0,"reasoning_tokens":1608,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:43:02.229200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an independent audit of the deduplication pipeline—publish the exact similarity threshold and the method (e.g., perceptual hashing versus visual review)—then re-evaluate the same models on the full, unfiltered BUSI dataset and on a different breast ultrasound dataset; if the state-of-the-art accuracy or calibration degrades markedly, the central claim fails.","supporting_citations":[],"review_version":1}