{"id":"5ea9ab06-05fb-4520-87d8-7573b2ebdcc3","arxiv_id":"2506.16589","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"New spatially aware metrics (BUC, BA-ECE, SPACE) evaluate segmentation uncertainty using boundary distance and local smoothing, and outperform voxel-wise metrics at distinguishing clean from noisy uncertainty maps on prostate MRI.","lead":"This paper proposes three new ways to score uncertainty maps in medical image segmentation, using distance to anatomical boundaries and spatial smoothing instead of treating each voxel independently. If they work, these metrics would help doctors and automated systems tell apart genuinely useful uncertainty signals from scattered noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation labels 'high quality' with the authors' own Rate-In maps, making reported discrimination partly self-consistent; independent uncertainty methods are needed to confirm generalizability.","rationale":"The paper proposes three spatially aware metrics and reports that they better discriminate 'high quality' from 'noisy' uncertainty maps. The strongest numerical result is Table 1, where SPACE reaches 95.83% accuracy and BA-ECE reaches Cohen's d = 1.83. However, the binary labels in that table are not externally validated: 'high quality' is defined by the authors' own Rate-In method, and 'noisy' is a fixed high-dropout MC map. Because BUC and BA-ECE explicitly reward boundary concentration and SPACE rewards local error alignment, and because Rate-In was intentionally designed to produce boundary-focused uncertainty, the high accuracy is to some extent baked into the experimental design. The reader's verdict correctly identifies this as the weakest assumption. I am not claiming the metrics are useless; they may well be valuable, but the current evidence does not establish that they track clinically meaningful uncertainty beyond agreeing with the authors' prior. The concrete test I propose would break the circularity by introducing an independent source of 'high quality' uncertainty maps. If the metrics still separate independent methods, the CONDITIONAL verdict can be upgraded; if not, the central claim would need substantial revision. Secondary issues (no code/data, unspecified hyperparameters, single small dataset) are real but less decisive than the reference-label problem. I therefore agree with the reader's CONDITIONAL verdict and recommend keeping it until an independent validation is provided.","tokens_in":5290,"tokens_out":2848,"duration_ms":36705,"concrete_test":"Run the Section 4.3 protocol using high-quality maps from an independent method not developed by the authors (e.g., deep ensembles or test-time augmentation) and low-quality maps from fixed high-rate MC dropout on the same 36 cases. If SPACE/BA-ECE/BUC accuracies and Cohen's d remain high, self-selection is not the driver; if they fall to near chance, the reported discrimination is specific to Rate-In maps.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2 defines the 'high quality' uncertainty map as the authors' Rate-In method and the 'low quality' map as fixed high-rate MC dropout; Section 4.3 then defines Accuracy as the fraction of cases where a metric ranks the Rate-In map above the noisy map. This makes the evaluation self-referential: Rate-In was designed to concentrate uncertainty at boundaries, so metrics that explicitly reward boundary concentration (BUC, BA-ECE) and local error alignment (SPACE) are expected to favor it by construction. Table 1 therefore demonstrates that the new metrics agree with the authors' prior about what good uncertainty should look like, not that they track clinically meaningful quality. No independent anchor (expert ratings, downstream task performance, or a second uncertainty method outside the author group) is provided, so the reported accuracies and effect sizes cannot separate genuine discriminative power from matching the construction of the reference label. Unspecified hyperparameters (sigma in SPACE, K bands in BA-ECE, boundary distance in BUC) further complicate interpretation, but the reference-label issue is the primary load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces three spatially-aware metrics for evaluating segmentation uncertainty maps: Boundary Uncertainty Concentration (BUC), Boundary-Aware Expected Calibration Error (BA-ECE), and Spatially-Aware Calibration Error (SPACE). The metrics incorporate boundary proximity and spatial smoothing, and are validated on a prostate zonal segmentation dataset (36 cases) by comparing how well each metric discriminates 'high quality' uncertainty maps generated by the authors' Rate-In method from 'low quality' maps generated with high-rate dropout. The authors report that their metrics achieve higher accuracy and effect sizes than traditional voxel-wise metrics (Table 1), and conclude that spatial information improves uncertainty evaluation.","tokens_in":5503,"tokens_out":6851,"duration_ms":68584,"significance":"The proposed metrics are conceptually reasonable and could be valuable for uncertainty evaluation if shown to align with clinically meaningful quality. The paper provides clear mathematical definitions and a comparison with existing measures. However, the empirical validation is weakened by the self-referential definition of 'high quality' (the authors' own Rate-In maps), the small sample size, and lack of sensitivity analysis, so the central claim that these metrics better distinguish meaningful from spurious uncertainty is not yet convincingly established.","major_comments":[{"comment":"The evaluation is circular. The 'high quality' uncertainty map is defined as the output of the authors' own Rate-In method, and 'Accuracy' is the percentage of cases where a metric ranks this map above the noisy high-rate dropout map. Because BUC, BA-ECE, and SPACE are explicitly constructed to reward boundary concentration and local error alignment, and Rate-In is designed to concentrate uncertainty at boundaries, the superior scores of the proposed metrics on this test are expected by construction. The reported improvements therefore demonstrate agreement with the authors' prior about what good uncertainty should look like, not general discriminative power. Provide an independent anchor, such as expert quality ratings, downstream task performance (e.g., segmentation error detection after human review), or a second, non-self-cited uncertainty method (e.g., deep ensembles), and re-run the analysis.","section":"Sec. 4.2-4.3"},{"comment":"The hyperparameters of the proposed metrics are unspecified and no sensitivity analysis is given. For BUC, the boundary region R is defined via 'a chosen distance' with 'e.g., the 95th percentile Hausdorff distance' as an example, but the actual choice is not reported. For BA-ECE, the number of bands K and the distance-weighting scheme are not given. For SPACE, the Gaussian kernel width sigma is not specified. The comparison in Table 1 is therefore conditional on unstated parameter values; please provide exact settings and test robustness of the ranking to these parameters.","section":"Sec. 3"},{"comment":"The statistical significance claims are incomplete and partly inconsistent. Only SPACE is said to significantly outperform all traditional metrics; BUC and BA-ECE are reported as significantly outperforming only metrics with ≤70% accuracy, an arbitrary threshold. Moreover, Section 5 states that 'SPACE achieved the highest accuracy and effect size,' but Table 1 shows BA-ECE has the highest Cohen's d (1.83 vs. 1.34). Please report full pairwise comparisons, correction for multiple testing, and confidence intervals, and fix the inconsistency.","section":"Sec. 4.4, Table 1"},{"comment":"The validation is limited to a single dataset (prostate zonal segmentation, 36 cases), a single pre-trained nnU-Net model, and one pair of uncertainty maps. The claimed general superiority of spatially-aware metrics over voxel-wise metrics is not supported beyond this narrow setting. At a minimum, the conclusions should be restricted to the evaluated setup, or additional data/models should be tested.","section":"Sec. 4.1-4.2"}],"minor_comments":[{"comment":"The notation for the mean uncertainty outside the boundary region uses the same symbol µR as the inside mean; please use a distinct symbol such as µ_{R^c} to avoid ambiguity.","section":"Sec. 3.1"},{"comment":"For metrics where lower values are better (e.g., SPACE, BA-ECE), the 'Mean Diff.' column should explicitly state the direction; the current positive values require the reader to infer the sign convention.","section":"Sec. 4.4, Table 1"},{"comment":"The related work could mention existing spatially-aware uncertainty evaluation approaches (e.g., distance-dependent calibration or edge-based methods) for context; currently the paper positions itself as the first to do this.","section":"Sec. 2"},{"comment":"The paper does not mention code availability or provide exact algorithmic steps (e.g., distance transform implementation, number of bins); adding a supplement or code would aid reproducibility.","section":"General"},{"comment":"The definition of 'Accuracy' would benefit from a formula, as the per-metric direction (whether higher or lower value indicates 'favoring' the high-quality map) is implicit.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical evaluation relies heavily on the authors' own Rate-In method as the reference, which may limit the perceived impartiality of the validation. The paper is presented at a workshop and might be better positioned as a methods paper with a provisional evaluation, rather than making strong claims of general superiority."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short paper by Zeevi et al. proposes three spatially aware metrics for segmentation uncertainty: BUC (boundary concentration), BA-ECE (distance-banded calibration), and SPACE (smoothed local difference). The definitions are clear, easy to implement, and each addresses a real limitation of voxel-wise metrics. That is the paper's genuine value, and the central idea is sound.\n\nThe results on 36 prostate cases show that all three metrics, especially SPACE, separate the authors' 'clean' from 'noisy' uncertainty maps better than several traditional baselines. The comparison includes a reasonable set of baselines and uses McNemar tests with Holm correction, which is more careful than most workshop papers.\n\nThe load-bearing weakness is the ground truth. In Section 4.2, the 'high quality' map is defined as the output of the authors' own Rate-In method, and the 'low quality' map is high dropout rate MC dropout. Accuracy is then the fraction of cases where a metric ranks Rate-In above the noisy map. So the evaluation measures agreement with the authors' prior about what good uncertainty should look like, not agreement with any independent clinical or external standard. The stress-test note gets this right. It is not a fatal flaw, because Rate-In is a published method and the comparison is still meaningful as a sanity check, but it leaves open the possibility that the new metrics are simply more sensitive to boundary concentration regardless of clinical value.\n\nThe other issues are minor in comparison. The hyperparameters (sigma, K, boundary distance R) are unspecified, no sensitivity analysis is given, and there is no code or data release. The typo in the BUC complement region is cosmetic. Also note that only SPACE significantly outperforms all traditional metrics; BUC and BA-ECE only clear the low-accuracy baselines, so the broad claim in the abstract is a bit stronger than the pairwise tests support.\n\nI would send this to peer review — it is a legitimate, clearly written contribution that would benefit from external validation. But as it stands, the evidence is not enough to support the 'thorough validation' claim in the abstract. The authors should provide an independent reference for map quality (e.g., expert raters or downstream task performance), sensitivity analysis, and ideally more than one dataset.\n\nFor your reading group, it is a useful sanity check on what spatial metrics can do, and a good starting point for discussion about evaluation design in uncertainty quantification.\n\n— Your name","headline":"Spatially aware metrics are a useful addition, but the self-referential validation against the authors' own Rate-In maps means the reported discrimination is partly agreement with a prior; still deserves refereeing.","tokens_in":6009,"tokens_out":3206,"would_cite":false,"duration_ms":36558,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that spatially aware metrics—boundary concentration, boundary-binned calibration, and smoothed error alignment—discriminate clinically meaningful segmentation uncertainty from scattered noise far better than voxel-wise…","keywords":["uncertainty quantification","segmentation uncertainty","spatially aware metrics","calibration error","boundary uncertainty","medical image segmentation","prostate MRI","Monte Carlo dropout"],"falsifier":"Take a new dataset where uncertainty map quality is labeled by expert radiologists or by measured impact on a downstream task such as surgical planning error, and compare SPACE, BUC, and BA-ECE rankings against voxel-wise metrics; if the spatially aware metrics do not track those external quality labels more closely, the discrimination claim is an artifact of the Rate-In-derived reference.","tokens_in":5092,"feed_emoji":"🩻","tokens_out":6880,"duration_ms":69966,"temperature":0.7,"pith_summary":"This paper tries to establish that evaluating segmentation uncertainty voxel-by-voxel misses the spatial pattern of where uncertainty sits, and that three new metrics which include boundary and neighborhood structure can tell clinically meaningful uncertainty maps from diffuse noise. On prostate zonal segmentation from the Medical Segmentation Decathlon, the proposed metrics—BUC, BA-ECE, and SPACE—separate high-quality from low-quality uncertainty maps more reliably than standard voxel-wise metrics, with SPACE reaching 95.83% accuracy and BA-ECE the largest effect size. If correct, the result gives medical imaging practitioners a way to judge whether a model's uncertainty is concentrated at anatomical boundaries where it is useful, rather than scattered randomly.","feed_headline":"Spatially aware metrics spot good uncertainty maps 96% of the time","feed_subtitle":"Boundary-focused uncertainty beats scattered noise under BUC, BA-ECE, and SPACE on prostate MRI segmentation.","key_machinery":"The central machinery is three definitions that replace independent-voxel statistics with spatial structure. BUC compares the mean uncertainty inside a band near the predicted boundary with the sum of inside and outside means, so values near $1$ mean uncertainty is concentrated at the boundary. BA-ECE bins voxels by distance from the ground-truth boundary and returns the weighted mean absolute difference between average uncertainty and average error per band, giving more weight to bands closest to the boundary. SPACE convolves the uncertainty map and the binary error map with a Gaussian kernel $G_\\sigma$ and takes the mean absolute difference of the smoothed results, so low values mean uncertainty tracks errors in local neighborhoods. Each metric turns a voxel-wise score into a statement about where uncertainty sits relative to anatomy.","core_discovery":"The paper's central claim is that existing voxel-wise uncertainty evaluation metrics are spatially agnostic: they score scattered noise and boundary-aligned uncertainty the same way whenever the voxel-level statistics match. To fix this, the authors define three spatially aware metrics. BUC quantifies how much uncertainty is concentrated near the predicted boundary. BA-ECE bins voxels by distance to the ground-truth boundary and computes a weighted calibration error in each band. SPACE smooths both uncertainty and error maps and measures their local mean absolute difference. The validation on 36 prostate MRI cases shows these metrics favor the method's designated high-quality boundary-focused uncertainty map over a noisy diffuse map more consistently than ECE, AUC, PAvPU, AU-ARC, and ratio-based baselines.","pith_inferences":["The accuracy figures depend on the paper's self-defined 'high quality' label: Rate-In adaptive dropout maps are the reference in Section 4.2, so a metric is 'correct' when it prefers those maps. If clinicians regard a different uncertainty pattern as preferable, rankings should be re-derived; this is an interpretation beyond the paper's experiments.","SPACE's smoothing scale $\\sigma$ is a free parameter whose value is not varied in the reported results, so its 95.83% accuracy may shift with anatomy size and slice spacing; a sensitivity analysis over $\\sigma$ would be a direct test of the metric's practical stability.","The same machinery could be applied outside prostate MRI—for example to other organ segmentations, tumor delineations, or natural-image segmentation—but the paper's evidence is limited to one dataset and one segmentation model, so transferability remains an open question.","If boundary-aligned uncertainty is what should be rewarded, BUC and BA-ECE suggest direct training objectives that penalize uncertainty that is miscalibrated or diffuse near boundaries; the paper lists this as future work rather than a demonstrated result."],"forward_implications":["SPACE, BUC, and BA-ECE give segmentation practitioners a way to identify uncertainty maps that concentrate at anatomical boundaries, which is the pattern the paper argues is clinically relevant.","BA-ECE outperforms standard ECE by 6.25 percentage points in accuracy and by about 68% in effect size, so binning by boundary distance captures calibration information that confidence-level binning misses.","Because BUC uses the predicted boundary and BA-ECE uses the ground-truth boundary, the two metrics measure different failure modes and can be reported together for quality control.","For model selection in medical imaging, these metrics could supplement voxel-wise calibration and discrimination scores when boundary regions matter for treatment planning and monitoring."],"supporting_citations":[{"why":"Supplies the prostate zonal MRI dataset and ground-truth annotations used for evaluation.","marker":"[1]"},{"why":"Monte Carlo dropout with 30 forward passes generates the uncertainty maps compared in the study.","marker":"[2]"},{"why":"Pre-trained nnU-Net model from the segmentation challenge produces the segmentations and error maps.","marker":"[6]"},{"why":"Rate-In adaptive dropout defines the 'high quality' boundary-focused uncertainty map that the metrics must identify.","marker":"[16]"},{"why":"Defines ECE and MCE, the voxel-wise calibration baselines that BA-ECE extends and outperforms.","marker":"[13]"},{"why":"Defines AUC-ROC and AUC-PR baselines used to measure discrimination accuracy.","marker":"[5]"},{"why":"Defines PAvPU, a patch-based baseline compared in Table 1 that remains insensitive to boundary structure.","marker":"[11]"},{"why":"Defines AU-ARC, a selective-prediction baseline compared in Table 1.","marker":"[12]"}],"fun_headline_variants":["Spatial scoring reveals true segmentation uncertainty","Boundary-focused metrics beat voxel-wise uncertainty checks","Context-aware uncertainty scoring improves MRI evaluation","Segmentation uncertainty: spatial structure matters","Prostate MRI uncertainty: new spatial metrics outshine"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Rate-In adaptive-dropout map is the correct 'high quality' reference, so a metric's accuracy is defined as how often it agrees with that self-chosen label; if that map is not actually the clinically better one, the reported superiority reflects agreement with the authors' own method rather than clinical value.","fun_headline_variants_meta":{"raw":{"variants":["Spatial scoring reveals true segmentation uncertainty","Boundary-focused metrics beat voxel-wise uncertainty checks","Context-aware uncertainty scoring improves MRI evaluation","Segmentation uncertainty: spatial structure matters","Prostate MRI uncertainty: new spatial metrics outshine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2878,"prompt_tokens":776,"completion_tokens":2102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":392,"completion_tokens_details":{"reasoning_tokens":2034}},"tokens_in":392,"tokens_out":2102,"duration_ms":19149,"temperature":1.0,"reasoning_tokens":2034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:36:58.921455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new dataset where uncertainty map quality is labeled by expert radiologists or by measured impact on a downstream task such as surgical planning error, and compare SPACE, BUC, and BA-ECE rankings against voxel-wise metrics; if the spatially aware metrics do not track those external quality labels more closely, the discrimination claim is an artifact of the Rate-In-derived reference.","supporting_citations":[{"cited_title":"The medical segmentation decathlon.Nature communications, 13(1):4128, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the prostate zonal MRI dataset and ground-truth annotations used for evaluation."},{"cited_title":"nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation","cited_arxiv_id":null,"evidence_quote":"Pre-trained nnU-Net model from the segmentation challenge produces the segmentations and error maps."},{"cited_title":"Staib, and John A","cited_arxiv_id":null,"evidence_quote":"Rate-In adaptive dropout defines the 'high quality' boundary-focused uncertainty map that the metrics must identify."},{"cited_title":"Obtaining well calibrated probabilities using bayesian binning","cited_arxiv_id":null,"evidence_quote":"Defines ECE and MCE, the voxel-wise calibration baselines that BA-ECE extends and outperforms."},{"cited_title":"Accuracy-rejection curves (arcs) for com- paring classification methods with a reject option","cited_arxiv_id":null,"evidence_quote":"Defines AU-ARC, a selective-prediction baseline compared in Table 1."}],"review_version":1}