{"id":"6cf69563-a988-426d-aa8f-b7163c3bf004","arxiv_id":"2411.17571","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":14,"one_line_summary":"Uncertainty maps from an ensemble of stochastic segmentation networks detect missed white matter lesions and improve automated Fazekas severity classification and segmentation quality control.","lead":"This paper compares methods for making white matter lesion segmentation models report uncertainty in brain MRI, and shows uncertainty maps catch small missed lesions and improve automated Fazekas severity scoring. It matters because it offers a practical route to more trustworthy automated assessment of small vessel disease in stroke and dementia research.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fazekas improvement from UQ features is not statistically tested; reported 95% CIs overlap, so the headline gain may be sampling noise.","rationale":"The reader's weakest assumption was the single-rater Fazekas labels. I identify a different, more immediately checkable issue: the paper's central claim that UQ features improve Fazekas classification is supported only by point estimates whose 95% confidence intervals overlap. The bootstrap protocol (1000 resampled train/test splits) makes a paired difference test trivial, but the paper does not report it. If the difference CIs include zero, the abstract's headline numbers would not demonstrate improvement. This does not diminish the paper's other contributions (silent-failure detection, QC, benchmarking), but it directly affects the novel downstream-utility claim. I also note a secondary inconsistency: the abstract states SSN-Ens yields the lowest AVD, but Table 2 shows Deep Ensembles have AVD 42.7% vs SSN-Ens 43.8%; the 'lowest' claim only holds for Top AVD. The reader's conditional verdict remains appropriate, but it should be tied to a significance test rather than only to label reliability.","tokens_in":47757,"tokens_out":9149,"duration_ms":81747,"concrete_test":"Re-run the 1000 bootstrap train/test splits and, for each split, compute the difference (SSN-Ens w/UQ minus SSN-Ens, and P-Unet w/UQ minus P-Unet) in balanced accuracy and root Brier score, for DWMH and PV separately. Report the 2.5th and 97.5th percentiles of each difference distribution. If any interval includes 0, the corresponding 'improvement' is not statistically supported at the 5% level and the abstract's wording should be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4 reports mean and 95% bootstrap CIs for each Fazekas classifier, but never reports the distribution of the difference between methods. For DWMH, SSN-Ens w/ UQ has balanced accuracy 0.74 (0.66,0.81) vs SSN-Ens without UQ 0.71 (0.63,0.77), and root Brier 0.65 (0.60,0.70) vs 0.68 (0.65,0.72); for PV, SSN-Ens w/UQ is 0.73 (0.66,0.79) vs 0.72 (0.65,0.79) and RBS 0.64 (0.60,0.69) vs 0.65 (0.61,0.70). All pairwise intervals overlap substantially. Since the bootstrap procedure resamples the same train/test splits for all methods, a paired difference test is straightforward and would be the correct evidence for the abstract's claim that 'incorporating WMH uncertainty information improves Fazekas classification performance and calibration.' Without it, the headline improvement is a point estimate compatible with zero effect. This is load-bearing because the downstream Fazekas result is the paper's main novel contribution; the silent-failure and QC results are less affected.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks several uncertainty quantification (UQ) methods for white matter hyperintensity (WMH) segmentation and evaluates their downstream utility. The authors train models on an in-house CVD dataset and evaluate on the MICCAI WMH Challenge as an out-of-distribution set, with additional qualitative and downstream analyses on ADNI and MSS3. They report that an ensemble of Stochastic Segmentation Networks (SSN-Ens) improves Dice and top-sample volume estimation, that UQ maps can highlight small deep WMH missed by the deterministic baseline, that spatial features extracted from WMH probability and UQ maps improve automated Fazekas classification over volume-only and segmentation-only features, and that UQ features improve detection of poor-quality segmentations. The experiments include six-fold cross-validation, multiple metrics, bootstrapped confidence intervals, hyperparameter sweeps, and both standard U-Net and nnU-Net backbones.","tokens_in":48120,"tokens_out":7303,"duration_ms":71406,"significance":"The paper addresses a clinically relevant problem and contains several strengths: it compares a wide range of UQ techniques on real out-of-distribution data, ships code, uses public Challenge ground truth for segmentation benchmarking, and includes qualitative analysis of silent failures and stroke-lesion confusion. If the Fazekas claim were statistically robust, the paper would make a valuable contribution by showing that UQ maps carry actionable information beyond the mean segmentation. However, the principal novel claim—that adding UQ features improves Fazekas classification—rests on overlapping confidence intervals and labels produced by a single rater, so the current evidence is weaker than the abstract suggests.","major_comments":[{"comment":"The abstract's headline claim that incorporating WMH uncertainty information improves Fazekas classification and calibration is not supported by the reported statistics. For DWMH, SSN-Ens with UQ has balanced accuracy 0.74 (95% CI 0.66, 0.81) versus 0.71 (0.63, 0.77) without UQ, and root Brier 0.65 (0.60, 0.70) versus 0.68 (0.65, 0.72). For PV WMH the corresponding values are 0.73 (0.66, 0.79) versus 0.72 (0.65, 0.79) and 0.64 (0.60, 0.69) versus 0.65 (0.61, 0.70). All pairwise intervals overlap substantially. Because the bootstrap resamples the same train/test splits for every feature set, a paired difference test or a bootstrap confidence interval on the difference is straightforward and should be reported. Without this, the claimed improvement is a point estimate compatible with zero effect. The same concern applies to the QC results in Table 5, where the balanced-accuracy intervals for SSN-Ens with and without UQ overlap (0.82 (0.71, 0.91) versus 0.79 (0.70, 0.87)).","section":"Table 4 and Section 3.5.3"},{"comment":"Fazekas labels for the Challenge and ADNI datasets were generated by a single expert rater for this study, and no inter-rater reliability or adjudication is reported. The MSS3 evaluation is then interpreted through a post hoc 'rater policy shift' explanation (Section 4.2.1), supported only by volume distributions per class rather than by any independent rating of the same images. If the labels are noisy or carry rater-specific criteria, the relative ranking of volume-only, spatial-feature, and UQ-feature classifiers in Table 4, and the conclusion that 'the model does generalise well to the MSS3 dataset given the rater policy disagreement', are not established. At minimum, the authors should report a second rater on a subset of images, or explicitly frame the single-rater design and the MSS3 interpretation as limitations and soften the corresponding conclusions.","section":"Section 3.1 and Section 4.2.1"},{"comment":"The abstract states that SSN-Ens 'yields the highest Dice and lowest Absolute Volume Difference % (AVD) score'. Table 2 shows that Deep Ensembles has a lower mean AVD (42.7%) than SSN-Ens (43.8%); SSN-Ens's advantage is in Top AVD (12.3% versus 19.2%). The text should specify 'lowest Top AVD' or otherwise reconcile the claim with the table, since the current wording is contradicted by the paper's own results.","section":"Abstract and Table 2"}],"minor_comments":[{"comment":"The notation in Eqs. (12) and (13) is confusing: the two standard-deviation conventions are described in prose, but the equations do not explicitly define the subscripts on the metric E. Please define E_{r,s} clearly so the reader can follow which average is taken over model runs versus subjects.","section":"Section 3.4.3"},{"comment":"The first sentence of Section 4.1 reads 'All methods outperform the baseline SEnt Dice score of for the standard models' and appears to be missing the baseline value; the sentence should be corrected.","section":"Section 4.1"},{"comment":"The caption contains the typo 'Periventriclar' and should read 'Periventricular'.","section":"Figure 2 caption"},{"comment":"The first sentence of Section 4.2.1 refers to 'PVWM' where the intended abbreviation is 'PVWMH'.","section":"Section 4.2.1"},{"comment":"The definition of 'poor quality' as the bottom 20th percentile of the SEnt baseline Dice is arbitrary but useful; please state explicitly that the QC experiment measures detection of low Dice relative to the deterministic baseline, not segmentation quality in general.","section":"Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a good fit for Medical Image Analysis and the experimental breadth is commendable. The requested revisions are feasible within the paper's scope: paired bootstrap tests on the existing resampling scheme, a second-rater reliability substudy or a clearly stated limitation, and a correction of the AVD claim in the abstract. I would not reject the paper, but the central Fazekas claim needs stronger statistical and label-reliability support before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a thorough, honest empirical study of UQ for WMH segmentation, and the missing-lesion coverage result—SSN-Ens flagging small deep WMH that the deterministic baseline silently misses—is genuinely useful. The Fazekas downstream claim is where it gets shaky. The abstract says adding UQ features improves classification and calibration, but the marginal 95% CIs for SSN-Ens w/ UQ versus without overlap on every metric (e.g., DWMH balanced accuracy 0.74 [0.66,0.81] vs 0.71 [0.63,0.77]). No paired difference test is reported, so the headline improvement is compatible with zero effect. That is load-bearing because the Fazekas result is the paper's main novel contribution.\n\nWhat is solid: the benchmark compares seven UQ methods on a real OOD dataset (WMH Challenge), with six-fold CV, bootstrap CIs, hyperparameter sweeps, and released code. The instance-level coverage metrics for silent failure are a good addition, and the qualitative analysis of stroke-lesion ambiguity is carefully done. The QC result (UQ features improve detection of low-quality segmentations) is more convincing, though the poor-quality labels come from a Dice threshold on the SEnt baseline—a mild circularity.\n\nSoft spots beyond the missing paired test: Fazekas ground truth for Challenge and ADNI was created by a single expert rater with no inter-rater reliability; the MSS3 'rater policy shift' explanation is post hoc; and there is no comparison to prior automated Fazekas classifiers (e.g., Joo et al. or Mu et al.). The first is worth a comment in the paper; the second is plausible but not proven.\n\nBottom line: the paper deserves a serious referee. The UQ benchmark and silent-failure analysis are solid and will be cited. But the Fazekas claim needs a paired test or softer wording, and the single-rater issue needs acknowledging. I would not desk reject; I'd send to review with the expectation of revision.","headline":"Solid UQ benchmark for WMH segmentation, but the headline Fazekas improvement lacks statistical support—overlapping 95% CIs make it compatible with noise.","tokens_in":48684,"tokens_out":2832,"would_cite":true,"duration_ms":78629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Uncertainty maps from a stochastic segmentation-network ensemble reveal small deep-white-matter lesions that the deterministic model silently misses, and adding uncertainty features to automated Fazekas grading lifts deep-WMH balanced…","keywords":["uncertainty quantification","white matter hyperintensity segmentation","Fazekas score prediction","deep ensembles","stochastic segmentation networks","brain MRI","silent failure detection","quality control"],"falsifier":"Have several independent expert raters re-grade the Challenge and ADNI scans and re-run the Fazekas comparison against consensus or multi-rater labels; if the deep-WMH balanced accuracy advantage of SSN-Ens UQ features over volume-only features (0.74 versus 0.62) shrinks or disappears under those labels, the claimed benefit of uncertainty features for Fazekas grading is an artifact of a single rater's policy.","tokens_in":47569,"feed_emoji":"🧠","tokens_out":11045,"duration_ms":88507,"temperature":0.7,"pith_summary":"White matter hyperintensities (WMH), bright spots in brain MRI that mark small-vessel disease, are usually graded by radiologists on the four-point Fazekas scale, while automated segmentation models often miss the smallest deep-white-matter lesions without warning. This paper claims that uncertainty quantification (UQ) closes that gap: the uncertainty map of a combined stochastic-segmentation-network and deep-ensemble model (SSN-Ens) flags small isolated WMH clusters that the deterministic baseline leaves unsegmented, and marks stroke lesions that look like WMH as ambiguous. It also proposes a Fazekas classifier that combines spatial features from WMH probability maps with features from UQ maps, reporting a deep-WMH balanced accuracy of 0.74 versus 0.67 with spatial features alone and 0.62 with volume alone, with better calibration (root Brier 0.65 versus 0.74). The same UQ features improve automated detection of poor-quality segmentations. If the paper is right, uncertainty maps are not a by-product of segmentation but a clinically usable signal for completeness, severity grading, and quality control.","feed_headline":"Catch silent failures: uncertainty maps find lesions AI misses","feed_subtitle":"Adding uncertainty features lifts deep-region Fazekas accuracy from 0.62 to 0.74 and flags missed lesions.","key_machinery":"The load-bearing object is SSN-Ens: an ensemble of Stochastic Segmentation Networks, where each member places a low-rank Gaussian distribution over logits to capture spatially correlated aleatoric uncertainty, and the ensemble of differently initialised members captures epistemic uncertainty. The UQ map is the predictive entropy of the averaged sample distribution, computed from ten inference samples. The second piece of machinery is the Fazekas feature extractor: rings at 5, 10, and 15 mm from the ventricles, within which volume, connected-component counts, and confluence features are computed from thresholded WMH probability and UQ maps, selected by recursive feature elimination and fed to a regularised logistic regression. The UQ features carry the downstream claim because they represent ambiguous penumbral tissue and small deep lesions that volume or mean-segmentation features do not encode.","core_discovery":"The paper's central claim is that voxelwise uncertainty maps carry information the mean segmentation throws away, and that this information is useful in the clinic. Concretely, it claims that SSN-Ens, an ensemble of ten Stochastic Segmentation Networks that each model spatially correlated aleatoric uncertainty while the ensemble covers epistemic uncertainty, produces uncertainty maps that identify small isolated deep WMH clusters the deterministic baseline neither segments nor flags as uncertain, highlight ambiguity between WMH and similar-looking stroke lesions, and supply spatial features that improve automated Fazekas severity grading. On the out-of-distribution Challenge data, SSN-Ens attains the highest Dice (0.70 for the standard model, 0.72 with the nnUNet backbone) and the lowest top-sample absolute volume difference (12.3% and 11.0%). In the Fazekas task its UQ features raise deep-WMH balanced accuracy from 0.62 (volume only) and 0.67 (spatial features only) to 0.74, with a root Brier score of 0.65 versus 0.74 for volume only, and the same features lift poor-segmentation detection from 0.74 to 0.82 balanced accuracy.","pith_inferences":["The ventricle-ring plus uncertainty-feature recipe is a general template for automating other ordinal visual rating scales in neuroimaging, such as perivascular-space severity, wherever a clinician grades from spatial pattern rather than total lesion volume.","A direct testable extension, which the paper only gestures at in its future-work section, is longitudinal: voxels flagged as uncertain but unsegmented in normal-appearing white matter may be the sites of future WMH, and follow-up scans could confirm this.","The MSS3 rater-policy finding implies that any automated Fazekas system trained on one institution's labels will systematically under- or over-grade elsewhere even on identical anatomy; training on multi-rater consensus labels, which this study does not do, would be the necessary remedy and a natural next experiment.","Because only total predictive entropy is used, the quality-control signal may be stronger if decomposed into epistemic versus aleatoric components; re-running the QC task on each component separately would show where the improvement actually comes from."],"forward_implications":["Uncertainty maps can be used as a lesion-level completeness check: regions of high uncertainty are worth a second look for small deep WMH that the segmentation missed.","Automated Fazekas grading should use spatial features rather than total WMH volume: spatial features alone raise deep-WMH balanced accuracy from 0.62 to 0.67, and SSN-Ens UQ features raise it to 0.74 with a better root Brier score.","Segmentation quality control can run on UQ features: the QC classifier reaches 0.82 balanced accuracy and a 0.50 root Brier score with SSN-Ens UQ features, versus 0.74 and 0.60 with volume only.","Sample diversity is what makes a stochastic method useful: high-diversity methods such as SSN-Ens bracket the true WMH volume in their sample distribution even for small lesions, while low-diversity methods such as Probabilistic U-Net and independent-voxel SSN do not.","Combining aleatoric and epistemic uncertainty modelling outperforms either alone, and the SSN-Ens advantage in Dice, top-sample AVD, and missed-lesion coverage persists when the method is re-implemented on an nnUNet backbone."],"supporting_citations":[{"why":"Defines the four-point periventricular and deep WMH severity scale that the downstream classifier is trained and evaluated to predict.","marker":"Fazekas et al., 1987"},{"why":"Introduces Stochastic Segmentation Networks, whose low-rank logit covariance gives SSN-Ens its spatially coherent samples and its ability to flag small missed lesions.","marker":"Monteiro et al., 2020"},{"why":"Provides the deep-ensemble component that supplies the epistemic half of SSN-Ens's uncertainty.","marker":"Lakshminarayanan et al., 2017"},{"why":"Defines the Probabilistic U-Net, the main alternative aleatoric model whose sample diversity is shown to be too low for small WMH.","marker":"Kohl et al., 2018"},{"why":"Supplies the WMH Challenge dataset used as the primary out-of-distribution evaluation of segmentation and uncertainty quality.","marker":"Kuijf et al., 2019"},{"why":"Establishes WMH volume as a Fazekas predictor, the baseline that the volume-only features in the Fazekas comparison are built on.","marker":"Andere et al., 2022"},{"why":"Introduces Monte-Carlo dropout, one of the benchmarked UQ techniques and a reference point for sample diversity.","marker":"Gal and Ghahramani, 2016"},{"why":"Provides the nnUNet backbone and training paradigm used to confirm that the SSN-Ens benefits transfer to a different architecture.","marker":"Isensee et al., 2021"},{"why":"Supplies the residual U-Net backbone and augmentation scheme on which all standard-model UQ methods are trained.","marker":"Mojiri Forooshani et al., 2022"}],"fun_headline_variants":["Uncertainty maps catch missed lesions, lift Fazekas accuracy","Silent failures revealed: UQ improves WMH grading","Adding uncertainty features boosts deep-WMH grading from 0.62 to 0.74","AI uncertainty spots overlooked WMH and enhances scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Fazekas grades used as ground truth for the Challenge and ADNI datasets were produced by a single expert rater for this study, with no second rater or adjudication, so the headline improvement from uncertainty features would be unsupported if those labels are noisy or follow a different policy than other raters.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty maps catch missed lesions, lift Fazekas accuracy","Silent failures revealed: UQ improves WMH grading","Adding uncertainty features boosts deep-WMH grading from 0.62 to 0.74","AI uncertainty spots overlooked WMH and enhances scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1615,"prompt_tokens":1154,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":770,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":770,"tokens_out":461,"duration_ms":20811,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:58:58.682591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several independent expert raters re-grade the Challenge and ADNI scans and re-run the Fazekas comparison against consensus or multi-rater labels; if the deep-WMH balanced accuracy advantage of SSN-Ens UQ features over volume-only features (0.74 versus 0.62) shrinks or disappears under those labels, the claimed benefit of uncertainty features for Fazekas grading is an artifact of a single rater's policy.","supporting_citations":[],"review_version":1}