{"id":"7a9013fd-127f-4e3b-bccb-28ba3449b18e","arxiv_id":"2506.14497","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Maximum-entropy regularization modestly changes uncertainty estimates and calibration for white matter hyperintensity segmentation under domain shift, but the numerical support for stronger uncertainty-error correlation is mixed.","lead":"This paper tests whether maximum-entropy regularization makes MRI brain-lesion segmentation models better at saying 'I am unsure' when the scanner or patient population changes. On two public datasets, the authors find entropy-based uncertainty can flag segmentation errors, though the reported improvements are small and partly inconsistent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported correlations contradict the headline claim: CE_KL (-0.807) is weaker than CE (-0.826) and CE_MEALL (-0.861) is strongest, so 'maximum-entropy regularization further strengthens the correlation' is not supported by the paper's own numbers.","rationale":"The reader's weakest_assumption concerns the ID/OOD dataset pairing, but the more immediate problem is internal: the paper's own numbers in Section 3.1 contradict its central claim. I checked the four Pearson coefficients stated in the text and found that CE_KL (-0.807) has weaker magnitude than CE (-0.826), and CE_MEALL (-0.861) is strongest, directly contradicting the sentence 'CE_MEEP and CE_KL exhibiting stronger negative correlations ... compared to CE ... and CE_MEALL'. The abstract and conclusions assert that maximum-entropy regularization strengthens the uncertainty-performance correlation; this is not what the reported correlations show. A secondary issue is that the correlation is likely pooled across ID and OOD, which can create a spurious negative association due to the domain mean shift; separate within-domain correlations are needed to support the 'proxy for segmentation errors' claim. The reader's rationale does mention the correlation inconsistency, but their weakest_assumption is about dataset composition; I consider the internal numerical contradiction to be more load-bearing. A single reanalysis that reports per-domain correlations and CIs would settle whether the claim holds. Until then, the central claim is unsupported, so I would move the verdict to REJECT, though a corrected analysis could restore a conditional acceptance.","tokens_in":19468,"tokens_out":5341,"duration_ms":52196,"concrete_test":"Recompute the Pearson correlation between per-scan mean foreground entropy and Dice from the data underlying Figure 3, separately for ID scans, OOD scans, and pooled, with bootstrap 95% confidence intervals. Then test the inequalities |r_CE_MEEP| > |r_CE| and |r_CE_KL| > |r_CE|; if either fails, the claim that entropy regularization strengthens the error proxy is unsupported. Also report the ECE values numerically and the grid-search lambda for each loss.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.1 reports Pearson correlations between mean foreground entropy and Dice: CE -0.826, CE_MEEP -0.835, CE_KL -0.807, CE_MEALL -0.861. The text then states that CE_MEEP and CE_KL 'exhibiting stronger negative correlations' compared to CE and CE_MEALL. This is numerically false: -0.807 is weaker (less negative) than -0.826, and CE_MEALL is the strongest. Therefore the abstract's claim that 'maximum-entropy regularization further strengthens the correlation between uncertainty and segmentation performance' is contradicted by the paper's own reported results. Only CE_MEEP improves over CE, and the difference (-0.835 vs -0.826) is tiny with no confidence intervals or significance test. Moreover, the correlation appears to be computed on pooled ID+OOD scans; if so, it may reflect the between-domain mean shift (OOD has higher entropy and lower Dice) rather than a per-scan proxy within each domain, and the paper does not report separate ID and OOD correlations. The central claim therefore lacks quantitative support as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies uncertainty estimation for white matter hyperintensity (WMH) segmentation under domain shift. The authors train a 3D U-Net on the WMH Segmentation Challenge dataset and evaluate on the 3D-MR-MS dataset as out-of-distribution (OOD). They compare cross-entropy (CE) training with three maximum-entropy regularizers: MEEP (maximum entropy on erroneous predictions), KL (KL divergence to uniform on erroneous predictions), and MEALL (maximum entropy on all predictions). The evaluation uses Dice, Hausdorff distance, expected calibration error (ECE), and the Pearson correlation between mean foreground entropy and Dice across scans. The paper claims that entropy-based uncertainty can serve as a proxy for segmentation errors and that maximum-entropy regularization strengthens this correlation while improving calibration under domain shift.","tokens_in":19660,"tokens_out":4433,"duration_ms":42504,"significance":"If the claims were fully supported, this would be a useful contribution to reliable medical image segmentation under distribution shift, with direct clinical relevance for flagging unreliable predictions. The paper addresses an important problem, uses two publicly available datasets, and provides a comparative study of three regularization strategies, which is a strength. However, the central quantitative claim is contradicted by the paper's own reported numbers, and the evaluation protocol has a circularity concern because the MEEP and KL regularizers use ground-truth labels to penalize erroneous predictions during training. These issues materially weaken the headline conclusions, although the underlying experimental work is partially salvageable with reanalysis and reframing.","major_comments":[{"comment":"The text states that CE_MEEP and CE_KL 'exhibiting stronger negative correlations (−0.835 and −0.807, respectively) compared to CE (−0.826) and CE_MEALL (−0.861).' This is numerically false: −0.807 is a weaker (less negative) correlation than −0.826, and CE_MEALL actually has the strongest correlation at −0.861. As written, the paper's own numbers contradict the abstract's claim that 'maximum-entropy regularization further strengthens the correlation between uncertainty and segmentation performance.' Only CE_MEEP shows a negligible improvement over CE (−0.835 vs. −0.826), and no confidence intervals or significance tests are provided for any of these differences. The authors must recompute or re-report the correlations, provide separate ID and OOD correlations, and revise the claims accordingly.","section":"Section 3.1, Figure 3"},{"comment":"The MEEP and KL regularizers are defined using ground-truth labels to identify erroneous predictions and explicitly penalize low entropy only on those voxels. Therefore, the observation in Section 3.1 and Figure 4 that erroneous predictions (FP/FN) have higher entropy, and that entropy anti-correlates with Dice, is partly a direct consequence of the training objective rather than an independent property of the resulting models. To support the claimed use of entropy as a proxy for segmentation errors without ground truth at inference, the authors should evaluate error-detection performance on a held-out validation set that was not used for training, or compare against a model trained without label-dependent regularization, and report metrics such as AUROC for flagging erroneous voxels or scans.","section":"Sections 2.2.2 and 3.1"},{"comment":"The OOD dataset (3D-MR-MS) differs from the WMH Segmentation Challenge dataset not only in scanner and acquisition parameters but also in disease type (multiple sclerosis vs. aging-related vascular disease), patient age (median 39 vs. mean 70), and lesion morphology. Consequently, the observed differences in calibration and uncertainty cannot be attributed specifically to domain shift as defined in the paper; they may reflect disease-related distributional differences. The authors should either use a same-disease scanner-shift evaluation protocol or explicitly acknowledge and discuss this confounding factor as a limitation.","section":"Section 2.4, Section 3.1"},{"comment":"The regularization weight λ for each strategy was selected by grid search 'balancing segmentation performance and the quality of uncertainty estimation.' No details are provided about the validation set used for this selection, the exact objective function, or whether the correlations reported in Section 3.1 are computed on the same data used for model selection. This creates a risk of overfitting to the evaluation metric and makes the comparison of correlation coefficients difficult to interpret. The authors should specify the selection criterion and the data split used for choosing λ, and preferably report the sensitivity of the main results to λ.","section":"Section 2.5"}],"minor_comments":[{"comment":"The expression for L_a contains a double negative that makes the formula ambiguous; it should be written as L_a(Y) = -H_b(Y) = sum over erroneous voxels of [y_i log2(y_i) + (1-y_i)log2(1-y_i)], but as typeset the minus signs are confusing.","section":"Section 2.2.1"},{"comment":"The notation for ground-truth and predicted labels is inconsistent: the text says that Y is the ground truth and \\hat{Y} is the predicted mask, but the loss equation is formatted ambiguously and appears to use Y for both. Please use consistent notation throughout.","section":"Section 2.2, Eq. (1)"},{"comment":"The caption refers to 'blue bars' and 'orange bars,' but the figure shows colored points, not bars; please correct the caption to match the plot.","section":"Figure 4 caption"},{"comment":"The text mentions that linear regression lines are fitted to each set of data points, but no slopes or intercepts are reported, only Pearson correlation coefficients. Either report the regression results or remove the mention of fitted lines.","section":"Section 3.1"},{"comment":"The claim that CE_KL demonstrates superior calibration in the OOD scenario is supported only by ECE values shown in the figure; please report the exact ECE numbers in the text so the reader does not have to infer them from the figure.","section":"Section 3.3, Figure 7"},{"comment":"There are several typos, including 'the the overall loss function' in Section 2.2.1 and 'condi' in the corresponding author's email address; please proofread the manuscript.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central quantitative claim is contradicted by its own reported numbers, which is a serious issue for the abstract and conclusions. However, the empirical study has merit, and the issues appear fixable through reanalysis, additional experiments, and reframing of the claims. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central comparative claim is undercut by its own numbers. It says MEEP and KL give stronger negative correlations between entropy and Dice, but the reported Pearson coefficients are CE -0.826, CE_MEEP -0.835, CE_KL -0.807, CE_MEALL -0.861. So KL is weaker than CE, MEEP is only marginally stronger, and MEALL is actually the strongest. The abstract and discussion repeat this incorrect comparison. That is the first thing you should know.\n\nWhat the paper does well: it is a clean, well-motivated application study. It takes two existing maximum-entropy regularizers (MEEP and KL from Larrazabal et al. 2023 and Pereyra et al. 2017), applies them to WMH segmentation in a multicentric setting, and evaluates calibration and uncertainty-error correlation on two public datasets. The experimental setup is standard: a 3D U-Net, Dice/Hausdorff/ECE/entropy metrics, and separate ID and OOD datasets. The lesion-size analysis and the voxel-level error breakdown (TP/FP/FN) are useful additions. The central phenomenon, that entropy can serve as a failure proxy, is supported qualitatively across all losses.\n\nWhere it falls short. The numeric contradiction is the load-bearing issue. No confidence intervals or significance tests accompany the correlation differences, and the differences are small (e.g., -0.826 vs -0.835). Lambda values from the grid search are not reported, and ECE is only shown in figures. More importantly, the correlations appear to be computed on pooled ID+OOD scans; if so, they mostly reflect the between-domain shift (OOD has higher entropy and lower Dice), not a per-scan proxy within a deployment domain. The paper needs per-domain correlations and a discussion of what the pooled number actually means. Also, MEEP and KL require ground-truth labels during training to select erroneous voxels, so the observed uncertainty-error alignment is partly a consequence of the training objective, not an independent discovery. The paper does not claim otherwise, but it should be explicit about this limitation.\n\nWho this is for: researchers working on uncertainty quantification in medical image segmentation, especially for MS/WMH. As an empirical study, it is useful but not definitive. It deserves a serious referee, but only with major revision: correct the claims, report hyperparameters, add error bars and per-domain correlations.","headline":"Application study with a solid setup but a central claim that its own reported correlations contradict; needs careful revision before acceptance.","tokens_in":20241,"tokens_out":2078,"would_cite":false,"duration_ms":26598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Maximum-entropy regularization makes a lesion segmentation model's predictive entropy a reliable, label-free proxy for its own errors under domain shift, and improves calibration on out-of-distribution MRI.","keywords":["white matter hyperintensity segmentation","uncertainty estimation","domain shift","maximum entropy regularization","model calibration","predictive entropy","multiple sclerosis MRI","out-of-distribution detection"],"falsifier":"Retrain the same U-Net with the same four losses but evaluate on a held-out WMH Challenge scanner (same disease, different acquisition site) instead of 3D-MR-MS; if the entropy-Dice anti-correlation and ECE gains over plain CE disappear, the reported OOD benefit is an artifact of dataset differences rather than domain shift.","tokens_in":19223,"feed_emoji":"🧠","tokens_out":9073,"duration_ms":89386,"temperature":0.7,"pith_summary":"White matter hyperintensity (WMH) segmentation models become overconfident when MRI data come from a new scanner or population, so their probability outputs cannot be trusted to flag bad segmentations. The authors set out to show that training with maximum-entropy regularization fixes this: adding a term that pushes erroneous voxel predictions toward a uniform distribution makes the model's entropy higher when it is wrong and keeps predicted probabilities closer to true accuracy. The result matters because a deployed segmentation tool can then use entropy, computed without ground-truth labels, to flag scans and regions that need human review. The paper studies a U-Net trained on the WMH Segmentation Challenge and tested on the 3D-MR-MS multiple sclerosis dataset, reporting stronger entropy-Dice anticorrelation and better calibration for regularized variants, with the KL-based variant remaining the most calibrated out-of-distribution.","feed_headline":"Entropy training makes brain-lesion errors visible on new MRI scanners","feed_subtitle":"Maximum-entropy training keeps a U-Net's uncertainty aligned with white-matter segmentation quality on unseen MRI","key_machinery":"The load-bearing object is the binary entropy of a voxel prediction: $H_b(p) = -p\\log_2 p - (1-p)\\log_2(1-p)$, which is 0 for confident predictions and 1 at $p = 0.5$. The paper trains with an extra loss term that maximizes this entropy on selected voxels. The overall-confidence penalty (CE_MEALL) applies it to all voxels; MEEP applies it only to voxels whose predicted class disagrees with the ground-truth label; and the KL variant replaces entropy maximization by minimizing the KL divergence between the erroneous predictions and the uniform distribution. These terms change the model's probability outputs, and the paper measures their effect with Dice, expected calibration error, and the Pearson correlation between per-scan mean foreground entropy and Dice.","core_discovery":"The paper's central claim is that predictive entropy can serve as a label-free proxy for segmentation errors in WMH segmentation, and that maximum-entropy regularization makes that proxy stronger under domain shift. In experiments with a 3D U-Net trained on the WMH Segmentation Challenge and evaluated on 3D-MR-MS as out-of-distribution data, the mean entropy of voxels predicted as lesions is anticorrelated with Dice across scans in both settings. Regularizing with maximum entropy on erroneous predictions (CE_MEEP) or via KL divergence to a uniform distribution (CE_KL) raises uncertainty for false positives and false negatives, widens the entropy gap between in-distribution and out-of-distribution cases, and lowers expected calibration error; the KL variant is the best-calibrated on out-of-distribution data. Reported Pearson correlations between mean foreground entropy and Dice are negative for all four training strategies, for example -0.826 for plain cross-entropy and -0.835 for CE_MEEP.","pith_inferences":["Beyond the paper: because the regularizer is a loss term, the same approach could carry to other MRI segmentation targets with ambiguous boundaries, such as brain tumors or prostate, though the paper only tests WMH.","Beyond the paper: the paper does not test referral decisions; an extension would be to set an entropy threshold for automatic expert review and measure how many missed lesions are caught per review hour.","Beyond the paper: the reported scan-level correlations could be complemented by voxel-level or per-lesion uncertainty metrics, since averaging entropy over foreground voxels may obscure small-lesion signals in large-lesion scans."],"forward_implications":["Clinicians can rank scans by mean foreground entropy and inspect the most uncertain ones first, since high entropy tracks low Dice without requiring labels.","Regularized models will be safer in multi-center deployment because their reported confidence is closer to actual accuracy, especially the KL variant under shift.","Small lesions, which are hardest to segment and where raters often disagree, will be preferentially flagged for review because they carry higher entropy.","The protocol can be applied to any segmentation model with a probabilistic output, since the regularizer is a loss term rather than an architectural change.","Improving calibration under domain shift may also improve downstream decisions that threshold probabilities for lesion load estimation."],"supporting_citations":[{"why":"supplies the entropy penalty on confident output distributions that the paper adapts into its overall-confidence regularizer","marker":"Pereyra et al., 2017"},{"why":"introduces maximum entropy on erroneous predictions, the MEEP loss that is central to this paper's uncertainty experiments","marker":"Larrazabal et al., 2023"},{"why":"shows entropy-based uncertainty applied to MS lesion segmentation, the baseline capability this paper extends to domain shift","marker":"Nair et al., 2020"},{"why":"provides the WMH Segmentation Challenge dataset used as the in-distribution training and evaluation domain","marker":"Kuijf et al., 2019"},{"why":"provides the 3D-MR-MS dataset used as the out-of-distribution test domain","marker":"Lesjak et al., 2018"},{"why":"frames the evaluation of predictive uncertainty under dataset shift, motivating the OOD protocol","marker":"Ovadia et al., 2019"},{"why":"supplies the expected calibration error framework and the observation that cross-entropy gives better-calibrated segmentation","marker":"Mehrtash et al., 2020"},{"why":"defines the U-Net architecture that carries all segmentation and entropy experiments","marker":"Ronneberger et al., 2015"}],"fun_headline_variants":["Entropy reveals white-matter lesion errors on unseen MRI","Uncertainty predicts WMH lesion failures under domain shift","Entropy-based training improves calibration for out-of-distribution MRI","Label-free error proxy for white-matter lesions using entropy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on treating the WMH Challenge and 3D-MR-MS comparison as domain shift; if scanner differences are entangled with disease type, lesion morphology, and preprocessing, the uncertainty gains cannot be specifically attributed to domain shift.","fun_headline_variants_meta":{"raw":{"variants":["Entropy reveals white-matter lesion errors on unseen MRI","Uncertainty predicts WMH lesion failures under domain shift","Entropy-based training improves calibration for out-of-distribution MRI","Label-free error proxy for white-matter lesions using entropy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3238,"prompt_tokens":910,"completion_tokens":2328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2260}},"tokens_in":526,"tokens_out":2328,"duration_ms":16879,"temperature":1.0,"reasoning_tokens":2260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:21.824302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same U-Net with the same four losses but evaluate on a held-out WMH Challenge scanner (same disease, different acquisition site) instead of 3D-MR-MS; if the entropy-Dice anti-correlation and ECE gains over plain CE disappear, the reported OOD benefit is an artifact of dataset differences rather than domain shift.","supporting_citations":[],"review_version":1}