Pith. sign in

REVIEW 4 major objections 6 minor 3 references

Towards Reliable WMH Segmentation under Domain Shift: An Application Study using Maximum Entropy Regularization to Improve Uncertainty Estimation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Maximum-entropy regularization makes a lesion segmentation model's predictive entropy a reliable, label-free proxy for its own errors under domain shift, and improves calibration on out-of-distribution MRI.

desk verdict Application study with a solid setup but a central claim that its own reported correlations contradict; needs careful revision before acceptance. read the letter →

arxiv 2506.14497 v1 pith:Y7WLZZKA submitted 2025-06-17 eess.IV cs.CV

Franco Matzkin , Agostina Larrazabal , Diego H Milone , Jose Dolz , Enzo Ferrante This is my paper · ORCID
classification eess.IVcs.CV
keywords whitematterhyperintensitysegmentationuncertaintyestimationdomainshiftmaximumentropyregularizationmodelcalibrationpredictivemultiplesclerosisMRIout-of-distributiondetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

White matter hyperintensity (WMH) segmentation models become overconfident when MRI data come from a new scanner or population, so their probability outputs cannot be trusted to flag bad segmentations. The authors set out to show that training with maximum-entropy regularization fixes this: adding a term that pushes erroneous voxel predictions toward a uniform distribution makes the model's entropy higher when it is wrong and keeps predicted probabilities closer to true accuracy. The result matters because a deployed segmentation tool can then use entropy, computed without ground-truth labels, to flag scans and regions that need human review. The paper studies a U-Net trained on the WMH Segmentation Challenge and tested on the 3D-MR-MS multiple sclerosis dataset, reporting stronger entropy-Dice anticorrelation and better calibration for regularized variants, with the KL-based variant remaining the most calibrated out-of-distribution.

What carries the argument

The load-bearing object is the binary entropy of a voxel prediction: $H_b(p) = -p\log_2 p - (1-p)\log_2(1-p)$, which is 0 for confident predictions and 1 at $p = 0.5$. The paper trains with an extra loss term that maximizes this entropy on selected voxels. The overall-confidence penalty (CE_MEALL) applies it to all voxels; MEEP applies it only to voxels whose predicted class disagrees with the ground-truth label; and the KL variant replaces entropy maximization by minimizing the KL divergence between the erroneous predictions and the uniform distribution. These terms change the model's probability outputs, and the paper measures their effect with Dice, expected calibration error, and the Pearson correlation between per-scan mean foreground entropy and Dice.

What would settle it

Retrain the same U-Net with the same four losses but evaluate on a held-out WMH Challenge scanner (same disease, different acquisition site) instead of 3D-MR-MS; if the entropy-Dice anti-correlation and ECE gains over plain CE disappear, the reported OOD benefit is an artifact of dataset differences rather than domain shift.

Watch

Extended reading notes

Core claim

The paper's central claim is that predictive entropy can serve as a label-free proxy for segmentation errors in WMH segmentation, and that maximum-entropy regularization makes that proxy stronger under domain shift. In experiments with a 3D U-Net trained on the WMH Segmentation Challenge and evaluated on 3D-MR-MS as out-of-distribution data, the mean entropy of voxels predicted as lesions is anticorrelated with Dice across scans in both settings. Regularizing with maximum entropy on erroneous predictions (CE_MEEP) or via KL divergence to a uniform distribution (CE_KL) raises uncertainty for false positives and false negatives, widens the entropy gap between in-distribution and out-of-distribution cases, and lowers expected calibration error; the KL variant is the best-calibrated on out-of-distribution data. Reported Pearson correlations between mean foreground entropy and Dice are negative for all four training strategies, for example -0.826 for plain cross-entropy and -0.835 for CE_MEEP.

Load-bearing premise

Everything rests on treating the WMH Challenge and 3D-MR-MS comparison as domain shift; if scanner differences are entangled with disease type, lesion morphology, and preprocessing, the uncertainty gains cannot be specifically attributed to domain shift.

Editorial extensions

If this is right

  • Clinicians can rank scans by mean foreground entropy and inspect the most uncertain ones first, since high entropy tracks low Dice without requiring labels.
  • Regularized models will be safer in multi-center deployment because their reported confidence is closer to actual accuracy, especially the KL variant under shift.
  • Small lesions, which are hardest to segment and where raters often disagree, will be preferentially flagged for review because they carry higher entropy.
  • The protocol can be applied to any segmentation model with a probabilistic output, since the regularizer is a loss term rather than an architectural change.
  • Improving calibration under domain shift may also improve downstream decisions that threshold probabilities for lesion load estimation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the regularizer is a loss term, the same approach could carry to other MRI segmentation targets with ambiguous boundaries, such as brain tumors or prostate, though the paper only tests WMH.
  • Beyond the paper: the paper does not test referral decisions; an extension would be to set an entropy threshold for automatic expert review and measure how many missed lesions are caught per review hour.
  • Beyond the paper: the reported scan-level correlations could be complemented by voxel-level or per-lesion uncertainty metrics, since averaging entropy over foreground voxels may obscure small-lesion signals in large-lesion scans.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies uncertainty estimation for white matter hyperintensity (WMH) segmentation under domain shift. The authors train a 3D U-Net on the WMH Segmentation Challenge dataset and evaluate on the 3D-MR-MS dataset as out-of-distribution (OOD). They compare cross-entropy (CE) training with three maximum-entropy regularizers: MEEP (maximum entropy on erroneous predictions), KL (KL divergence to uniform on erroneous predictions), and MEALL (maximum entropy on all predictions). The evaluation uses Dice, Hausdorff distance, expected calibration error (ECE), and the Pearson correlation between mean foreground entropy and Dice across scans. The paper claims that entropy-based uncertainty can serve as a proxy for segmentation errors and that maximum-entropy regularization strengthens this correlation while improving calibration under domain shift.

Significance. If the claims were fully supported, this would be a useful contribution to reliable medical image segmentation under distribution shift, with direct clinical relevance for flagging unreliable predictions. The paper addresses an important problem, uses two publicly available datasets, and provides a comparative study of three regularization strategies, which is a strength. However, the central quantitative claim is contradicted by the paper's own reported numbers, and the evaluation protocol has a circularity concern because the MEEP and KL regularizers use ground-truth labels to penalize erroneous predictions during training. These issues materially weaken the headline conclusions, although the underlying experimental work is partially salvageable with reanalysis and reframing.

major comments (4)
  1. [Section 3.1, Figure 3] The text states that CE_MEEP and CE_KL 'exhibiting stronger negative correlations (−0.835 and −0.807, respectively) compared to CE (−0.826) and CE_MEALL (−0.861).' This is numerically false: −0.807 is a weaker (less negative) correlation than −0.826, and CE_MEALL actually has the strongest correlation at −0.861. As written, the paper's own numbers contradict the abstract's claim that 'maximum-entropy regularization further strengthens the correlation between uncertainty and segmentation performance.' Only CE_MEEP shows a negligible improvement over CE (−0.835 vs. −0.826), and no confidence intervals or significance tests are provided for any of these differences. The authors must recompute or re-report the correlations, provide separate ID and OOD correlations, and revise the claims accordingly.
  2. [Sections 2.2.2 and 3.1] The MEEP and KL regularizers are defined using ground-truth labels to identify erroneous predictions and explicitly penalize low entropy only on those voxels. Therefore, the observation in Section 3.1 and Figure 4 that erroneous predictions (FP/FN) have higher entropy, and that entropy anti-correlates with Dice, is partly a direct consequence of the training objective rather than an independent property of the resulting models. To support the claimed use of entropy as a proxy for segmentation errors without ground truth at inference, the authors should evaluate error-detection performance on a held-out validation set that was not used for training, or compare against a model trained without label-dependent regularization, and report metrics such as AUROC for flagging erroneous voxels or scans.
  3. [Section 2.4, Section 3.1] The OOD dataset (3D-MR-MS) differs from the WMH Segmentation Challenge dataset not only in scanner and acquisition parameters but also in disease type (multiple sclerosis vs. aging-related vascular disease), patient age (median 39 vs. mean 70), and lesion morphology. Consequently, the observed differences in calibration and uncertainty cannot be attributed specifically to domain shift as defined in the paper; they may reflect disease-related distributional differences. The authors should either use a same-disease scanner-shift evaluation protocol or explicitly acknowledge and discuss this confounding factor as a limitation.
  4. [Section 2.5] The regularization weight λ for each strategy was selected by grid search 'balancing segmentation performance and the quality of uncertainty estimation.' No details are provided about the validation set used for this selection, the exact objective function, or whether the correlations reported in Section 3.1 are computed on the same data used for model selection. This creates a risk of overfitting to the evaluation metric and makes the comparison of correlation coefficients difficult to interpret. The authors should specify the selection criterion and the data split used for choosing λ, and preferably report the sensitivity of the main results to λ.
minor comments (6)
  1. [Section 2.2.1] The expression for L_a contains a double negative that makes the formula ambiguous; it should be written as L_a(Y) = -H_b(Y) = sum over erroneous voxels of [y_i log2(y_i) + (1-y_i)log2(1-y_i)], but as typeset the minus signs are confusing.
  2. [Section 2.2, Eq. (1)] The notation for ground-truth and predicted labels is inconsistent: the text says that Y is the ground truth and \hat{Y} is the predicted mask, but the loss equation is formatted ambiguously and appears to use Y for both. Please use consistent notation throughout.
  3. [Figure 4 caption] The caption refers to 'blue bars' and 'orange bars,' but the figure shows colored points, not bars; please correct the caption to match the plot.
  4. [Section 3.1] The text mentions that linear regression lines are fitted to each set of data points, but no slopes or intercepts are reported, only Pearson correlation coefficients. Either report the regression results or remove the mention of fitted lines.
  5. [Section 3.3, Figure 7] The claim that CE_KL demonstrates superior calibration in the OOD scenario is supported only by ECE values shown in the figure; please report the exact ECE numbers in the text so the reader does not have to infer them from the figure.
  6. [General] There are several typos, including 'the the overall loss function' in Section 2.2.1 and 'condi' in the corresponding author's email address; please proofread the manuscript.

Circularity Check

1 steps flagged · score 6.0 of 10

MEEP/KL correlation claim is partly self-definitional: the loss explicitly maximizes entropy on ground-truth erroneous voxels, so the reported entropy-error correlation is substantially a reflection of the training objective rather than an independent discovery.

  1. self definitional [Section 2.2.2 (MEEP definition) and Section 3.1 (correlation results)]
    "we resort to the maximum entropy on erroneous predictions (MEEP) regularizer, L_m(Y_w), which penalizes low entropy only for erroneous predictions. We will use y_w to define the set of voxels whose label was incorrectly predicted, and hence we can define the regularizer as L_m(Y_w) = - H_b(Y_w). This regularizer will penalize low entropy (i.e. peaky) distributions only when the predictions are wrong."

    The regularizer is defined over Y_w, the set of voxels whose label was incorrectly predicted (identified using ground-truth labels during training), and its objective is to raise entropy precisely at those voxels. Section 3.1 then reports that CE_MEEP and CE_KL 'offer the highest uncertainty for these cases, both in and out of distribution' and that they achieve stronger negative correlations between entropy and Dice than CE. But low-Dice scans contain more erroneously predicted voxels, and the training loss explicitly inflates entropy at erroneous voxels, so the observed correlation is partly a direct readout of the training objective rather than an independent confirmation that uncertainty tracks errors.

full rationale

The paper's general claim that entropy can serve as a proxy for segmentation errors has independent support from the CE baseline, so the paper is not wholly circular. However, the central comparative claim that maximum-entropy regularization (MEEP/KL) strengthens the entropy-Dice correlation is substantially self-definitional: these losses are literally defined as raising the entropy of voxels known (via ground truth) to be misclassified, and the evaluation metric (mean foreground entropy vs Dice) is sensitive to exactly that. Additionally, the paper's own numbers contradict the headline: CE_KL (-0.807) is weaker than CE (-0.826), and CE_MEALL (-0.861) is the strongest, so the abstract's statement that regularization 'further strengthens the correlation' is not supported by the reported coefficients. This is a consistency/correctness concern rather than circularity, but it reinforces that the MEEP/KL advantage is not robustly demonstrated. The grid-search selection of regularization weights 'balancing segmentation performance and the quality of uncertainty estimation' (Section 2.5) also means some of the uncertainty-quality improvements may reflect hyperparameter tuning rather than an intrinsic property of the losses, though details are insufficient to establish statistical forcing. Overall, the derivation is not circular in its ID/OOD comparison or calibration analysis, but the flagship correlation claim partially reduces to the training objective, warranting a score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger is light because the paper proposes no new mathematical entity. The load-bearing inputs are the choice of datasets as ID/OOD, label quality, and the foreground threshold. The regularization weights lambda are fitted to the evaluation metrics through grid search, so they are free parameters rather than derived quantities.

free parameters (4)
  • Regularization weight lambda for CE+MEEP = Not reported
    Selected by grid search per strategy to balance segmentation performance and quality of uncertainty estimation; affects all reported correlation and calibration results.
  • Regularization weight lambda for CE+KL = Not reported
    Selected by grid search per strategy to balance segmentation performance and quality of uncertainty estimation; affects all reported correlation and calibration results.
  • Regularization weight lambda for CE+MEALL = Not reported
    Selected by grid search per strategy to balance segmentation performance and quality of uncertainty estimation; affects all reported correlation and calibration results.
  • Foreground probability threshold = 0.5
    Average entropy is computed only over voxels predicted as foreground with p > 0.5; the choice affects the uncertainty proxy and is not varied.
assumptions (4)
  • domain assumption The WMH Challenge dataset and the 3D-MR-MS dataset constitute a valid in-distribution / out-of-distribution domain-shift pair.
    Section 2.4 defines ID as WMH Challenge and OOD as 3D-MR-MS. If disease mix and preprocessing confound the shift, the conclusions about domain shift generalization weaken.
  • domain assumption Ground-truth WMH segmentations are reliable enough to define erroneous predictions during training and as evaluation reference.
    MEEP and KL regularizers need voxel-level labels to identify wrong predictions. Label noise would make the apparent uncertainty-error link partly an artifact.
  • domain assumption Predictive entropy over foreground voxels is a meaningful measure of clinically relevant uncertainty.
    Section 2.1 and Section 2.3.3 assume mean foreground entropy tracks segmentation quality; other uncertainty types such as epistemic or aleatoric are not modeled.
  • standard math Binary entropy and softmax outputs are standard probabilistic tools.
    Used to define uncertainty and regularizers in Section 2.1 and Section 2.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Reliable WMH Segmentation under Domain Shift: An Application Study using Maximum Entropy Regularization to Improve Uncertainty Estimation." pith.science (2026). https://pith.science/paper/Y7WLZZKA

@misc{pith2026250614497,
  author       = {Pith},
  title        = {Pith review of: Towards Reliable WMH Segmentation under Domain Shift: An Application Study using Maximum Entropy Regularization to Improve Uncertainty Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7WLZZKA}},
  note         = {Machine review of arXiv:2506.14497}
}
read the original abstract

Accurate segmentation of white matter hyperintensities (WMH) is crucial for clinical decision-making, particularly in the context of multiple sclerosis. However, domain shifts, such as variations in MRI machine types or acquisition parameters, pose significant challenges to model calibration and uncertainty estimation. This study investigates the impact of domain shift on WMH segmentation by proposing maximum-entropy regularization techniques to enhance model calibration and uncertainty estimation, with the purpose of identifying errors post-deployment using predictive uncertainty as a proxy measure that does not require ground-truth labels. To do this, we conducted experiments using a U-Net architecture to evaluate these regularization schemes on two publicly available datasets, assessing performance with the Dice coefficient, expected calibration error, and entropy-based uncertainty estimates. Our results show that entropy-based uncertainty estimates can anticipate segmentation errors, and that maximum-entropy regularization further strengthens the correlation between uncertainty and segmentation performance while also improving model calibration under domain shift.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

  1. [1]

    J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y.,

    Cardoso, M. J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., ... & Feng, A. (2022). Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701. Chaves H, Serra MM, Shalom DE et al. (2024) Assessing robustness and generalization of a deep neural network for brain MS lesion segmentation on real-world data. Eur Radi...

    arXiv 2022

  2. [48]

    (2018) A probabilistic U-Net for segmentation of ambiguous images

    Available at: https://proceedings.mlr.press/v48/gal16.html Kohl S, Romera-Paredes B, Meyer C et al. (2018) A probabilistic U-Net for segmentation of ambiguous images. In: Advances in Neural Information Processing Systems, Vol. 31 [dataset] Kuijf HJ, Biesbroek JM, De Bresser J et al. (2019) Standardized assessment of automatic segmentation of white matter ...

    work page arXiv 2018

  3. [2023]

    Springer, Cham

    Lecture Notes in Computer Science, vol 14222. Springer, Cham. 273-283 [dataset] Lesjak Ž, Galimzianova A, Koren A et al. (2018) A novel public MR image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus. Neuroinformatics 16:51-63. DOI: 10.1007/s12021-017-9348-7 Mehrtash A, Wells WM, Tempany CM, Abolmaesumi P, K...

    work page arXiv 2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.