REVIEW 4 major objections 4 minor 39 references
Comparative evaluation of training strategies using partially labelled datasets for segmentation of white matter hyperintensities and stroke lesions in FLAIR MRI
T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Pseudolabelling is the most effective way to train a single model that segments both white matter hyperintensities and ischaemic stroke lesions from partially labelled FLAIR MRI.
desk verdict A useful large-scale comparison of partial-labelling strategies for WMH/ISL segmentation, but the claim that pseudolabels is 'most effective' is not statistically supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing ingredient is the pseudolabel training set. Starting from a multiclass network, the paper trains a 'marginal loss' model that merges missing classes into background, then uses that model to produce pseudolabels for the missing class on every partially labelled volume. These pseudolabels, with available ground truth masks left intact, form a fully labelled multiclass training set on which a standard segmentation network is trained with summed cross-entropy and Dice losses over background/WMH/ISL. This lets the model learn both classes jointly and exploit all volumes, rather than discarding scans that lack one label.
What would settle it
Train the same six strategies on the same partially labelled data, but evaluate on a held-out test set whose ground truth was produced by a single strict FLAIR-only protocol with one annotation policy; if the pseudolabels model no longer ranks first on mean average precision, the paper's central claim is shown to be an artifact of heterogeneous reference standards.
Extended reading notes
Core claim
Across 12 datasets aggregated into 2,052 training volumes (1,341 with WMH labels, 1,152 with ISL labels), the paper trains six partially-supervised methods and evaluates on 1,007 test volumes. The pseudolabels model, which uses a marginal-loss model to fill missing labels, achieves the best mean average precision (65.58% vs 51.20% for the multiclass baseline that sees only fully labelled scans), and also best DSC and AVD, with ISL AP rising from 48.05% to 55.17%. Qualitative analysis indicates the model's WMH segmentations are as plausible as the ground truth in most cases and that it detects the majority of FLAIR-positive ISL; the main failure mode is tumours, where peritumoral oedema is mi
Load-bearing premise
The pooled test set treats ground-truth masks from twelve datasets, produced under different annotation policies (conservative versus liberal WMH delineation, DWI-derived ISL masks), as interchangeable reference standards; if those policies are not equivalent, the ranking of training strategies could reflect label-policy mixture instead of true segmentation quality.
Editorial extensions
If this is right
- Joint WMH/ISL segmentation models can be trained without any fully double-annotated dataset; existing single-label cohorts suffice.
- Pseudolabelling outperforms more complex partial-supervision schemes (marginal loss, class-adaptive loss, phased training) on the primary metric, suggesting simple pipelines are hard to beat.
- ISL segmentation on FLAIR improves substantially when DWI-derived labels are filtered for FLAIR visibility and pseudolabels fill the missing WMH channel.
- The consistent WMH policy learned across datasets means the model can act as a harmonisation reference for comparing datasets with different annotation styles.
- The study's scale (2,052 volumes, 1,007 test volumes) gives a more reliable comparison than typical small-cohort evaluations.
Reading between the lines
- Because the pseudolabel advantage is largely in ISL, the method's benefit may depend on the relative difficulty and label completeness of each class; a task with balanced label availability may not show the same gap.
- The hand-chosen FLAIR-visibility thresholds used to filter ISL cases are a sensitive choice; a learnable or automated version could alter which cases enter training and possibly change the ranking.
- A natural testbed extension is to model dataset-specific annotation policies explicitly (e.g., per-protocol output heads or policy-conditioned losses), which might push absolute performance beyond the pseudolabelling ceiling.
- The observed failure on tumour cases suggests pseudolabelling amplifies the model's tendency to treat any hyperintensity as a lesion; a small amount of labelled tumour cases could plug this gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates six practical strategies for training a joint WMH/ISL segmentation model from partially labelled FLAIR data, aggregating 12 datasets (2052 volumes, 1007 test volumes). The strategies are multiclass baseline (trained on the fully labelled subset), multi-model binary with/without temperature scaling, class-conditional, pseudolabels (teacher = marginal loss), phased, class-adaptive loss, and marginal loss. The primary metric is average precision (AP), chosen as operating-point independent; DSC, AVD, ASD, lesion-level metrics and qualitative analysis are also reported. The paper claims that pseudolabelling is the most effective strategy, with a mean AP of 65.58% vs 65.13% for class-adaptive loss and 64.72% for marginal loss, and that partially labelled data improve over the fully-labelled baseline for most methods.
Significance. If the ranking were robust, this would be a valuable practical benchmark: the study is one of the largest of its kind for WMH/ISL segmentation, uses a sensible primary metric (AP), aggregates a heterogeneous cohort, releases code, and includes qualitative validation that is often missing in such comparisons. The careful discussion of annotation-policy variability and of the FLAIR-only evaluation setting is a strength. However, the central claim that pseudolabelling is 'most effective' is currently carried by sub-1pp point estimates with no uncertainty quantification, and the comparison is entangled with the choice of pseudolabel teacher. The paper's contribution is therefore useful but its headline conclusion needs additional statistical support before it can be accepted.
major comments (4)
- [§4.3, Table 4] The central claim that pseudolabelling is the most effective strategy is not statistically supported. On the primary metric AP, pseudolabels achieve 65.58% versus 65.13% for class-adaptive and 64.72% for marginal loss; on WMH specifically, pseudolabels (76.00%) are lower than both class-adaptive (76.62%) and marginal (76.67%). Three seeds were averaged but no per-seed spread, confidence intervals, or paired tests over the 1007 test volumes are reported. The boxplots in Fig. 3 show substantial per-dataset overlap. The abstract's 'most effective' overstates the cautious 'suggest' in §4.3. Please add uncertainty quantification (e.g., bootstrap over subjects/datasets, per-seed results) and/or per-dataset rankings to determine whether the observed ordering is distinguishable from noise.
- [§3.2, pseudolabels] The pseudolabel strategy uses the marginal loss model—itself one of the compared baselines—as the teacher to generate labels for missing classes. This is not a logical circularity, but it means the comparison 'pseudolabels vs marginal loss' is not an evaluation of pseudolabelling in general but of a specific coupling to one of the competitors. The reported advantage over marginal loss could be sensitive to the teacher choice. Please either report results with an alternative teacher (e.g., multiclass baseline or class-adaptive) or, at minimum, qualify the abstract/conclusion as 'pseudolabelling with a marginal-loss teacher' and discuss this limitation explicitly.
- [§4.4 vs Table 4] The pooled test set mixes datasets with materially different annotation policies (e.g., conservative vs liberal WMH delineation, DWI-derived ISL masks). The paper acknowledges this in §4.4, but the aggregate numerical ranking in Table 4 does not stratify by policy. Since the differences between the top methods are under 1pp on the primary metric, the aggregate ranking could reflect the particular mixture of label policies rather than true segmentation quality. Please report per-dataset AP for all six methods (or at least stratify by policy groups) and demonstrate that the ranking is consistent across datasets.
- [§2.2] The FLAIR-visibility discard thresholds for ISLES/SOOP (mean intensity difference <0.05/0.1; voxel fraction >20%/10%) are hand-chosen. Because the same filter applies to all compared methods, it does not invalidate the relative comparison, but the absolute ISL results and the composition of the evaluation set depend on these choices. Please justify the thresholds or provide a sensitivity analysis, and report the number of scans discarded per dataset.
minor comments (4)
- [Table 1] The table formatting is difficult to read; the sub-columns for N, WMH, and ISL are not clearly separated, making it hard to verify split totals (e.g., MSS1 and MSS2 rows). Please reformat for clarity.
- [§4.2] The qualitative discussion of BRATS false positives is valuable, but it is a limitation that this behaviour is not reflected in the quantitative metrics. Consider making this limitation more prominent in the Discussion rather than only in §4.2.
- [§2.1.12] Typo: 'Edniburgh' should be 'Edinburgh' in §2.1.3.
- [§3.3.2] The DSCθ metric is introduced with θ=2mm but no reference to the original source for the choice of θ is given beyond [37]. A brief justification of θ=2mm would help.
Circularity Check
No circularity found: empirical head-to-head comparison with no definitional reduction; ranking lacks CIs (robustness, not circularity).
full rationale
This paper is an empirical benchmark comparing six training strategies on a fixed, independently labelled test set. It makes no analytic derivation in which an output is equal to an input by construction. The only candidate concern is the pseudolabel design: the missing labels are generated by the marginal loss model, which is itself one of the compared baselines. This is not circular in the required sense. The teacher (marginal loss model) is trained independently on ground-truth partial labels, and the pseudolabel model is then trained on the augmented set; the pseudolabel model's predictions are not the teacher's outputs by definition, and the comparison between pseudolabel and marginal loss is an empirical result, not a logical consequence of the construction. The paper's central claim—that pseudolabelling is most effective—rests on point estimates in Table 4 without confidence intervals or significance tests, and the margins to class-adaptive loss are small; however, this is a statistical robustness limitation, not circularity. No load-bearing argument relies on a self-citation: the cited related methods ([10], [36], etc.) are external sources, and the paper does not invoke any uniqueness theorem or prior result by the same authors to force its conclusions. There is no fitted parameter relabelled as a prediction, and no known result is merely renamed. Therefore the derivation chain is self-contained with respect to the circularity criteria, and the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- FLAIR-visibility discard thresholds for ISLES/SOOP =
ISLES: mean intensity diff <0.05, voxel fraction >20%; SOOP: <0.1, >10%
- Pseudolabel teacher selection =
marginal loss model
- DSC_theta tolerance =
2 mm
assumptions (4)
- domain assumption Rigid co-registration with FSL-FLIRT is sufficient to align label maps to FLAIR images.
- domain assumption SynthStrip and SynthSeg provide accurate enough brain extraction and anatomical segmentation for the FLAIR-visibility filter.
- domain assumption WMH and ISL are mutually exclusive at the voxel level and can be represented with a single softmax output.
- domain assumption Ground-truth labels from different datasets, despite inter-rater variability, are usable as equivalent reference standards for aggregate metric comparison.
Cite this review
Pith. "Pith review of Comparative evaluation of training strategies using partially labelled datasets for segmentation of white matter hyperintensities and stroke lesions in FLAIR MRI." pith.science (2026). https://pith.science/paper/HIWLTN6V
@misc{pith2026260120503,
author = {Pith},
title = {Pith review of: Comparative evaluation of training strategies using partially labelled datasets for segmentation of white matter hyperintensities and stroke lesions in FLAIR MRI},
year = {2026},
howpublished = {\url{https://pith.science/paper/HIWLTN6V}},
note = {Machine review of arXiv:2601.20503}
}
read the original abstract
White matter hyperintensities (WMH) and ischaemic stroke lesions (ISL) are key imaging biomarkers of cerebral small vessel disease (SVD) detectable on magnetic resonance imaging (MRI). The development of robust deep learning models to automatically segment and differentiate these pathologies remains challenging. Specifically, WMH and ISL frequently co-occur within the same subject and present as visually confounding hyperintensities on fluid-attenuated inversion recovery (FLAIR) sequences, complicating their accurate delineation. To address the scarcity of fully annotated cohorts, we systematically evaluated six accessible strategies for training a joint WMH and ISL segmentation model using partially labelled data. We aggregated privately held and publicly available datasets to curate a large-scale cohort of 2,052 MRI volumes, of which 1341 and 1152 volumes contained ground truth annotations for WMH and ISL, respectively. Our analysis indicates that multiple strategies effectively leverage partially labelled data to enhance overall model performance, with pseudolabelling emerging as the most effective approach. This model exhibited a consistent WMH segmentation policy and successfully detected the majority of FLAIR-positive ISL. These findings demonstrate the viability of using partially labelled data to develop reliable automated segmentation tools, which can support ongoing SVD monitoring and high-throughput biomarker extraction for large-scale clinical research.
Figures
Reference graph
Works this paper leans on
-
[1]
Debette, S
S. Debette, S. Schilling, M.-G. Duperron, S. C. Larsson, H. S. Markus, Clinical significance of magnetic resonance imaging markers of vascular brain injury: a systematic review and meta-analysis, JAMA neurology 76 (1) (2019) 81–94
2019
-
[2]
Dmitriev, A
K. Dmitriev, A. E. Kaufman, Learning multi-class segmentations from single-class datasets, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9501–9511
2019
-
[3]
Zhang, Y
J. Zhang, Y. Xie, Y. Xia, C. Shen, Dodnet: Learning to segment multi- organ and tumors from multiple partially labeled datasets, in: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 1195–1204
2021
-
[4]
R. Deng, Q. Liu, C. Cui, Z. Asad, Y. Huo, et al., Single dynamic network for multi-label renal pathology image segmentation, in: International Conference on Medical Imaging with Deep Learning, PMLR, 2022, pp. 304–314
2022
-
[5]
X. Fang, P. Yan, Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction, IEEE Transactions on Medical Imaging 39 (11) (2020) 3619–3629
2020
-
[6]
G. Shi, L. Xiao, Y. Chen, S. K. Zhou, Marginal lossand exclusion lossfor partially supervised multi-organ segmentation, Medical Image Analysis 70 (2021) 101979
2021
-
[7]
Fidon, M
L. Fidon, M. Aertsen, D. Emam, N. Mufti, F. Guffens, T. Deprest, P. Demaerel, A. L. David, A. Melbourne, S. Ourselin, et al., Label- set loss functions for partial supervision: application to fetal brain 3d mri parcellation, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, Septe...
2021
-
[8]
Y. Zhou, Z. Li, S. Bai, C. Wang, X. Chen, M. Han, E. Fishman, A. L. Yuille, Prior-aware neural network for partially-supervised multi-organ segmentation, in: Proceedings of the IEEE/CVF international confer- ence on computer vision, 2019, pp. 10672–10681. 25
2019
Show all 39 references
-
[9]
Q.Liu, H.Zeng, Z.Sun, X.Li, G.Zhao, Y.Liang, Manybirds, onestone: Medical image segmentation with multiple partially labelled datasets, Pattern Recognition (2024) 110636
2024
-
[10]
S. N. Llambias, M. Nielsen, M. M. Ghazi, Heterogeneous learning for brain lesion segmentation, detection, and classification, in: Northern Lights Deep Learning Conference 2024, 2024
2024
-
[11]
H. J. Kuijf, J. M. Biesbroek, J. De Bresser, R. Heinen, S. Andermatt, M. Bento, M. Berseth, M. Belyaev, M. J. Cardoso, A. Casamitjana, et al., Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge, IEEE...
2019
-
[12]
M. R. Hernandez Petzsche, E. de la Rosa, U. Hanning, R. Wiest, W. Valenzuela, M. Reyes, M. Meyer, S.-L. Liew, F. Kofler, I. Ezhov, et al., Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset, Scientific data 9 (1) (2022) 762
2022
-
[13]
Commowick, M
O. Commowick, M. Kain, R. Casey, R. Ameli, J.-C. Ferré, A. Kerbrat, T. Tourdias, F. Cervenansky, S. Camarasu-Pop, T. Glatard, et al., Mul- tiple sclerosis lesions segmentation from multiple experts: The miccai 2016 challenge dataset, Neuroimage 244 (2021) 118589
2016
-
[14]
Styner, J
M. Styner, J. Lee, B. Chin, M. Chin, O. Commowick, H. Tran, S. Markovic-Plese, V. Jewells, S. Warfield, 3d segmentation in the clinic: A grand challenge ii: Ms lesion segmentation, MIDAS journal 2008 (2008) 1–6
2008
-
[15]
J. M. Wardlaw, F. Doubal, P. Armitage, F. Chappell, T. Carpenter, S. Muñoz Maniega, A. Farrall, C. Sudlow, M. Dennis, B. Dhillon, La- cunar stroke is associated with diffuse blood–brain barrier dysfunction, Annals of Neurology: Official Journal of the American Neurological As-...
2009
-
[16]
J. M. Wardlaw, S. J. Makin, M. C. V. Hernández, P. A. Armitage, A. K. Heye, F. M. Chappell, S. Munoz-Maniega, E. Sakka, K. Shuler, M. S. Dennis, et al., Blood-brain barrier failure as a core mechanism in cerebral small vessel disease and dementia: evidence from a cohort study,...
2017
-
[17]
Clancy, D
U. Clancy, D. J. Garcia, M. S. Stringer, M. J. Thrippleton, M. C. Valdés- Hernández, S. Wiseman, O. K. Hamilton, F. M. Chappell, R. Brown, G. W. Blair, et al., Rationale and design of a longitudinal study of cerebral small vessel diseases, clinical and imaging outcomes in pati...
2021
-
[18]
J. M. Wardlaw, M. E. Bastin, M. C. Valdés Hernández, S. M. Man- iega, N. A. Royle, Z. Morris, J. D. Clayden, E. M. Sandeman, E. Eadie, C. Murray, et al., Brain aging, cognition in youth and old age and vas- cular disease in the lothian birth cohort 1936: rationale, design and ...
1936
-
[19]
J. M. Wardlaw, E. E. Smith, G. J. Biessels, C. Cordonnier, F. Fazekas, R. Frayne, R. I. Lindley, J. T O’Brien, F. Barkhof, O. R. Benavente, et al., Neuroimaging standards for research into small vessel disease and its contribution to ageing and neurodegeneration, The Lancet Ne...
2013
-
[20]
B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al., The multi- modal brain tumor image segmentation benchmark (brats), IEEE trans- actions on medical imaging 34 (10) (2014) 1993–2024
2014
-
[21]
Ronneberger, P
O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international con- ference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, Spri...
2015
-
[22]
Absher, S
J. Absher, S. Goncher, R. Newman-Norlund, N. Perkins, G. Yourganov, J. Vargas, S. Sivakumar, N. Parti, S. Sternberg, A. Teghipco, et al., The stroke outcome optimization project: Acute ischemic strokes from a comprehensive stroke center, Scientific Data 11 (1) (2024) 839
2024
-
[23]
J. M. Wardlaw, K. W. Muir, M.-J. Macleod, C. Weir, F. McVerry, T. Carpenter, K. Shuler, R. Thomas, P. Acheampong, K. Dani, et al., Clinical relevance and practical implications of trials of perfusion and 27 angiographic imaging in patients with acute ischaemic stroke: a mul- t...
2013
-
[24]
M. del C. Valdés Hernández, L. C. Maconick, S. Muñoz Maniega, X. Wang, S. Wiseman, P. A. Armitage, F. N. Doubal, S. Makin, C. L. Sudlow, M. S. Dennis, et al., A comparison of location of acute symp- tomatic vs.‘silent’small vessel lesions, International Journal of Stroke 10 (7...
2015
-
[25]
M. A. Rodrigues, N. Samarasekera, C. Lerpiniere, C. Humphreys, M. O. McCarron, P. M. White, J. A. Nicoll, C. L. Sudlow, C. Cordonnier, J. M. Wardlaw, et al., The edinburgh ct and genetic diagnostic criteria for lobar intracerebral haemorrhage associated with cerebral amyloid a...
2018
-
[26]
Jenkinson, S
M. Jenkinson, S. Smith, A global optimisation method for robust affine registration of brain images, Med. Image Anal. 5 (2) (2001) 143–156
2001
-
[27]
Jenkinson, P
M. Jenkinson, P. Bannister, M. Brady, S. Smith, Improved optimization for the robust and accurate linear registration and motion correction of brain images, Neuroimage 17 (2) (2002) 825–841
2002
-
[28]
N. J. Tustison, B. B. Avants, P. A. Cook, Y. Zheng, A. Egan, P. A. Yushkevich, J. C. Gee, N4itk: improved n3 bias correction, IEEE trans- actions on medical imaging 29 (6) (2010) 1310–1320
2010
-
[29]
Hoopes, J
A. Hoopes, J. S. Mora, A. V. Dalca, B. Fischl, M. Hoffmann, Synthstrip: skull-stripping for any brain image, NeuroImage 260 (2022) 119474
2022
-
[30]
Billot, D
B. Billot, D. N. Greve, O. Puonti, A. Thielscher, K. Van Leemput, B. Fischl, A. V. Dalca, J. E. Iglesias, et al., Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining, Medical image analysis 86 (2023) 102789
2023
-
[31]
M. J. Cardoso, W. Li, R. Brown, N. Ma, E. Kerfoot, Y. Wang, B. Murrey, A. Myronenko, C. Zhao, D. Yang, et al., Monai: An open-source framework for deep learning in healthcare, arXiv preprint arXiv:2211.02701 (2022). 28
2022 arXiv
-
[32]
William, The Pytorch Lightning team, Pytorch lightning (2019)
F. William, The Pytorch Lightning team, Pytorch lightning (2019). doi:10.5281/zenodo.3828935
2019 doi
-
[33]
Pérez-García, R
F. Pérez-García, R. Sparks, S. Ourselin, Torchio: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning, Computer Methods and Programs in Biomedicine 208 (2021) 106236
2021
-
[34]
Isensee, P
F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, K. H. Maier-Hein, nnu- net: a self-configuring method for deep learning-based biomedical image segmentation, Nature methods 18 (2) (2021) 203–211
2021
-
[35]
C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On calibration of mod- ern neural networks, in: International conference on machine learning, PMLR, 2017, pp. 1321–1330
2017
-
[36]
Y. He, P. Guo, Y. Tang, A. Myronenko, V. Nath, Z. Xu, D. Yang, C. Zhao, B. Simon, M. Belue, et al., Vista3d: Versatile imaging seg- mentation and annotation model for 3d computed tomography, arXiv preprint arXiv:2406.05285 (2024)
2024 arXiv
-
[37]
Strumia, F
M. Strumia, F. R. Schmidt, C. Anastasopoulos, C. Granziera, G. Krueger, T. Brox, White matter ms-lesion segmentation using a geo- metric brain model, IEEE transactions on medical imaging 35 (7) (2016) 1636–1646
2016
-
[38]
Sakka, U
B.Philps, M.delC.ValdesHernandez, S.MunozManiega, M.E.Bastin, E. Sakka, U. Clancy, J. M. Wardlaw, M. O. Bernabeu, Stochastic uncer- tainty quantification techniques fail to account for inter-analyst variabil- ity in white matter hyperintensity segmentation, in: Annual Conferen...
2024
-
[39]
Isensee, T
F. Isensee, T. Wald, C. Ulrich, M. Baumgartner, S. Roy, K. Maier-Hein, P. F. Jaeger, nnu-net revisited: A call for rigorous validation in 3d medical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2024, ...
2024
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.