How We Won the ISLES'24 Challenge by Preprocessing

Hitender Oswal; Juampablo E. Heras Rivera; Mehmet Kurt; Sophie Walters; Tianyi Ren; William Henry; Yutong Pan

REVIEW 4 major objections 4 minor 1 cited by

How We Won the ISLES'24 Challenge by Preprocessing

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that skull stripping and clinically informed intensity windowing, applied before a standard nnU-Net, were the decisive factor in winning the ISLES'24 stroke lesion segmentation challenge.

desk verdict A genuine challenge win and a plausible preprocessing recipe, but the 10-point Dice gain rests on a single selected fold, so the central claim is not yet established. read the letter →

arxiv 2505.18424 v2 pith:GRJYKCMY submitted 2025-05-23 eess.IV cs.AIcs.CV

Tianyi Ren , Juampablo E. Heras Rivera , Hitender Oswal , Yutong Pan , William Henry , Sophie Walters , Mehmet Kurt This is my paper · ORCID

classification eess.IVcs.AIcs.CV

keywords ischemicstrokesegmentationCTperfusionimagingintensitywindowingskullstrippingnnU-NetISLES'24challengepreprocessing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the decisive factor in winning the ISLES'24 stroke lesion segmentation challenge was preprocessing, not the segmentation architecture. The authors pair a standard large residual nnU-Net with deep-learning-based skull stripping and custom intensity windows for each CT-derived modality. In their best-performing cross-validation fold, custom windowing raised the Dice score from 21.8% to 31.0%, and adding Z-score normalization reached 31.8%. On the challenge test set, their pipeline achieved a mean Dice of 28.5 with a standard deviation of 21.27, placing first. If correct, the work shows that clinically motivated image preprocessing can matter as much as model design for CT-based stroke imaging.

What carries the argument

The central mechanism is intensity windowing: each CT-derived input modality (CTA, CBF, CBV, MTT, Tmax) is clipped to a clinically motivated range—for example, CBF to (0, 35) mL/100g/min, MTT to (0, 20) s, and Tmax to (0, 7) s—so that ischemic tissue occupies a larger fraction of the network input range. This is combined with SynthStrip, a deep-learning brain-extraction tool, which removes the skull and background from the non-contrast CT and transfers the resulting mask to co-registered scans. Together these steps provide the segmentation network with a cleaner learning signal than a single global percentile window.

What would settle it

A reader could rerun the authors' 10-fold cross-validation and compare the mean validation Dice across all folds for the standard pipeline versus the custom-window pipeline; if the average improvement is near zero, the central claim that preprocessing was the key to winning would lose its support.

Watch

Extended reading notes

Core claim

The central claim is that standard preprocessing pipelines, such as Z-score normalization with [1,99] percentile windowing, are insufficient for ischemic stroke lesion segmentation on CT, and that a deliberately tailored preprocessing pipeline—SynthStrip skull stripping plus per-modality intensity windowing—makes a standard nnU-Net competitive enough to win the ISLES'24 challenge. On the best validation fold, custom windowing alone improved Dice from 21.8% to 31.0%, and custom windowing combined with Z-score normalization reached 31.8%; the final test submission achieved a mean Dice of 28.5 with a standard deviation of 21.27, the highest mean among the top three leaderboard entries.

Load-bearing premise

The paper's headline 10-point improvement is measured on the single best of ten cross-validation folds, and the argument assumes that this one fold reflects the average benefit of custom windowing.

Editorial extensions

If this is right

Standard nnU-Net preprocessing is insufficient for CT-based ischemic stroke segmentation, and custom windowing gives a 10-point validation Dice improvement on the selected fold.
Each CT-derived modality benefits from its own clipping range rather than a single global window.
A standard large residual nnU-Net, without architectural novelty, can win a segmentation challenge when the input preprocessing is tailored.
Predicting final infarct from admission CT remains high-variance: the winning test Dice is 28.5 with a standard deviation of 21.27, so robustness is an open problem.
Future improvements may come from incorporating clinical priors beyond windowing into the preprocessing pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Averaged over all ten cross-validation folds rather than the single best fold, the windowing benefit is likely smaller than 10 Dice points; the large test standard deviation suggests the effect is patient-dependent.
The same recipe—modality-specific windowing plus skull stripping on a standard U-Net—could transfer to other CT-based lesion segmentation tasks where the signal is subtle, such as hemorrhage or traumatic brain injury.
Because windowing discards skull and background intensities, it may also reduce sensitivity to scanner and protocol variation; this is a testable hypothesis the paper does not directly demonstrate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a legitimately winning challenge entry, and the paper is a short, honest write-up of the preprocessing choices. The specific windowing bounds and the SynthStrip-on-CT step are useful practical details. But the paper's central claim—that custom preprocessing was decisive—is supported only by an ablation from the single best of 10 folds. That's a selected maximum, not an estimate of the average effect, and the test result (Dice 28.5 ± 21.27) is actually lower than the selected fold's 31.8, with enormous variance. So the 10-point improvement could easily be fold-selection luck.

What the paper does well: it is candid about the high variance and calls for future robustness work; it reports specific window ranges so others can reproduce the recipe; and the use of SynthStrip on NCCT and transferring the mask to co-registered modalities is a sensible, shareable trick. The writing is clear and appropriately modest in scope—this is a challenge report, not a general method claim.

The soft spots are real but proportionate to the paper's ambitions. First, only the best fold's ablation is shown; no mean or standard deviation across folds, and no significance test. Without that, the 31.8 vs 21.8 comparison is anecdotal. Second, there is an internal inconsistency: the text says the final submission used Z-score normalization, but Table 2 labels that row 'Windowing + Histogram Equalization'. That must be resolved. Third, skull stripping is a named component of the pipeline but is never ablated, so its contribution is unknown. Fourth, no code or detailed pipeline is provided, which limits reproducibility, though the window ranges are at least listed.

The circularity burden is low—the windowing parameters were tuned on validation and then tested on the challenge test set, which is normal challenge practice. The weakness is purely statistical reporting.

Who gets value from this: anyone preparing for a medical imaging challenge, and people working on CT-based stroke lesion segmentation. It's a useful data point, not a methodological breakthrough. I'd send it to review: a serious referee could ask for the full fold distribution, the table fix, and ideally a mask-off ablation. With those, the paper would be solid. As is, it's a reasonable short paper whose main claim needs strengthening.

My recommendation: engage with it, but read the numbers with a grain of salt.

Referee Report

4 major / 4 minor

Summary. The paper reports the preprocessing pipeline used by the authors to win the ISLES'24 stroke lesion segmentation challenge. The proposed method applies deep-learning-based skull stripping (SynthStrip) and custom intensity windowing for each CT-derived modality (CTA, CBF, CBV, MTT, Tmax), followed by a standard residual encoder nnU-Net. The authors report a validation Dice of 31.8% on the best-performing fold of a 10-fold cross-validation, compared with 21.8% for standard Z-score normalization with [1,99] percentile windowing, and a test Dice of 28.5% (std 21.27) that ranked first on the challenge leaderboard. The central claim is that carefully designed preprocessing, especially custom windowing, was key to the winning performance.

Significance. If the central claim is established, the paper provides a simple, reproducible, and potentially transferable preprocessing recipe for CT-only stroke lesion segmentation, a task with low intrinsic contrast. The paper is clearly written and the pipeline is described with sufficient detail to be reimplemented, including the exact window bounds in Table 1. The use of a standard nnU-Net baseline isolates the effect of preprocessing, which is a strength of the experimental design. The principal weakness is that the key ablation evidence is reported only for a single selected fold, not for the full cross-validation distribution, and there is an internal inconsistency about the exact final preprocessing configuration. The significance of the work therefore depends on whether the authors can provide stronger evidence that the preprocessing benefit is consistent across folds and not an artifact of fold selection or validation-set overfitting.

major comments (4)

[§4, Table 2, §6] The central claim that custom windowing improves Dice by 10 percentage points rests on the single best-performing fold out of 10-fold cross-validation. Section 4 states that 'the fold with the highest validation Dice score was selected for submission,' and Section 6 repeats the 'best-performing fold' comparison. Reporting the selected maximum rather than the mean and standard deviation across folds makes the 10-point improvement uninterpretable as an estimate of the average preprocessing benefit. The test-set mean (28.5, SD 21.27, Table 3) is lower than the selected-fold value (31.8) and has a very large spread, so the reader cannot rule out that the reported improvement is within fold-to-fold variability. Please report the full 10-fold results (per-fold and mean ± SD) for each preprocessing variant, and provide a paired significance test across folds (e.g., paired t-test or Wilcoxon signed-rank test) to support the claim.
[§4 vs. Table 2] There is an internal inconsistency about the final preprocessing configuration. The text of Section 4 says that 'our submission combining custom windowing with Z-score normalization further increased the validation Dice score to 31.8%,' but Table 2 labels the 0.318 row as 'Windowing + Histogram Equalization.' These two descriptions are not equivalent, and they imply different final pipelines. This discrepancy must be resolved, because the exact contribution of custom windowing cannot be assessed if the baseline and final configurations are not clearly and consistently specified.
[§3.2, §4, Abstract] The abstract and conclusion credit a preprocessing pipeline that includes deep-learning-based skull stripping, but the ablation in Section 4 does not include a no-skull-stripping condition. Since SynthStrip is presented as a substantive component of the method, the paper should report at least one experiment without skull stripping (or with a simpler mask) to establish that this step contributes to the reported Dice improvements. Without this ablation, the claim that 'the preprocessing pipeline including deep-learning-based skull stripping' is beneficial is not directly supported by the presented evidence.
[§3.3, §4] The windowing parameters are described as 'empirically adjusted to optimize model performance' (Section 3.3). The reported improvement is measured on validation data that were likely used for these adjustments, which introduces a risk of overfitting to the validation set. Please clarify whether the window bounds in Table 1 were fixed before the 10-fold cross-validation was run or tuned using the same validation fold that was later selected for the head-to-head comparison. If the latter, the reported 10-point gain should be interpreted as a selected and potentially optimistic estimate, and the paper should state this limitation explicitly.

minor comments (4)

[§1] There is a typo in the introduction: 'low senstivity' should be 'low sensitivity.'
[§3.2] The sentence 'we applied SynthStrip on the non-contrast CT (NCCT) scans to obtain a brain masks' has a subject-verb agreement error and should read 'to obtain a brain mask' or 'to obtain brain masks.'
[§6] The phrase 'improves CT-based segmentation by 10 Dice percentage points' could be misinterpreted because Dice is a ratio, not a percentage in the strict sense; '10 percentage points of Dice' would be clearer.
[Table 3] The table note 'Bold= best, blue = second best' references color coding that is not visible in a monochrome print version; consider using symbols or underlining to distinguish the second-best entry.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the preprocessing claim is an empirical comparison, and self-citations are not load-bearing.

full rationale

The paper's central claim that tailored preprocessing improves stroke lesion segmentation is supported by an ablation on the best of 10 validation folds (Section 4) and by an external test-set leaderboard result (Section 5). The windowing bounds in Section 3.3 are described as the result of 'empirical adjustments made to optimize model performance,' but the reported Dice improvement is a measured comparison on a validation fold, not a quantity derived from those bounds by construction. The risk that the 10-point improvement is inflated by fold selection or by tuning on validation is a statistical validity concern rather than circularity of the kind defined here. The paper itself acknowledges large test variance in Section 6, which further supports treating that as a robustness issue, not a definitional equivalence. The only self-citations (Ren et al., 2024; Heras Rivera et al., 2024) appear in passing statements about CNN architectures and are not load-bearing for the preprocessing claim. No equations, definitions, or uniqueness theorems are used to make the result equivalent to its inputs. Accordingly, no circular step is identified.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on five empirically tuned window bounds, on the unverified accuracy of SynthStrip masks across modalities, and on the assumption that a single best validation fold represents the average effect. No new entities are introduced.

free parameters (5)

CTA window = (0, 90)
Chosen empirically to enhance contrast, following methodology in Pulli et al., with manual adjustments; Table 1, Section 3.3.
CBF window = (0, 35)
Based on clinical threshold CBF<17, widened to 35 empirically; Table 1.
CBV window = (0, 10)
Based on clinical threshold CBV>2, widened to 10 empirically; Table 1.
MTT window = (0, 20)
Based on MTT>145% of contralateral, adjusted to 0-20; Table 1.
Tmax window = (0, 7)
Based on Tmax>6 s threshold, widened to 7; Table 1.

assumptions (3)

domain assumption SynthStrip accurately removes skull from NCCT and the resulting mask is valid for other co-registered modalities.
Section 3.2: the brain mask from NCCT is applied to CTP, CTA, etc. No validation of mask accuracy on these modalities is provided.
domain assumption Ground truth from follow-up MRI is an appropriate target for predicting the final infarct from admission CT.
The challenge premise; the paper does not discuss or justify this mapping, but relies on it for all evaluation.
ad hoc to paper The best fold's validation Dice is a reliable indicator of model and preprocessing quality.
Section 4 uses only the best fold for ablation and model selection, without reporting fold variance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How We Won the ISLES'24 Challenge by Preprocessing." pith.science (2026). https://pith.science/paper/GRJYKCMY

@misc{pith2026250518424,
  author       = {Pith},
  title        = {Pith review of: How We Won the ISLES'24 Challenge by Preprocessing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GRJYKCMY}},
  note         = {Machine review of arXiv:2505.18424}
}

read the original abstract

Stroke is among the top three causes of death worldwide, and accurate identification of stroke lesion boundaries is critical for diagnosis and treatment. Supervised deep learning methods have emerged as the leading solution for stroke lesion segmentation but require large, diverse, and annotated datasets. The ISLES'24 challenge addresses this need by providing longitudinal stroke imaging data, including CT scans taken on arrival to the hospital and follow-up MRI taken 2-9 days from initial arrival, with annotations derived from follow-up MRI. Importantly, models submitted to the ISLES'24 challenge are evaluated using only CT inputs, requiring prediction of lesion progression that may not be visible in CT scans for segmentation. Our winning solution shows that a carefully designed preprocessing pipeline including deep-learning-based skull stripping and custom intensity windowing is beneficial for accurate segmentation. Combined with a standard large residual nnU-Net architecture for segmentation, this approach achieves a mean test Dice of 28.5 with a standard deviation of 21.27.

Figures

Figures reproduced from arXiv: 2505.18424 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Clinically-Informed Preprocessing Improves Stroke Segmentation in Low-Resource Settings
eess.IV 2025-08 unverdicted novelty 5.0 of 10

A clinically-informed preprocessing pipeline improves CT-based ischemic stroke lesion segmentation by 38% Dice over baseline nnU-Net, and further by 21% with CTA vessel maps.

Reference graph

Works this paper leans on

11 extracted references · 1 canonical work pages · cited by 1 Pith paper

[3]

5 Ren Rivera Oswal Pan Henry W alters Kurt Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu

URLhttps://arxiv.org/abs/2405.18368. 5 Ren Rivera Oswal Pan Henry W alters Kurt Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. InInternational MICCAI Brainlesion Workshop, pages 272–284. Springer,

arXiv
[6]

URL https://doi.org/10.1038/s41592-020-01008-z

doi: 10.1038/s41592-020-01008-z. URL https://doi.org/10.1038/s41592-020-01008-z. Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. InInternational Conference on Medical Image Computing and Computer-Assist...

doi:10.1038/s41592-020-01008-z
[7]

Neuroimaging modalities used for ischemic stroke diagnosis and monitor- ing.Medicina, 59(11):1908,

Jasmin J Nukovic, Valentina Opancina, Elisa Ciceri, Mario Muto, Nebojsa Zdravkovic, Ah- met Altin, Pelin Altaysoy, Rebeka Kastelic, Diana Maria Velazquez Mendivil, Jusuf A Nukovic, et al. Neuroimaging modalities used for ischemic stroke diagnosis and monitor- ing.Medicina, 59(11):1908,

work page 1908
[11]

URLhttps://www

doi: 10.1161/01.STR.0000209238.61459.39. URLhttps://www. ahajournals.org/doi/abs/10.1161/01.STR.0000209238.61459.39. M. Yousufuddin and N. Young. Aging and ischemic stroke.Aging (Albany NY), 11(9): 2542–2544, May

arXiv
[2006]

29609.22

doi: 10.1161/01.STR.0000217418. 29609.22. URLhttps://www.ahajournals.org/doi/abs/10.STR.0000217418.29609

arXiv
[2009]

Epub 2008 Dec

doi: 10.1161/STROKEAHA.108.526954. Epub 2008 Dec

doi:10.1161/strokeaha.108.526954 2008
[2015]

6 How We Won the ISLES’24 Challenge by Preprocessing Max Wintermark, Adam E

URLhttp://arxiv.org/ abs/1505.04597. 6 How We Won the ISLES’24 Challenge by Preprocessing Max Wintermark, Adam E. Flanders, Birgitta Velthuis, Reto Meuli, Maarten van Leeuwen, Dorit Goldsher, Carissa Pineda, Joaquin Serena, Irene van der Schaaf, Annet Waai- jer, James Anderson, Gary Nesbit, Igal Gabriely, Victoria Medina, Ana Quiles, Scott Pohlman, Marcel...

arXiv
[2019]

URLhttps://doi.org/10.18632/ aging.101931

doi: 10.18632/aging.101931. URLhttps://doi.org/10.18632/ aging.101931. 7

doi:10.18632/aging.101931

Show all 11 references

[2021]

Heras Rivera, Agamdeep S Chopra, Tianyi Ren, Hitender Oswal, Yutong Pan, Zineb Sordo, Sophie Walters, William Henry, Hooman Mohammadi, Riley Olson, et al

Juampablo E. Heras Rivera, Agamdeep S Chopra, Tianyi Ren, Hitender Oswal, Yutong Pan, Zineb Sordo, Sophie Walters, William Henry, Hooman Mohammadi, Riley Olson, et al. An ensemble approach for brain tumor segmentation and synthesis.arXiv preprint arXiv:2411.17617,

arXiv
[2022]

doi: 10.1016/j.neuroimage.2022.119474

ISSN 1053-8119. doi: 10.1016/j.neuroimage.2022.119474. URLhttp://dx.doi.org/10. 1016/j.neuroimage.2022.119474. Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image ...

2022
[2024]

URL https://arxiv.org/abs/2408.10966. Maria Correia de Verdier, Rachit Saluja, Louis Gagnon, Dominic LaBella, Ujjwall Baid, Nourel Hoda Tahon, Martha Foltyn-Dumitru, Jikai Zhang, Maram Alafif, Saif Baig, Ken Chang, Gennaro D’Anna, Lisa Deptula, Diviya Gupta, Muhammad Ammar Hai...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

5 Ren Rivera Oswal Pan Henry W alters Kurt Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu

URLhttps://arxiv.org/abs/2405.18368. 5 Ren Rivera Oswal Pan Henry W alters Kurt Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. InInternational MICCAI Brainlesion Workshop, pages 272–284. Springer,

arXiv

[2] [6]

URL https://doi.org/10.1038/s41592-020-01008-z

doi: 10.1038/s41592-020-01008-z. URL https://doi.org/10.1038/s41592-020-01008-z. Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. InInternational Conference on Medical Image Computing and Computer-Assist...

doi:10.1038/s41592-020-01008-z

[3] [7]

Neuroimaging modalities used for ischemic stroke diagnosis and monitor- ing.Medicina, 59(11):1908,

Jasmin J Nukovic, Valentina Opancina, Elisa Ciceri, Mario Muto, Nebojsa Zdravkovic, Ah- met Altin, Pelin Altaysoy, Rebeka Kastelic, Diana Maria Velazquez Mendivil, Jusuf A Nukovic, et al. Neuroimaging modalities used for ischemic stroke diagnosis and monitor- ing.Medicina, 59(11):1908,

work page 1908

[4] [11]

URLhttps://www

doi: 10.1161/01.STR.0000209238.61459.39. URLhttps://www. ahajournals.org/doi/abs/10.1161/01.STR.0000209238.61459.39. M. Yousufuddin and N. Young. Aging and ischemic stroke.Aging (Albany NY), 11(9): 2542–2544, May

arXiv

[5] [2006]

29609.22

doi: 10.1161/01.STR.0000217418. 29609.22. URLhttps://www.ahajournals.org/doi/abs/10.STR.0000217418.29609

arXiv

[6] [2009]

Epub 2008 Dec

doi: 10.1161/STROKEAHA.108.526954. Epub 2008 Dec

doi:10.1161/strokeaha.108.526954 2008

[7] [2015]

6 How We Won the ISLES’24 Challenge by Preprocessing Max Wintermark, Adam E

URLhttp://arxiv.org/ abs/1505.04597. 6 How We Won the ISLES’24 Challenge by Preprocessing Max Wintermark, Adam E. Flanders, Birgitta Velthuis, Reto Meuli, Maarten van Leeuwen, Dorit Goldsher, Carissa Pineda, Joaquin Serena, Irene van der Schaaf, Annet Waai- jer, James Anderson, Gary Nesbit, Igal Gabriely, Victoria Medina, Ana Quiles, Scott Pohlman, Marcel...

arXiv

[8] [2019]

URLhttps://doi.org/10.18632/ aging.101931

doi: 10.18632/aging.101931. URLhttps://doi.org/10.18632/ aging.101931. 7

doi:10.18632/aging.101931