REVIEW 4 major objections 4 minor 1 cited by
crossMoDA Challenge: Evolution of Cross-Modality Domain Adaptation Techniques for Vestibular Schwannoma and Cochlea Segmentation from 2021 to 2023
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The crossMoDA challenge reports that the 2023 winning model, trained on the most heterogeneous multi-institutional dataset, reduced outliers on earlier single-site test sets while maintaining vestibular schwannoma segmentation accuracy…
desk verdict Solid challenge report with valuable public benchmark results, but the causal claim about data heterogeneity driving robustness is confounded by winner-model differences and needs a softer framing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the challenge benchmark's retrospective winner-model comparison. Each edition's winning pipeline follows the same synthesis-then-segment recipe: an unpaired image-to-image translation network (2D CycleGAN in 2021, NiceGAN in 2022, 3D QS-Attn in 2023) converts labelled ceT1 scans into synthetic T2 scans, a 3D nnUNet is trained on the synthetic T2 scans with the ceT1 labels, and self-training on unlabelled real T2 scans refines the model. The analysis then fixes each edition's test set and evaluates all three winners on it, using Dice and average symmetric surface distance, and counts outliers in box-plot distributions. This cross-evaluation is what isolates the effect of changing training data composition.
What would settle it
Train a single fixed model pipeline on each edition's training data—same architecture, same augmentation, same ensemble size—and evaluate on a common held-out set of homogeneous scans. If the model trained only on the 2021 data matches the outlier count and Dice of the model trained on the full 2023 data, the data-heterogeneity explanation fails. Similarly, removing the intra-/extra-meatal split from the 2023 training objective and observing unchanged cochlea Dice would falsify the claimed cause of the cochlea decline.
Extended reading notes
Core claim
The retrospective analysis shows that the 2023 winning model, trained on the combined London, Tilburg, and UK multi-centre routine data, reduced the number of outliers on the 2021 and 2022 testing data compared with the 2021 and 2022 winners, even though those test sets were acquired under controlled, homogeneous protocols. On the single-institute London test set, the 2023 winner's VS Dice was significantly higher than the 2021 winner's and slightly higher than the 2022 winner's. By contrast, the 2021 winner's performance dropped sharply when applied to the 2022 and 2023 data, and the 2022 winner degraded on the 2023 data. The cochlea Dice of the 2023 winner declined relative to the 2022 winner on the shared 2022 test sets, which the authors attribute to the new intra-/extra-meatal tumour sub-annotation task rather than to poorer translation quality. The conclusion is that heterogeneous training data can improve segmentation robustness even on homogeneous target distributions, while added task complexity can cost accuracy on small structures.
Load-bearing premise
The load-bearing premise is that the design differences between the three winning models do not explain their performance gap, so the better generalization of the 2023 winner can be credited to its more diverse training data rather than to a better model.
Editorial extensions
If this is right
- If the central claim holds, future cross-modality benchmarks should include heterogeneous routine-acquired data, because single-site controlled data understates the generalization gap.
- Models trained only on homogeneous planning data should be expected to produce outlier segmentations when deployed at other sites, so clinical deployment without multi-site training data carries a robustness risk.
- Adding clinically motivated sub-tasks such as intra-/extra-meatal splitting can lower accuracy on small nearby structures, so task complexity needs to be budgeted against core segmentation targets.
- Leading challenge methods are plateauing on the current task, so a harder cross-modal benchmark may be needed to continue driving methodological progress.
- T2-only automated segmentation, if robust across sites, could reduce the need for gadolinium contrast in vestibular schwannoma surveillance, cutting cost and safety risk.
Reading between the lines
- A controlled version of this comparison would randomize model design: for instance, the same team's same architecture could be trained on each edition's dataset, isolating data diversity from translation and ensembling choices; the current paper does not do this.
- The data-heterogeneity benefit is probably not linear: the 2023 dataset adds both more patients and much wider scanner and protocol variation, so separating sheer volume from diversity would clarify which factor drives the outlier reduction.
- If the cochlea decline is truly caused by the sub-segmentation task, lightweight task-specific decoding or a multi-task loss that shields the cochlea head may recover the lost accuracy; this is a testable design hypothesis.
- The same evaluation logic could apply to other cross-modality benchmarks, suggesting a general recipe: judge methods not only on their own test set but on previous editions' test sets to measure robustness transfer.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports the crossMoDA 2022 and 2023 challenge editions and retrospectively analyzes the 2021-2023 series. The challenge task is unsupervised cross-modality segmentation of vestibular schwannoma and cochlea from ceT1 to T2 MRI. The paper describes the datasets (London SC-GK, Tilburg SC-GK, UK MC-RC), the segmentation and Koos classification tasks, the metrics and ranking scheme, and the methods used by participating teams. It then presents the 2022 and 2023 results, a bootstrap-based ranking stability analysis, and a cross-edition comparison of the winning models. The central claim is that the 2023 winner, trained on the most diverse multi-institutional dataset, reduced the number of outliers on the 2021 and 2022 test sets, demonstrating that increased data heterogeneity improves segmentation even on homogeneous data; a secondary claim is that cochlea Dice declined in 2023 due to added sub-annotation complexity.
Significance. If the central claims held, this would be a valuable benchmark paper: it delivers openly available multi-institutional datasets, a reproducible evaluation pipeline (rank-then-aggregate with bootstrapped stability analysis), detailed method summaries from all participating teams, and a longitudinal view of how unsupervised domain-adaptation techniques evolved over three years. The ranking stability analysis in §5.6 is carefully executed and the data-sharing and evaluation-code links are concrete assets. However, the paper's headline causal inference—that data heterogeneity is the driver of the 2023 winner's improved robustness—is not identified by the presented comparison, because the winning models differ across multiple design dimensions simultaneously. The paper's descriptive and benchmarking content is solid, but the causal framing needs either a controlled analysis or a substantial weakening.
major comments (4)
- [§6.2, Fig. 12, Table 8] The central claim that increased data heterogeneity reduces outliers is confounded by winner-model differences. The 2021, 2022, and 2023 winners differ simultaneously in image-translation architecture (2D CycleGAN/PAST, NiceGAN, 3D QS-Attn), segmentation backbone, augmentation strategy, self-training details, and ensemble size (5-model vs. 11-model ensembles). For example, the 2023 winner Vandy365 uses an 11-model ensemble and site-specific style synthesis, whereas the 2022 winner ne2e uses NiceGAN with a 5-model nnUNet ensemble. The observed improvement on the London SC-GK test set could arise from any of these differences, so this is not a controlled test of data heterogeneity. A fixed-architecture ablation varying only the training-data composition—e.g., the 2023 winner's pipeline trained on 2021-only, 2022-only, and 2023 data—would be needed to support the causal reading; absent that, the abstract's 'demonstrating' should be softened to 'consistent with' or 'suggesting.'
- [§6.2, Fig. 12] The statistical evidence behind the p-values is unspecified. The text reports p<0.01 and p<0.005 for several comparisons in Fig. 12, but no test name, test statistic, pairing scheme, or multiple-comparison correction is given. Additionally, 'outliers' and 'outlier spread' are not defined in this section; Section 5.5 refers informally to box-plot outliers, but the reader cannot verify the claim that 'the number of outliers decreases with an expanding dataset' without a concrete outlier rule (e.g., Tukey fences) and outlier counts. Please specify the exact test used (e.g., paired Wilcoxon signed-rank test on per-case Dice values), state whether the test is paired across winners on the same test cases, report effect sizes, and correct for the multiple pairwise comparisons implied by Fig. 12.
- [§6.2 and Abstract] The attribution of the 2023 cochlea decline to sub-segmentation complexity is speculative. The abstract says the decline is 'likely due to the introduction of a subdivided tumour annotation' and §6.2 says it is 'likely due to the increased complexity introduced by multi-institutional training data and increased challenge posed by the new intra/extra-VS sub-segmentation requirements.' However, the 2023 winner differs from the 2022 winner in architecture, training data, and task objective (the 2023 ranking includes intra-meatal, extra-meatal, and cochlea, rather than combined VS and cochlea), so the decline cannot be uniquely attributed to sub-annotation complexity. At minimum, the paper should test or explicitly acknowledge alternative explanations, such as the different ranking objective or the different synthesis/ensemble design.
- [§6.2, Fig. 12] The cross-test evaluation of older winners may be affected by task mismatch. The 2021 and 2022 winners were trained for two-class segmentation (VS and cochlea), while the 2023 winner was trained with intra-meatal and extra-meatal sub-labels. When the 2021/2022 winners are evaluated on the 2023 test set, it is not stated how the combined VS ground truth is derived from the intra/extra masks or whether the older models' predictions were post-processed to match the combined evaluation. If the maps are simply merged, the comparison may penalize older models for a task they were not designed for; please describe the label mapping and any preprocessing applied for these retrospective evaluations.
minor comments (4)
- [§3.2, Eq. (3)] The notation T_c for the set of test samples in class c conflicts with the use of T for target domain elsewhere in the paper; consider renaming to D_c or S_c.
- [§2.4] The text cites 'Erickson et al.' in the Koos grading protocol description without a year or a full reference entry; please add the complete citation.
- [§5.6] The claim that the top-ranked team maintained its position across all bootstrap samples is reported separately for 2022 and 2023, but the blob plot in Fig. 9 would be easier to verify if the percentage of bootstrap samples in which each team held its exact rank were stated numerically in the caption.
- [§3.3.1] The sentence describing the bootstrap sample size states that each sample retained approximately 63% of distinct cases; this is the expected fraction for sampling with replacement, but it would be clearer to state that the statement follows from the Poisson approximation.
Circularity Check
No significant circularity: the paper's central claims are empirical evaluations on held-out test sets; the heterogeneity interpretation is observational and externally checkable, not derived from its own assumptions.
full rationale
The paper's load-bearing assertions—that the 2023 winner reduced outliers on earlier test sets and that cochlea Dice declined—are empirical summaries of held-out challenge evaluations (Tables 5, 8 and Fig. 12). The winning models were submitted by independent teams and scored with pre-defined metrics (DSC, ASSD, MA-MAE) on private test sets, so the results are not produced by fitting a parameter and renaming it a prediction. No result equation is equivalent to an input by construction. The citations to the organizers' earlier crossMoDA 2021 paper (Dorent et al., 2023) document the ranking scheme and prior benchmark results; they do not supply the 2022/2023 numbers or the outlier reduction claim. The conclusion that 'increased data heterogeneity can enhance segmentation performance' is an observational interpretation of a comparison in which winner models differ in architecture, augmentation, and ensemble size as well as training data. That is a confounding/validity concern about causal attribution, not circularity: the observation is still independent evidence subject to alternative explanations. No uniqueness theorem, ansatz-smuggling citation, or renaming of a known result was found. The self-citations present are routine benchmark provenance and are not load-bearing in a circular sense, so the circularity burden is minimal.
Assumptions & free parameters
assumptions (4)
- domain assumption Manual annotations by expert raters are treated as ground truth for VS and cochlea segmentation.
- domain assumption Dice and ASSD capture clinically meaningful segmentation quality for this task.
- domain assumption The winning models are representative of the progress of domain adaptation techniques per edition.
- domain assumption The leaderboard design (one validation submission per day, private test set) prevents substantial overfitting to the test set.
Cite this review
Pith. "Pith review of crossMoDA Challenge: Evolution of Cross-Modality Domain Adaptation Techniques for Vestibular Schwannoma and Cochlea Segmentation from 2021 to 2023." pith.science (2026). https://pith.science/paper/CQTCVYYQ
@misc{pith2026250612006,
author = {Pith},
title = {Pith review of: crossMoDA Challenge: Evolution of Cross-Modality Domain Adaptation Techniques for Vestibular Schwannoma and Cochlea Segmentation from 2021 to 2023},
year = {2026},
howpublished = {\url{https://pith.science/paper/CQTCVYYQ}},
note = {Machine review of arXiv:2506.12006}
}
read the original abstract
The cross-Modality Domain Adaptation (crossMoDA) challenge series, initiated in 2021 in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), focuses on unsupervised cross-modality segmentation, learning from contrast-enhanced T1 (ceT1) and transferring to T2 MRI. The task is an extreme example of domain shift chosen to serve as a meaningful and illustrative benchmark. From a clinical application perspective, it aims to automate Vestibular Schwannoma (VS) and cochlea segmentation on T2 scans for more cost-effective VS management. Over time, the challenge objectives have evolved to enhance its clinical relevance. The challenge evolved from using single-institutional data and basic segmentation in 2021 to incorporating multi-institutional data and Koos grading in 2022, and by 2023, it included heterogeneous routine data and sub-segmentation of intra- and extra-meatal tumour components. In this work, we report the findings of the 2022 and 2023 editions and perform a retrospective analysis of the challenge progression over the years. The observations from the successive challenge contributions indicate that the number of outliers decreases with an expanding dataset. This is notable since the diversity of scanning protocols of the datasets concurrently increased. The winning approach of the 2023 edition reduced the number of outliers on the 2021 and 2022 testing data, demonstrating how increased data heterogeneity can enhance segmentation performance even on homogeneous data. However, the cochlea Dice score declined in 2023, likely due to the added complexity from tumour sub-annotations affecting overall segmentation performance. While progress is still needed for clinically acceptable VS segmentation, the plateauing performance suggests that a more challenging cross-modal task may better serve future benchmarking.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 1 Pith paper
-
IntraStyler: Intra-Domain Style Synthesis for Cross-Modality MRI Domain Adaptation
An exemplar-based style synthesis method that learns scanner-like style vectors via contrastive learning and uses them to generate diverse synthetic T2 images, improving downstream cross-modality segmentation.
Reference graph
Works this paper leans on
-
[4]
How to Cure Cancer (in images) with Unpaired Image Translation, in: MIDL 2018, pp. 1–3. Connor, S.E.,
work page 2018
-
[8]
Optimized u-net for brain tumor segmentation, in: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 7th International Workshop, BrainLes 2021, Held in Conjunction with MICCAI 2021, Virtual Event, September 27, 2021, Revised Selected Papers, Part II, Springer. pp. 15–29. Guan, H., Liu, M.,
work page 2021
-
[10]
Adaptive t-vMF Dice Loss for Multi-class Medical Image Segmentation
Adaptive t-vmf dice loss for multi-class medical image segmentation. arXiv preprint arXiv:2207.07842 . Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y ., Isola, P., Maschinot, A., Liu, C., Krishnan, D.,
-
[11]
Unsupervised domain adaptation for mri volume segmentation and classification using image-to-image translation. arXiv preprint arXiv:2302.08016 . Koos, W.T., Day, J.D., Matula, C., Levy, D.I.,
-
[12]
Hierarchical consistency regularized mean teacher for semi-supervised 3d left atrium segmentation, in: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), IEEE. pp. 3395–3398. Liu, H., Fan, Y ., Li, H., Wang, J., Hu, D., Cui, C., Lee, H.H., Zhang, H., Oguz, I., 2022a. Moddrop ++: A dynamic filter network...
work page 2021
-
[14]
arXiv preprint arXiv:2206.01653
Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv preprint arXiv:2206.01653 . Maier-Hein, L., Reinke, A., Kozubek, M., Martel, A.L., Arbel, T., Eisenmann, M., Hanbury, A., Jannin, P., M ¨uller, H., Onogur, S., et al.,
-
[16]
arXiv preprint arXiv:2203.16557
Cosmos: cross- modality unsupervised domain adaptation for 3d medical image segmentation based on target-aware domain translation and iterative self-training. arXiv preprint arXiv:2203.16557 . Torbunov, D., Huang, Y ., Yu, H., Huang, J., Yoo, S., Lin, M., Viren, B., Ren, Y .,
-
[17]
/ Medical Image Analysis (2025) 25 density changing regularization
Unsupervised image-to-image translation with Wijethilake, Dorent et al. / Medical Image Analysis (2025) 25 density changing regularization. Advances in Neural Information Processing Systems 35, 28545–28558. Yang, T., Wang, L.,
work page 2025
Show all 19 references
-
[19]
IEEE Transactions on Medical Imaging 42, 633–646
Le-uda: Label- efficient unsupervised domain adaptation for medical image segmentation. IEEE Transactions on Medical Imaging 42, 633–646. doi: 10.1109/TMI. 2022.3214766. Zhu, J.Y ., Park, T., Isola, P., Efros, A.A.,
2022
-
[2009]
Evaluation measures for ordinal regression, in: 2009 Ninth international conference on intelligent systems design and applications, IEEE. pp. 283–287. Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shino- hara, R.T., Berger, C., Ha, S.M., Rozycki, M., et al.,
2009
-
[2015]
IEEE transactions on medical imaging 34, 1976–1988
Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion. IEEE transactions on medical imaging 34, 1976–1988. Wijethilake, Dorent et al. / Medical Image Analysis (2025) 23 2021/uni00A0winner/uni00A0 /uni00A02021/uni00A0test/uni00A0s...
2025
-
[2017]
Medical image analysis 35, 250–269
Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Medical image analysis 35, 250–269. Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A.P., Carass, A., et al.,
2015
-
[2018]
arXiv preprint arXiv:1811.02629
Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 . Baldeon Calisto, M., Lai-Yuen, S.K., Puente-Mejia, B., . Stac-da: Structure aware cr...
-
[2019]
CoRR abs/1912.04958
Ana- lyzing and improving the image quality of StyleGAN. CoRR abs/1912.04958. URL: http://arxiv.org/abs/1912.04958, arXiv:1912.04958. Kato, S., Hotta, K.,
1912 arXiv
-
[2020]
Contrastive learning for unpaired image-to-image translation, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, Springer. pp. 319–345. Sall´e, G., Andrade-Miranda, G., Conze, P.H., Boussion, N., Bert, J., Visviki...
2020
-
[2021]
arXiv preprint arXiv:2109.14219
Unsupervised do- main adaptation in semantic segmentation based on pixel alignment and self-training. arXiv preprint arXiv:2109.14219 . Dong, Z., He, Y ., Qi, X., Chen, Y ., Shu, H., Coatrieux, J.L., Yang, G., Li, S.,
-
[2022]
arXiv preprint arXiv:2205.04846
Mnet: Rethinking 2d /3d networks for anisotropic medical image segmentation. arXiv preprint arXiv:2205.04846 . Dorent, R., Kujawa, A., Ivory, M., Bakas, S., Rieke, N., Joutard, S., Glocker, B., Cardoso, J., Modat, M., Batmanghelich, K., et al.,
-
[2023]
Medical Image Analysis 83, 102628
Crossmoda 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation. Medical Image Analysis 83, 102628. 24 Wijethilake, Dorent et al. / Medical Image Analysis (2025) Erickson, N.J., Schmalz, P.G., Agee, B.S., Fo...
2025
-
[2025]
arXiv preprint arXiv:2502.15193 URL: https://arxiv.org/abs/2502.15193
Image translation-based unsupervised cross- modality domain adaptation for medical image segmentation. arXiv preprint arXiv:2502.15193 URL: https://arxiv.org/abs/2502.15193. Yushkevich, P.A., Piven, J., Cody Hazlett, H., Gimpel Smith, R., Ho, S., Gee, J.C., Gerig, G.,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.