REVIEW 4 major objections 5 minor 2 cited by
This paper claims that a U-Net can extend its input with a single modality-agnostic channel, trained on synthetic MRI contrasts, so it can segment scans from MRI sequences never seen during training, without losing performance on the sequen
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 19:21 UTC pith:IBOMHD7R
load-bearing objection A genuinely novel idea for handling unseen MRI modalities at inference, but single-run numbers and one failed scenario leave the main claim unproven. the 4 major comments →
Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a simple architectural addition—one extra input channel, optionally with a small convolutional pathway—can absorb an MRI contrast that was absent from training, so long as its training signal is a set of synthetic contrasts built by transforming existing modalities. The paper demonstrates this on eight databases covering five pathologies: when a held-out sequence (DWI or FLAIR) is fed through the agnostic channel at inference, segmentation Dice improves on the unseen-modality test sets, while performance on seen-modality datasets remains close to the standard model. The authors interpret this as evidence that the agnostic channel learns a general contrast-invariant
What carries the argument
The core mechanism is a modality-agnostic input channel (Cagn) added to the U-Net's set of modality-specific channels. During training this channel receives dropped modalities and synthetically augmented versions of them—shift/scale, lesion switch, inversion, mixup—applied separately to lesion and healthy tissue regions. An optional 'agnostic path' of three convolutional layers processes this channel separately and concatenates its 8 feature maps with the other inputs. This gives the network a designated slot for a new contrast at inference without disturbing the modality-specific filters.
Load-bearing premise
The whole scheme rests on the idea that synthetic contrasts made by linear transforms of training modalities (scale, shift, inversion, mixup, lesion switching) are close enough in the network's feature space to real unseen MRI contrasts like DWI or FLAIR; if they are not, the agnostic channel adds noise instead of signal.
What would settle it
A concrete check: take a model trained with the proposed augmentations and feed it a genuinely novel contrast whose physical statistics are far from the synthetic ones—e.g., an MR sequence with very different T1/T2 weighting, or a synthetic contrast from a physics simulator—and measure whether the agnostic channel still improves Dice. A simpler within-paper check: in Setting 1 on ISLES22, the shuffled-channel baseline already scored 40.8 Dice with DWI, while the proposed agnostic path scored 35.6, so an independent replication should verify whether the agnostic channel actually beats random ch
If this is right
- A single trained model can be deployed in a center whose MRI protocol differs from training: any available sequence can be routed through the agnostic channel, and the model still segments.
- Performance on modalities seen during training is preserved: on held-out datasets with only seen modalities (e.g., TUMOUR2), the agnostic path model scores within a small margin of the standard model.
- The benefit transfers across pathologies: the same augmentation-plus-agnostic-channel recipe improved Dice for acute stroke, sub-acute stroke, and white-matter hyperintensities, suggesting it is not tuned to one lesion type.
- Fine-tuning a pre-trained agnostic-path model with the new modality in the agnostic channel outperforms fine-tuning a standard model with a randomly initialized channel, giving a practical recipe for adapting to a new center's protocol.
- The augmentation ablation shows that both tissue-specific and uniform synthetic contrasts contribute, but their relative value depends on the target modality—so augmentation choice is an active lever for improving transfer.
Where Pith is reading between the lines
- The synthetic-to-real gap is the binding constraint: on ISLES22, the shuffled-channel baseline scored higher with DWI (40.8) than the agnostic path (35.6), implying that when the synthetic contrasts are far from the true contrast, the channel adds noise rather than signal.
- A physics-based augmentation strategy (e.g., simulating MR contrast equations) could produce synthetic modalities that better cover the space of real sequences; the current linear transforms are a first-order approximation.
- The agnostic channel may be learning a generic lesion-saliency detector rather than a true modality-invariant representation; if so, feeding it two unseen modalities at once (or a modality very unlike any synthetic one) would stress-test that gloss.
- Since the method makes no MRI-specific assumptions beyond the augmentations, the same architecture could be tried for combining CT and PET or other multi-contrast imaging, though the paper only demonstrates MRI.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a simple modification to a U-Net for multimodal brain MRI lesion segmentation: a modality-agnostic input channel, optionally with a small convolutional pathway, dedicated to processing modalities not seen during training. During training, this channel is fed with synthetic modalities generated by linear intensity augmentations (scale, shift, inversion, mixup, lesion switch) applied to dropped or retained modalities. The method is evaluated on eight brain MRI datasets across two training configurations, with DWI or FLAIR held out as the unseen modality. The central claim is that the agnostic path preserves performance on seen modalities while improving segmentation when an unseen modality is supplied at inference. Fine-tuning experiments are also reported.
Significance. If the central claim were solidly established, this would be a practically useful contribution: a simple, architecture-level way to let a multimodal segmentation model accept a new MRI contrast without retraining. The work leverages several public datasets plus two private datasets, covers multiple lesion types, and the code is released. The augmentation scheme is clearly described. However, the main quantitative evidence is a single table of average Dice values from single runs, with no error bars, seeds, or significance tests. Several of the key gains are small or negative relative to baselines. The strength of the conclusion therefore currently exceeds the strength of the evidence.
major comments (4)
- [§3.2, Table 2] The central claim that the agnostic path 'can leverage unseen modalities across most settings' rests entirely on average Dice from single training runs. No error bars, seeds, or significance tests are reported. In Setting 2, the improvement on ISLES15 from adding unseen FLAIR is 51.1→51.6 (+0.5 Dice), which is within the typical run-to-run variation of 3D U-Nets on small test sets. In Setting 1, the ISLES22 gain with DWI is 30.4→35.6, but the Shuffled baseline reaches 40.8 and the Single-input baseline reaches 38.5. As reported, the evidence does not establish that the method reliably improves segmentation with unseen modalities. Please provide multiple seeds (e.g., at least 5) with confidence intervals and paired significance tests for the key comparisons.
- [§3.2, Setting 1 / Table 2] The paper's own discussion concedes that on ISLES22 the shuffled model achieved the best performance, while the agnostic path with unseen DWI (35.6) is worse than both Shuffled (40.8) and Single-input (38.5). This is a direct counterexample to the claim that the method improves segmentation with unseen modalities. The conclusion 'across most settings' is too strong as stated. The manuscript should characterize the conditions under which the agnostic path helps versus hurts, and should not count ISLES22 as a success. Simply saying the model 'failed to learn the combination of FLAIR and DWI effectively' does not explain why the proposed mechanism underperforms naive baselines.
- [§2.2, Eq. (4)] The load-bearing premise is that synthetic modalities built by linear intensity transforms (shift, scale, inversion, mixup, lesion switch) are close enough in feature space to genuine unseen physical contrasts (DWI, FLAIR) that the agnostic filters transfer. The paper never tests this premise directly. The failure on acute stroke ISLES22 is consistent with the synthetic augmentations poorly covering the statistics of DWI in that dataset. Please add a feature-distribution analysis or an oracle experiment (e.g., training with the real unseen modality in the agnostic channel) to estimate the upper bound of the augmentation strategy and to validate that the synthetic contrasts occupy a useful region of feature space.
- [Tables 3 and 4] The ablation study and fine-tuning experiments also report single Dice numbers without variance. For example, in Table 3 the difference between 'Lesion Switch' (54.3) and 'Mix up' (53.8) on WMH, or between 'Comb. of augs.' (57.6) and 'Invert' (57.3), is not interpretable without error bars. Moreover, the augmentation hyperparameters are not given: the probabilities p, the shift/scale bounds (a,b,c,d), the mixup λ, and the number of feature maps in the agnostic path are not specified. The code release mitigates this, but the paper should state these values to make the experiments reproducible.
minor comments (5)
- [§2.2, Eq. (2)] The notation Msp ⊕ Maug is used without defining the operator. It should be clarified as channel-wise concatenation. Similarly, the notation Csp ∪ {Cagn} is unusual for channel sets; a consistent notation would improve readability.
- [§3.2, Discussion] There is an encoding artifact in 'modelâĂŸs attention' that should be fixed (likely a Unicode issue).
- [§3.1, Train Setting 2] The text says 'Train Setting 2: ISLES2022, MSSEG, BRATS, TBI, ATLAS (FLAIR removed).' It should be clarified whether 'FLAIR removed' means the FLAIR channel is removed from the model's channel set, or whether FLAIR inputs are simply zeroed during training while the channel remains in the architecture. This affects the interpretation of the Standard model's low WMH score (27.9).
- [§2.2, Augmentation parameters] The text mentions 'independently calculated probabilities p∈[0,1]' for each region and augmentation, but no values or ranges are reported. The 'same augmentation parameters were applied to both settings' claim is not verifiable without these values.
- [Table 1] The column header 'Train/Test Total Size' is ambiguous. It appears to list train/test split and total size, but the formatting should be clearer for readers. Also, ISLES15 is listed as 0 train / 28 test and TUMOUR2 as 0 train / 57 test; it would help to state explicitly that these are used only for evaluation.
Circularity Check
No significant circularity: unseen modalities are genuinely held out and evaluated against external Dice labels; self-citations are lineage, not load-bearing.
full rationale
The central claim is that adding a modality-agnostic channel/path, trained with synthetic augmentations, lets a multimodal U-Net exploit MRI modalities absent from training. The evaluation (Table 2) places DWI or FLAIR, which are absent from all training databases in Settings 1 and 2 respectively, into the agnostic channel and measures Dice against external ground truth. No parameter is fitted to the test labels, and the synthetic augmentations (Eq. 4) are generic intensity/contrast transformations, not learned mappings from the unseen modalities. The architectural definitions in Eqs. 1-3 are definitions of the compared configurations, not a derivation that equates the prediction with the training signal. The references to the authors' prior work [20] are used only for the standard multi-dataset U-Net backbone and Modality Dropout; these are disclosed and are not used to justify the new unseen-modality capability. The paper's own discussion acknowledges that the agnostic path underperforms the shuffled baseline on ISLES22 in Setting 1, which is a limitation of transfer effectiveness rather than circularity. Statistical concerns about single-run evaluation are validity risks, not definitional or self-referential reductions. No enumerated circularity pattern applies.
Axiom & Free-Parameter Ledger
free parameters (4)
- Augmentation probabilities p per region and per augmentation =
not reported
- Shift and scale bounds (a,b,c,d) =
not reported
- Number of output feature maps of the agnostic pathway =
8
- Augmentation application order =
Lesion Switch, Inversion, MixUp, Scale, Shift
axioms (4)
- domain assumption Synthetic augmented contrasts adequately represent the distribution of genuine unseen MRI modalities
- domain assumption Input modalities are spatially co-registered so the unseen modality can be concatenated as an extra channel
- domain assumption Lesion and brain masks used by the augmenter are accurate enough that tissue-specific transforms preserve the binary label
- domain assumption A U-Net with zero-filled absent input channels is a sound backbone for joint learning across heterogeneous modality sets
Cite this review
Pith. "Pith review of Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training." pith.science (2026). https://pith.science/paper/IBOMHD7R
@misc{pith2026250909290,
author = {Pith},
title = {Pith review of: Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/IBOMHD7R}},
note = {Machine review of arXiv:2509.09290}
}
read the original abstract
Segmentation models are important tools for the detection and analysis of lesions in brain MRI. Depending on the type of brain pathology that is imaged, MRI scanners can acquire multiple, different image modalities (contrasts). Most segmentation models for multimodal brain MRI are restricted to fixed modalities and cannot effectively process new ones at inference. Some models generalize to unseen modalities but may lose discriminative modality-specific information. This work aims to develop a model that can perform inference on data that contain image modalities unseen during training, previously seen modalities, and heterogeneous combinations of both, thus allowing a user to utilize any available imaging modalities. We demonstrate this is possible with a simple, thus practical alteration to the U-net architecture, by integrating a modality-agnostic input channel or pathway, alongside modality-specific input channels. To train this modality-agnostic component, we develop an image augmentation scheme that synthesizes artificial MRI modalities. Augmentations differentially alter the appearance of pathological and healthy brain tissue to create artificial contrasts between them while maintaining realistic anatomical integrity. We evaluate the method using 8 MRI databases that include 5 types of pathologies (stroke, tumours, traumatic brain injury, multiple sclerosis and white matter hyperintensities) and 8 modalities (T1, T1+contrast, T2, PD, SWI, DWI, ADC and FLAIR). The results demonstrate that the approach preserves the ability to effectively process MRI modalities encountered during training, while being able to process new, unseen modalities to improve its segmentation. Project code: https://github.com/Anthony-P-Addison/AGN-MOD-SEG
Figures
Forward citations
Cited by 2 Pith papers
-
Pretraining EHR Foundation Models with Patient-Aware Sampling
Patient-aware sampling of pretraining windows, with patients weighted by a tunable exponent, improves downstream AUROC/AUPRC over a global token-stream baseline in autoregressive EHR models.
-
Autoregressive EHR Foundation Models with Multimodal Inputs
Latent compression improves multimodal fusion, but ICU mortality predictions from EHR plus ECG, CXR, and notes are generally no better than EHR-only.
Reference graph
Works this paper leans on
-
[1]
International conference on medical imaging with deep learning (2022)
Azad, R., Khosravi, N., Merhof, D.: SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities. International conference on medical imaging with deep learning (2022)
2022
-
[2]
Scientific data (2017)
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., Davatzikos, C.: Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data (2017)
2017
-
[3]
Medical Image Analysis (2023)
Billot, B., Greve, D.N., Puonti, O., Thielscher, A., Van Leemput, K., Fischl, B., Dalca, A.V., Iglesias, J.E.: SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining. Medical Image Analysis (2023)
2023
-
[4]
BrainLes Workshop, International Conference on Medical Image Computing and Computer-Assisted Intervention (2019)
Chen, W., Liu, B., Peng, S., Sun, J., Qiao, X.: S3D-UNET: Separable 3D U-Net for brain tumor segmentation. BrainLes Workshop, International Conference on Medical Image Computing and Computer-Assisted Intervention (2019)
2019
-
[5]
Scientific Reports 2018 8:1 (2018)
Commowick, O., Istace, A., Kain, M., Laurent, B., Leray, F., Simon, M., Pop, S.C., Girard, P., Ameli, R., Ferré, J.C., et al.: Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure. Scientific Reports 2018 8:1 (2018)
2018
-
[6]
Advances in neural information processing systems32(2019)
Dou, Q., Coelho de Castro, D., Kamnitsas, K., Glocker, B.: Domain generalization via model-agnostic learning of semantic features. Advances in neural information processing systems32(2019)
2019
-
[7]
Interna- tional Conference on Medical Image Computing and Computer-Assisted Interven- tion (2017)
Ghafoorian, M., Mehrtash, A., Kapur, T., Karssemeijer, N., Marchiori, E., Pesteie, M., G Guttmann, C.R., de Leeuw, F.E., Tempany, C.M., van Ginneken, B., Fe- dorov, A., Abolmaesumi, P., Platel, B., Wells III, W.M.: Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation. Interna- tional Conference on Medical Image Computin...
2017
-
[8]
International Conference on Medical Image Computing and Computer-Assisted Intervention (2016)
Havaei, M., Guizard, N., Chapados, N., Bengio, Y.: HeMIS: Hetero-modal im- age segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention (2016)
2016
-
[9]
Scientific Data 2022 9:1 (2022) 12 A.P.Addison et al
Hernandez Petzsche, M.R., de la Rosa, E., Hanning, U., Wiest, R., Valenzuela, W., Reyes, M., Meyer, M., Liew, S.L., Kofler, F., Ezhov, I., et al.: ISLES 2022: A multi- center magnetic resonance imaging stroke lesion segmentation dataset. Scientific Data 2022 9:1 (2022) 12 A.P.Addison et al
2022
-
[10]
International Conference on Medical Image Computing and Computer- Assisted Intervention (2024)
Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jäger, P.: nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Seg- mentation. International Conference on Medical Image Computing and Computer- Assisted Intervention (2024)
2024
-
[11]
Information Processing in Medical Imaging (IPMI) (2017)
Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., et al.: Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. Information Processing in Medical Imaging (IPMI) (2017)
2017
-
[12]
Medical Image Analysis (2017)
Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis (2017)
2017
-
[13]
IEEE TMI (2019)
Kuijf, H.J., Biesbroek, J.M., De Bresser, J., Heinen, R., Andermatt, S., Bento, M., Berseth, M., Belyaev, M., Cardoso, M.J., Casamitjana, A., et al.: Standard- ized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge. IEEE TMI (2019)
2019
-
[14]
Scientific Data 2022 9:1 (2022)
Liew, S.L., Lo, B.P., Donnelly, M.R., Zavaliangos-Petropulu, A., Jeong, J.N., Barisano, G., Hutton, A., Simon, J.P., Juliano, J.M., Suri, A., et al.: A large, curated, open-source stroke neuroimaging dataset to improve lesion segmentation algorithms. Scientific Data 2022 9:1 (2022)
2022
-
[15]
Nature Communications 2024 15:1 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 2024 15:1 (2024)
2024
-
[16]
Medical image analysis (2017)
Maier, O., Menze, B.H., Von der Gablentz, J., Häni, L., Heinrich, M.P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., other: ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical image analysis (2017)
2015
-
[17]
IEEE Transactions on Medical Imaging (2022)
Ouyang, C., Chen, C., Li, S., Li, Z., Qin, C., Bai, W., Rueckert, D.: Causality- inspired Single-source Domain Generalization for Medical Image Segmentation. IEEE Transactions on Medical Imaging (2022)
2022
-
[18]
Diagnostics (2023)
Shi, P., Qiu, J., Abaxi, S.M.D., Wei, H., Lo, F.P.W., Yuan, W.: Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation. Diagnostics (2023)
2023
-
[19]
Neu- roImage: Clinical (2019)
Valverde, S., Salem, M., Cabezas, M., Pareto, D., Vilanova, J.C., Ramió-Torrentà, L., Rovira, Ã., Salvi, J., Oliver, A., Lladó, X.: One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks. Neu- roImage: Clinical (2019)
2019
-
[20]
Medical Imaging with Deep Learning (2024)
Xu, W., Moffat, M., Seale, T., Liang, Z., Wagner, F., Whitehouse, D., Menon, D., Newcombe, V., Voets, N., Banerjee, A., Kamnitsas, K.: Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentation. Medical Imaging with Deep Learning (2024)
2024
-
[21]
International Conference on Learning Representations (2018)
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond Empirical Risk Minimization. International Conference on Learning Representations (2018)
2018
-
[22]
In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention
Zhang, X., Ou, N., Basaran, B.D., Visentin, M., Qiao, M., Gu, R., Ouyang, C., Liu, Y., Matthews, P.M., Ye, C., et al.: A foundation model for brain lesion seg- mentation with mixture of modality experts. In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention. pp. 379–389. Springer (2024)
2024
-
[23]
DLMIA 2018 workshop, Inter- national Conference on Medical Image Computing and Computer-Assisted Inter- vention (2018)
Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. DLMIA 2018 workshop, Inter- national Conference on Medical Image Computing and Computer-Assisted Inter- vention (2018)
2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.