Pith. sign in

REVIEW 3 major objections 6 minor 15 references

Multi-modal brain MRI synthesis based on SwinUNETR

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A stock SwinUNETR, with only its input and output channel counts changed to three and one, synthesizes missing brain MRI modalities well enough that downstream brain-tumor segmentation shows no significant differences when the synthetic…

desk verdict A straight stock-SwinUNETR baseline for MRI missing-modality synthesis with credible external test scores; the baseline claim holds, but the abstract oversells and the validation Dice is a pipeline-agreement metric, not ground truth. read the letter →

arxiv 2506.02467 v1 pith:B3JOUUMY submitted 2025-06-03 eess.IV cs.CV

classification eess.IVcs.CV
keywords MRIsynthesismissingmodalitySwinUNETRbraintumorsegmentationTransformerBraTSmedicalimageencoder-decoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a stock SwinUNETR, with no architectural change except setting three input channels and one output channel, can synthesize a missing brain MRI modality from the other three. On the BraTS 2023 synthesis validation set it reports whole-image SSIM between 93% and 95% depending on which modality is missing, and on the challenge test set an SSIM of 0.8182. The paper also claims that replacing a real modality with the synthesized one leaves brain-tumor segmentation essentially unchanged, with Dice scores against the all-real pipeline around 0.78-0.90 on the validation set and 0.70-0.83 against ground-truth masks on the test set. If this is right, a plain MSE-trained SwinUNETR is a serviceable baseline for missing-modality synthesis, and intricate generative models are not required to get usable images.

What carries the argument

The object that carries the argument is the SwinUNETR architecture itself: a U-shaped encoder-decoder in which the encoder is a hierarchical Swin Transformer built from shifted-window self-attention blocks, and the decoder is a stack of CNN residual blocks with deconvolutions and skip connections from each encoder stage. The paper's only modification is to set the first layer to accept three input channels and the last to emit one output channel, so the same network learns the mapping from the known modalities to the missing one. Training uses Z-score standardized patches and MSE loss, with a sliding-window, Gaussian-weighted fusion at inference; no tumor masks, auxiliary data, or pre-trained weights are used.

What would settle it

Compute the same validation-pipeline Dice after replacing the synthesized modality with a constant image or with the per-dataset mean; if the Dice against the all-real pipeline remains comparable to Table 1, then the reported metric is insensitive to synthesis quality. Alternatively, re-evaluate the validation cases with ground-truth tumor masks (once available) and show that segmentation with the synthetic scan matches ground truth as well as segmentation with the real scan does.

Watch

Extended reading notes

Core claim

The central claim is that SwinUNETR's U-shaped design, a hierarchical Swin Transformer encoder with shifted-window self-attention paired with a CNN decoder and skip connections, is sufficient for cross-modal brain MRI synthesis. Four models are trained, one per missing modality (T1w, T2w, FLAIR, T1CE), each taking the other three modalities as input and supervised only by MSE loss on 128-cubed patches. The paper reports the synthesized images as realistic, and argues their clinical value through segmentation: when the synthetic scan replaces the missing one, tumor segmentation masks agree with those obtained from four real scans, and on the held-out test set the challenge's downstream segmentation (FeTS) achieves Dice of 0.697 (ET), 0.7425 (TC), and 0.8343 (WT) against ground truth. The authors conclude there are no significant differences, and that the synthesized missing modality can be used effectively in brain tumor segmentation.

Load-bearing premise

The load-bearing premise is that the validation Dice between segmentations made with a synthetic scan and segmentations made with all real scans measures the clinical value of the synthetic scan; because the three real modalities already carry most of the tumor information, similar segmentations do not prove the synthetic scan itself is accurate.

Editorial extensions

If this is right

  • A stock SwinUNETR trained with plain MSE loss gives a reproducible baseline for missing-modality MRI synthesis; more elaborate models must be compared against it rather than against no baseline.
  • The synthesized modality can substitute for the missing scan in a downstream brain-tumor segmentation pipeline without a large drop in Dice, which supports clinically relevant use.
  • The method generalizes across all four missing-modality scenarios with a single architecture and per-modality models, suggesting the architecture captures cross-modal relationships without task-specific design.
  • Since no tumor masks or pre-trained weights are needed, the pipeline is easy to reproduce on a single 32 GB GPU.
  • Synthesis quality is modality-dependent (T1w highest, FLAIR and T1CE lowest), which points to where future loss functions or adversarial components would help most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The validation Dice is computed between two segmentation pipelines, not against ground truth; because the three real modalities already contain most tumor signal, high agreement may persist even if the synthesized image is noticeably wrong, so the validation numbers should be read as an upper bound on segmentation interchangeability, not a direct measure of synthesis fidelity.
  • A direct test of the paper's 'no significant differences' claim would be to replace the synthesized modality with a constant or with the mean of training images and recompute the validation Dice; if the Dice stays similar, the metric is not sensitive to synthesis quality.
  • The paper does not compare with pGAN, ResViT, MMT, or M2DN, so its contribution is a simplicity baseline rather than a demonstrated quality improvement; a fair comparison on the same challenge split would show whether the extra complexity of those methods buys anything.
  • The same architecture could be tested on other multi-modal settings (e.g., missing sequences in stroke or prostate MRI) to see whether the three-to-one input-output mapping transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper applies the SwinUNETR architecture, with only input/output channel counts changed, to synthesize missing brain MRI modalities (T1w, T2w, FLAIR, T1CE) from the three available modalities. Four models are trained with MSE loss on BraTS challenge data without tumor masks or pre-trained weights. Results are reported on the challenge validation set (Dice between segmentations using synthetic versus real inputs, and whole-image SSIM) and on the test set using organizer-computed metrics (SSIM, Dice, and HD95 for tumor segmentation). The authors conclude that SwinUNETR effectively synthesizes missing modalities and that the synthetic images can be used in brain tumor segmentation.

Significance. If the central claim holds, the paper provides a useful and simple baseline for missing-modality synthesis: an unmodified SwinUNETR trained with MSE loss produces usable synthetic images. The test-set evaluation was performed externally by challenge organizers on hidden labels, which is a genuine strength, and the manuscript clearly states its design choices. However, the evidence as presented does not isolate the contribution of the synthesized image from the other three real modalities, and several claims ('significant improvements', 'no significant differences') are not supported by the reported analysis. The paper is therefore a plausible baseline contribution whose verification needs additional controls before the central claim can be accepted.

major comments (3)
  1. [§3, Table 1] The Table 1 Dice values are overlaps between segmentations obtained with four real modalities and segmentations obtained with three real plus one synthetic modality, not Dice against ground-truth tumor masks. With three real modalities present, a segmentation network can produce nearly identical masks even if the synthetic image is poor or effectively ignored. The conclusion that 'the synthesized missing modality can be effectively used in brain tumor segmentation tasks' therefore does not follow from Table 1. Please add a minimal control on data with ground-truth masks (for example, a split of the challenge training set): compare four-real inputs, three-real inputs with the missing modality omitted, three-real plus a trivial replacement (zero-fill or mean template), and three-real plus the synthetic image. The central claim is supported only if the synthetic condition improves over the omission and trivial-replacement conditions.
  2. [Abstract, §3 Table 3, §4 Discussion] The externally evaluated test-set scores in Table 3 are a strength, but they report only the synthetic-infilled condition. The abstract claims 'significant improvements' and the Discussion claims 'no significant differences' without any statistical test and without any comparison condition. Please add a comparison to the cited synthesis methods (pGAN/cGAN, ResViT, MMT, M2DN) where published numbers are available, or at minimum add the missing-modality-omitted and all-real baselines, and report appropriate statistical significance testing. If such comparisons cannot be added, the comparative and significance claims in the Abstract and Discussion should be removed or scaled back to descriptive statements about the achieved scores.
  3. [§3, Tables 2 and 3; §4] Whole-image SSIM is dominated by large extra-tumoral brain regions and is insensitive to errors in small tumor regions. The manuscript acknowledges that tumor-region SSIM could not be computed, but it does not qualify the conclusions about 'clinically valuable images' and 'anatomical consistency' accordingly. Please provide a region-aware evaluation (for example, SSIM within enhancing tumor, edema, and whole-tumor regions using the available training-set masks) or explicitly discuss how the reported SSIM would behave if tumor regions were corrupted; without this, the clinical-value claim is not supported by the reported metrics.
minor comments (6)
  1. [Abstract] There is a grammatical issue in 'to the synthesize of missing modalities', and the phrase 'significant improvements' is not justified by any comparison in the manuscript; please rephrase.
  2. [§2.3] The manuscript says 'the default learning rate' for Adam but does not specify the exact learning rate, weight decay, or the number of training patches sampled per epoch; please provide these details for reproducibility.
  3. [Eq. (2)] The summation notation 'nX' in Eq. (2) appears to be a formatting error; the equation should be typeset cleanly as a standard MSE formula.
  4. [§3, Table 1 caption] The table caption contains the typo 'T able 1'; also, the caption should explicitly state that the Dice is computed between two segmentation outputs rather than against ground-truth masks.
  5. [General] No code or data availability statement is included; since the paper is intended as a baseline, releasing the training code and the four trained models would materially increase its usefulness.
  6. [§3, Figures 3 and 4] The figure labels are difficult to parse (for example, 'Syn T1W' and 'Real ImageSynthesized image'); please use a clearer layout with explicit row/column legends indicating which images are real and which are synthetic.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SwinUNETR synthesis is an empirical benchmark with external test evaluation; the weak segmentation-agreement control is a measurement issue, not a derivation that reduces to its inputs.

full rationale

The paper makes no claimed derivation that reduces to its inputs. SwinUNETR is used as an off-the-shelf architecture cited to external works [5, 15], with only input and output channel counts changed; training uses MSE against the real target modality (Eq. 2) on the challenge training set, and evaluation is on held-out validation and test data. The test-set metrics in Table 3 were computed by the challenge organizers with the external FeTS algorithm, so the reported numbers are not generated by the authors' own fitting loop and there is no evidence that the test set was used for model selection. The validation Dice in Table 1 is a pipeline-agreement measure between synthetic-infilled and all-real segmentations, which is a weak experimental control for isolating the synthetic modality's contribution, but that is a measurement-validity limitation, not circular reasoning. No load-bearing premise is justified by a same-author citation; no parameter is fitted and then renamed a prediction; no uniqueness theorem or ansatz is imported from the authors' prior work. The central claim therefore has independent empirical content and does not reduce by construction to its inputs, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No invented entities are introduced. The central result is an empirical evaluation of an existing architecture, so the ledger contains one hand-chosen training configuration and four domain-level modeling assumptions about preprocessing, loss, patch-based training, and the external benchmark. The paper does not present a derivation whose constants need auditing.

free parameters (1)
  • Training hyperparameters = patch size 128x128x128; 100 epochs; batch size 1; Adam default learning rate
    Chosen by hand in Section 2.3; no sensitivity analysis or ablation is provided, so the reported performance is conditional on this configuration.
assumptions (4)
  • domain assumption Per-modality Z-score standardization is sufficient preprocessing for cross-subject synthesis
    Section 2.2 applies Z-score normalization without bias-field correction or skull stripping; the method assumes this normalization aligns intensity distributions across subjects.
  • domain assumption Mean squared error is an adequate training objective for clinically useful MRI synthesis
    Section 2.3 uses only MSE loss; the paper's own evaluation relies on SSIM and downstream segmentation, which are not optimized by MSE.
  • domain assumption Patch-based training with random 128^3 patches and Gaussian-weighted sliding-window fusion preserves whole-image consistency
    Section 2.3; no analysis of stitching artifacts or patch-boundary effects is reported.
  • domain assumption The BraTS BraSyn challenge data splits and organizer-reported test evaluation are reliable external benchmarks
    Sections 2.3 and 3 depend entirely on challenge-provided data and on the organizers' FeTS evaluation for the test-set numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-modal brain MRI synthesis based on SwinUNETR." pith.science (2026). https://pith.science/paper/B3JOUUMY

@misc{pith2026250602467,
  author       = {Pith},
  title        = {Pith review of: Multi-modal brain MRI synthesis based on SwinUNETR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3JOUUMY}},
  note         = {Machine review of arXiv:2506.02467}
}
read the original abstract

Multi-modal brain magnetic resonance imaging (MRI) plays a crucial role in clinical diagnostics by providing complementary information across different imaging modalities. However, a common challenge in clinical practice is missing MRI modalities. In this paper, we apply SwinUNETR to the synthesize of missing modalities in brain MRI. SwinUNETR is a novel neural network architecture designed for medical image analysis, integrating the strengths of Swin Transformer and convolutional neural networks (CNNs). The Swin Transformer, a variant of the Vision Transformer (ViT), incorporates hierarchical feature extraction and window-based self-attention mechanisms, enabling it to capture both local and global contextual information effectively. By combining the Swin Transformer with CNNs, SwinUNETR merges global context awareness with detailed spatial resolution. This hybrid approach addresses the challenges posed by the varying modality characteristics and complex brain structures, facilitating the generation of accurate and realistic synthetic images. We evaluate the performance of SwinUNETR on brain MRI datasets and demonstrate its superior capability in generating clinically valuable images. Our results show significant improvements in image quality, anatomical consistency, and diagnostic value.

Figures

Figures reproduced from arXiv: 2506.02467 by the authors.

Figure 1
Figure 1. Multi-modal MRI plays a critical role in comprehensive assessment of brain tumors. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of SwinUNETR. simple method based on a encoder-decoder network to synthesize MRI, which provides a baseline for future research. 2 Methods 2.1 Network structure In this work, we apply SwinUNETR [5,15] to synthesize missing modalities in brain MRI. SwinUNETR utilizes a U-shaped architecture, with the Swin Trans￾former [10] serving as the encoder and a CNN-based decoder connected via skip connections … view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The segmentation results of the four examples shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: An example of missing modality synthesis, where three modalities are used to synthesize the missing one. The first row displays the four real modalities, while the second row presents the four synthesized modalities [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

  1. [1]

    arXiv preprint arXiv:2107.02314 (2021)

    Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., Colak, E., Fara- hani, K., Kalpathy-Cramer, J., Kitamura, F.C., Pati, S., et al.: The RSNA-ASNR- MICCAI BraTS 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021)

  2. [2]

    Scientific Data 4(1), 1–13 (2017)

    Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., Davatzikos, C.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data 4(1), 1–13 (2017)

  3. [3]

    IEEE Transactions on Medical Imaging41(10), 2598–2614 (2022)

    Dalmaz, O., Yurt, M., Çukur, T.: ResViT: residual vision transformers for mul- timodal medical image synthesis. IEEE Transactions on Medical Imaging41(10), 2598–2614 (2022)

  4. [4]

    IEEE Transactions on Medical Imaging38(10), 2375–2388 (2019)

    Dar, S.U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., Cukur, T.: Image synthe- sis in multi-contrast MRI with conditional generative adversarial networks. IEEE Transactions on Medical Imaging38(10), 2375–2388 (2019)

  5. [5]

    In: International MICCAI brainlesion workshop

    Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in MRI images. In: International MICCAI brainlesion workshop. pp. 272–284. Springer (2021)

  6. [6]

    Advances in Neural Information Processing Systems33, 6840–6851 (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems33, 6840–6851 (2020)

  7. [7]

    Nature Machine Intelligence5(7), 799–810 (2023)

    Karargyris, A., Umeton, R., Sheller, M.J., Aristizabal, A., George, J., Wuest, A., Pati, S., Kassem, H., Zenk, M., Baid, U., et al.: Federated benchmarking of medi- cal artificial intelligence with medperf. Nature Machine Intelligence5(7), 799–810 (2023)

  8. [8]

    Li, H.B., Conte, G.M., Anwar, S.M., Kofler, F., Ezhov, I., van Leemput, K., Piraud, M.,Diaz,M.,Cole,B.,Calabrese,E.,etal.:Thebraintumorsegmentation(BraTS) Multi-modal brain MRI synthesis based on SwinUNETR 9 challenge2023:BrainMRimagesynthesisfortumorsegmentation(BraSyn).ArXiv (2023)

Show all 15 references
  1. [9]

    IEEE Transactions on Medical Imaging42(9), 2577–2591 (2023)

    Liu, J., Pasumarthi, S., Duffy, B., Gong, E., Datta, K., Zaharchuk, G.: One model to synthesize them all: Multi-contrast multi-scale transformer for missing data imputation. IEEE Transactions on Medical Imaging42(9), 2577–2591 (2023)

  2. [10]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchical visiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021)

  3. [11]

    IEEE Transactions on Medical Imaging (2024)

    Meng, X., Sun, K., Xu, J., He, X., Shen, D.: Multi-modal Modality-masked Dif- fusion Network for Brain MRI Synthesis with Random Modality Missing. IEEE Transactions on Medical Imaging (2024)

  4. [12]

    IEEE Transactions on Medical Imaging 34(10), 1993–2024 (2014)

    Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging 34(10), 1993–2024 (2014)

  5. [13]

    Nature Communications 13(1), 7346 (2022)

    Pati, S., Baid, U., Edwards, B., Sheller, M., Wang, S.H., Reina, G.A., Foley, P., Gruzdev, A., Karkada, D., Davatzikos, C., et al.: Federated learning enables big data for rare cancer boundary detection. Nature Communications 13(1), 7346 (2022)

  6. [14]

    Physics in Medicine & Biology67(20), 204002 (2022)

    Pati, S., Baid, U., Edwards, B., Sheller, M.J., Foley, P., Reina, G.A., Thakur, S., Sako, C., Bilello, M., Davatzikos, C., et al.: The federated tumor segmentation (FeTS) tool: an open-source solution to further solid tumor research. Physics in Medicine & Biology67(20), 204002 (2022)

  7. [15]

    In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition

    Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., Xu, D., Nath, V., Hatamizadeh, A.: Self-supervised pre-training of swin transformers for 3D med- ical image analysis. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 20730–20740 (2022)

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.