REVIEW 4 major objections 5 minor 27 references
DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Explicitly disentangling MRI anatomy from scan-specific style improves brain tumor segmentation when modalities are missing.
desk verdict New disentanglement idea with a solid ablation, but the SOTA comparison rests on quoted baselines and needs protocol-matched reruns. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is bidirectional contrastive learning on full 3D image feature maps. The anatomical contrastive loss $\mathcal{L}_{\mathrm{ana}}$ uses channel-wise SSIM to pull together the anatomical representations $a^i_j$ of the same subject across modalities and push apart representations from different subjects; the modality contrastive loss $\mathcal{L}_{\mathrm{mod}}$ uses cosine similarity on the 8-element modality codes $m^i_j$ to pull together codes of the same sequence type across subjects and push apart codes of different sequences. A reconstruction loss with Bernoulli modality dropout keeps both subspaces informative enough to re-render any input modality, and a weight-shared decoder regularizer forces every single-modality anatomical map to predict the segmentation alone. Together these make the fused anatomical representation $z_i$ robust to arbitrary subsets of missing modalities.
What would settle it
Retrain RobustSeg, RFNet, mmFormer, M3AE, and DC-Seg on the same BraTS 2020 split with identical preprocessing, patch size, augmentation, epochs, and evaluation script; if DC-Seg no longer leads on most of the 15 modality combinations in whole-tumor, core, and enhancing Dice, the claimed superiority is an artifact of comparison protocol. A cheaper version is to retrain just RobustSeg and RFNet under DC-Seg's exact training protocol and compare with Table 1.
Extended reading notes
Core claim
The paper's central claim is that bidirectional contrastive disentanglement is a better inductive bias for incomplete multimodal segmentation than a shared latent space. Anatomical contrastive learning reinforces that the same anatomy viewed through T1, T1ce, T2, and FLAIR lands on the same feature map, while modality contrastive learning enforces that each sequence's style code is consistent across subjects. The reconstruction branch keeps both subspaces informative enough to regenerate any input modality, and the segmentation regularizer stops the fusion path from leaning on the discriminative sequences T1ce and FLAIR. The reported result is an average Dice of 87.54 on whole tumor, 79.63 on tumor core, and 65.00 on enhancing tumor over all 15 missing-modality combinations, close to the dedicated group-self-support baseline that trains 15 separate models, and a 75.43 average Dice on the two-modality white matter hyperintensity task.
Load-bearing premise
The reported advantage over earlier methods assumes that Dice values quoted from their papers are directly comparable to DC-Seg's own runs because all use the same data split as RFNet; different preprocessing, augmentation, training budgets, or evaluation code would make the gaps protocol effects rather than method effects.
Editorial extensions
If this is right
- A single DC-Seg model can replace the 15 dedicated models of the group-self-support approach with roughly unchanged whole-tumor Dice, cutting memory and training cost for missing-modality deployment.
- The per-modality regularizer makes each sequence alone produce a usable segmentation, so losing discriminative modalities such as T1ce or FLAIR causes graceful rather than catastrophic degradation.
- The method needs no generative model to synthesize absent sequences at inference time, removing the extra training costs of modality-synthesis pipelines.
- The same loss design transfers from four-modal tumor imaging to two-modal white matter hyperintensity imaging, suggesting the disentanglement is not tuned to a particular set of MR contrasts.
Reading between the lines
- A matched-protocol rerun under identical preprocessing, augmentation, training schedule, and evaluation code would determine how much of the reported gap over the prior baselines is the method itself rather than inherited published numbers; the paper's comparison quotes previous results directly.
- The same anatomical-versus-modality split could apply to other paired multimodal inputs where one channel carries style, such as CT and PET or RGB and depth, provided a reconstruction branch can be defined.
- Varying the dimensionality of the 8-element modality code while holding everything else fixed would test whether the bottleneck is what enforces the separation between anatomical and modality-specific information.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents DC-Seg, a missing-modality brain tumor segmentation method that disentangles each input modality into a modality-invariant anatomical representation and a modality-specific representation. It trains these with a sigmoid-based anatomical contrastive loss (Eq. 1), a modality contrastive loss (Eq. 4), a reconstruction loss (Eq. 5), and a per-modality segmentation regularizer (Eq. 7). Experiments on BraTS 2020 and a private WMH dataset compare against RobustSeg, RFNet, mmFormer, M3AE, and GSS. The component ablation in Table 2 supports the usefulness of each loss term, but the state-of-the-art comparisons in Table 1 and Table 3 are based on directly quoted baseline numbers rather than protocol-matched reruns, and no variance or significance information is reported.
Significance. If the empirical claims hold, the paper contributes a clean and plausible combination of bidirectional contrastive disentanglement with reconstruction and a segmentation regularizer. The code release and the component-wise ablation are strengths, and the idea of aligning anatomical representations across modalities while separating modality-specific ones is a reasonable inductive bias for missing-modality segmentation. However, the headline claim that DC-Seg significantly outperforms state-of-the-art methods currently rests on cross-paper baseline comparisons, so the scientific significance cannot be fully assessed until the comparison protocol is controlled and statistical variability is reported.
major comments (4)
- [Section 3, Table 1] The claim that DC-Seg 'significantly outperforms the state-of-the-art methods' is based on directly referencing baseline Dice values from prior papers under the assertion 'we use the same data split as [5]'. A shared split label does not imply equal preprocessing, patch size, training budget, augmentation, or evaluation protocol. The full-modality Complete row illustrates the risk: RFNet is quoted at 91.11 and DC-Seg at 90.95, and the average Complete gap over RFNet is only 0.56 Dice. The same protocol concern applies to the GSS numbers quoted in the text (Complete 87.33, Core 79.38, Enhancing 65.54), which are not even shown in Table 1. Unless the baselines are rerun under the authors' protocol or the comparisons are otherwise demonstrated to be protocol-matched, the headline SOTA claim is not supported.
- [Section 3, Table 3] The WMH comparison does not state whether RobustSeg, RFNet, and mmFormer were retrained on the private SAHZU-WMH split or whether their numbers are quoted from BraTS experiments. WMH segmentation involves different modalities and lesion characteristics, so either choice needs explicit protocol details (training split, preprocessing, hyperparameters, augmentation) before the generalization claim in Section 3 can be evaluated.
- [Section 3, Tables 1 and 2] No standard deviations, confidence intervals, or significance tests are reported anywhere. In Table 2, the ablation gaps (e.g., Complete 86.31 without anatomical contrastive learning versus 87.54 for the full model) may be within run-to-run noise, especially with a batch size of 2 and 500 epochs. The paper should report multiple runs and statistical testing before concluding that each component contributes and that the method is superior to the baselines.
- [Section 2.1, Eqs. (1), (3), (5)] The central disentanglement loss is not fully reproducible because several key hyperparameters are unspecified: the temperature t in Eq. (1) and Eq. (4), the SSIM constants C1 and C2 in Eq. (3), and the modality-dropout probability for the Bernoulli indicator delta in Eq. (5). Section 3 gives only alpha = 0.4. These values should be reported in the implementation details.
minor comments (5)
- [Section 2.1, Eq. (3)] The SSIM computation on 3D feature maps needs the window size, stride, and assumed data range to be specified, since these choices affect the contrastive loss values and hence the learned representations.
- [Section 2.1] The phrase '8-bit vector' for the modality representation is confusing; if the intended meaning is an 8-dimensional vector, the notation m_i^j in R^C should be adjusted accordingly.
- [Section 3] The augmentation description 'Random flips, cropping, and intensity shifts' does not give ranges for these augmentations; specifying them would improve reproducibility.
- [Section 3, Figure 2] MedSAM is compared only qualitatively with a bounding-box prompt; reporting a quantitative Dice score for MedSAM would make the dismissal of the foundation-model baseline more convincing.
- [Section 3] The text reports GSS numbers in prose rather than in Table 1; adding GSS as a separate row or table would clarify the comparison and make the protocol details easier to verify.
Circularity Check
No significant circularity: all losses are supervised by external MRI data and ground-truth labels, and the SOTA comparison relies on independent published baselines rather than on the model's own fitted outputs.
full rationale
The paper's derivation chain is a training objective (Eqs. 1-9) optimized on BraTS 2020 and a private WMH dataset and then evaluated on held-out scans. Anatomical contrastive learning aligns features across modalities of the same subject, modality contrastive learning separates modality-specific codes, the reconstruction loss supervises decoded images against original inputs, and both the segmentation regularizer and main segmentation loss use ground-truth tumor masks. No equation defines a predicted quantity in terms of its own output, and no parameter is fitted to a subset of data and then reported as a prediction for a closely related quantity. The disentanglement assumption is attributed to external prior work [10], not to the authors' own prior results, so it does not create a self-citation chain. The Table 1 claim 'we use the same data split as in [5] and directly reference the results' is a protocol-matching weakness that could affect the empirical comparison, but it is not circularity: the quoted baseline Dice values are independent external evidence rather than outputs of the present method. The ablation study in Table 2 evaluates each component with external labels and reports no fitted-value-as-prediction behavior. No circular step is exhibited.
Assumptions & free parameters
free parameters (4)
- temperature t in contrastive losses =
not reported
- disentanglement tradeoff alpha =
0.4
- modality dropout probability for delta =
not reported
- SSIM constants C1 and C2 =
not reported
assumptions (5)
- domain assumption Images from BraTS 2020 and SAHZU-WMH are co-registered to a common anatomical template (Section 3).
- domain assumption A fused anatomical representation z_i is sufficient to predict the tumor label (Eq. 8; Eq. 5 uses z_i for reconstruction).
- domain assumption For successful disentanglement, the anatomical representation is re-renderable with any modality code [10], Section 2.1.
- standard math The sigmoid contrastive loss formulation of [23] is used as a standard objective without modification.
- standard math Dice and weighted cross-entropy are standard segmentation losses.
Cite this review
Pith. "Pith review of DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities." pith.science (2026). https://pith.science/paper/5K3Z6UU5
@misc{pith2026250511921,
author = {Pith},
title = {Pith review of: DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities},
year = {2026},
howpublished = {\url{https://pith.science/paper/5K3Z6UU5}},
note = {Machine review of arXiv:2505.11921}
}
read the original abstract
Accurate segmentation of brain images typically requires the integration of complementary information from multiple image modalities. However, clinical data for all modalities may not be available for every patient, creating a significant challenge. To address this, previous studies encode multiple modalities into a shared latent space. While somewhat effective, it remains suboptimal, as each modality contains distinct and valuable information. In this study, we propose DC-Seg (Disentangled Contrastive Learning for Segmentation), a new method that explicitly disentangles images into modality-invariant anatomical representation and modality-specific representation, by using anatomical contrastive learning and modality contrastive learning respectively. This solution improves the separation of anatomical and modality-specific features by considering the modality gaps, leading to more robust representations. Furthermore, we introduce a segmentation-based regularizer that enhances the model's robustness to missing modalities. Extensive experiments on the BraTS 2020 and a private white matter hyperintensity(WMH) segmentation dataset demonstrate that DC-Seg outperforms state-of-the-art methods in handling incomplete multimodal brain tumor segmentation tasks with varying missing modalities, while also demonstrate strong generalizability in WMH segmentation. The code is available at https://github.com/CuCl-2/DC-Seg.
Figures
Reference graph
Works this paper leans on
-
[5]
In: Proceedings of the IEEE/CVF inter- national conference on computer vision
Ding, Y., Yu, X., Yang, Y.: Rfnet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation. In: Proceedings of the IEEE/CVF inter- national conference on computer vision. pp. 3975–3984 (2021)
work page 2021
-
[1]
In: International Conference on Medical Imaging with Deep Learning
Azad, R., Khosravi, N., Merhof, D.: Smu-net: Style matching u-net for brain tumor segmentation with missing modalities. In: International Conference on Medical Imaging with Deep Learning. pp. 48–62. PMLR (2022)
2022
-
[2]
Chen, C., Liu, X., Ding, M., Zheng, J., Li, J.: 3d dilated multi-fiber network for real-time brain tumor segmentation in mri. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part III 22. pp. 184–192. Springer (2019)
work page 2019
-
[3]
Chen, C., Dou, Q., Jin, Y., Chen, H., Qin, J., Heng, P.A.: Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion. In: Medical Im- age Computing and Computer Assisted Intervention–MICCAI 2019: 22nd Inter- national Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part III
work page 2019
-
[4]
IEEE transactions on medical imag- ing 41(3), 621–632 (2021)
Chen, C., Dou, Q., Jin, Y., Liu, Q., Heng, P.A.: Learning with privileged multi- modal knowledge for unimodal segmentation. IEEE transactions on medical imag- ing 41(3), 621–632 (2021)
work page 2021
-
[6]
Fidon, L., Li, W., Garcia-Peraza-Herrera, L.C., Ekanayake, J., Kitchen, N., Ourselin, S., Vercauteren, T.: Scalable multimodal convolutional networks for brain tumour segmentation. In: Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part II...
work page 2017
-
[7]
Havaei, M., Guizard, N., Chapados, N., Bengio, Y.: Hemis: Hetero-modal image segmentation. In: Medical Image Computing and Computer-Assisted Intervention– MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. pp. 469–477. Springer (2016)
work page 2016
-
[8]
Hu, M., Maillard, M., Zhang, Y., Ciceri, T., La Barbera, G., Bloch, I., Gori, P.: Knowledge distillation from multi-modal to mono-modal segmentation networks. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23. pp. 772–781. Springer (2020)
work page 2020
Show all 27 references
-
[9]
Medical image analysis36, 61–78 (2017)
Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical image analysis36, 61–78 (2017)
2017
-
[10]
Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M., Yang, M.H.: Diverse image-to- imagetranslationviadisentangledrepresentations.In:ProceedingsoftheEuropean conference on computer vision (ECCV). pp. 35–51 (2018)
2018
-
[11]
In: Pro- ceedings of the AAAI Conference on Artificial Intelligence
Liu, H., Wei, D., Lu, D., Sun, J., Wang, L., Zheng, Y.: M3ae: multimodal repre- sentation learning for brain tumor segmentation with missing modalities. In: Pro- ceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1657–1665 (2023)
2023
-
[12]
Nature Communications15(1), 654 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)
2024
-
[13]
IEEE Transactions on Medical Imaging (2024) 10 H
Meng,X.,Sun,K.,Xu,J.,He,X.,Shen,D.:Multi-modalmodality-maskeddiffusion network for brain mri synthesis with random modality missing. IEEE Transactions on Medical Imaging (2024) 10 H. Li et al
2024
-
[14]
IEEE transactions on medical imaging 34(10), 1993–2024 (2014)
Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34(10), 1993–2024 (2014)
2014
-
[15]
In: Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27
Ouyang, J., Adeli, E., Pohl, K.M., Zhao, Q., Zaharchuk, G.: Representation dis- entanglement for multi-modal brain mri analysis. In: Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27. pp. 3...
2021
-
[16]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Qiu, Y., Chen, D., Yao, H., Xu, Y., Wang, Z.: Scratch each other’s back: Incom- plete multi-modal brain tumor segmentation via category aware group self-support learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21317–21326 (2023)
2023
-
[17]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
2021
-
[18]
IEEE transactions on medical imaging40(4), 1113–1122 (2020)
Shen, L., Zhu, W., Wang, X., Xing, L., Pauly, J.M., Turkbey, B., Harmon, S.A., Sanford, T.H., Mehralivand, S., Choyke, P.L., et al.: Multi-domain image comple- tion for random missing input data. IEEE transactions on medical imaging40(4), 1113–1122 (2020)
2020
-
[19]
In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition
Tseng, K.L., Lin, Y.L., Hsu, W., Huang, C.Y.: Joint sequence learning and cross- modality convolution for 3d biomedical segmentation. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6393–6400 (2017)
2017
-
[20]
Van Tulder, G., de Bruijne, M.: Why does synthesized data improve multi-sequence classification? In: Medical Image Computing and Computer-Assisted Intervention– MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18. pp. 531–538....
2015
-
[21]
Wang, Y., Zhang, Y., Liu, Y., Lin, Z., Tian, J., Zhong, C., Shi, Z., Fan, J., He, Z.: Acn: adversarial co-training network for brain tumor segmentation with missing modalities. In: Medical Image Computing and Computer Assisted Intervention– MICCAI 2021: 24th International Conf...
2021
-
[22]
pp. 447–456. Springer (2019)
2019
-
[23]
IEEE Transactions on Medical Imaging41(10), 2953–2964 (2022)
Yang, Q., Guo, X., Chen, Z., Woo, P.Y., Yuan, Y.: D 2-net: Dual disentanglement network for brain tumor segmentation with missing modalities. IEEE Transactions on Medical Imaging41(10), 2953–2964 (2022)
2022
-
[24]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11975–11986 (2023)
2023
-
[25]
IEEE Transactions on Image Processing29, 9032–9043 (2020)
Zhang, D., Huang, G., Zhang, Q., Han, J., Han, J., Wang, Y., Yu, Y.: Exploring task structure for brain tumor segmentation from multi-modality mr images. IEEE Transactions on Image Processing29, 9032–9043 (2020)
2020
-
[26]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Zhang, Y., He, N., Yang, J., Li, Y., Wei, D., Huang, Y., Zhang, Y., He, Z., Zheng, Y.: mmformer: Multimodal medical transformer for incomplete multimodal learn- ing of brain tumor segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Inter...
2022
-
[27]
In: Medical Image Comput- ing and Computer Assisted Intervention–MICCAI 2018: 21st International Con- ference, Granada, Spain, September 16-20, 2018, Proceedings, Part III 11
Zhou, C., Ding, C., Lu, Z., Wang, X., Tao, D.: One-pass multi-task convolutional neural networks for efficient brain tumor segmentation. In: Medical Image Comput- ing and Computer Assisted Intervention–MICCAI 2018: 21st International Con- ference, Granada, Spain, September 16-...
2018
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.