REVIEW 3 major objections 5 minor 26 references
TextBraTS: Text-Guided Volumetric Brain Tumor Segmentation with Innovative Dataset Development and Fusion Module Exploration
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Adding volume-level radiological text reports to MRI input improves brain tumor segmentation beyond image-only baselines, reaching an average Dice of 85.3.
desk verdict The TextBraTS dataset is a real contribution, but the reported segmentation gains come from test-time leakage: the text reports are written from the ground-truth masks, so the paper overstates what text-guided fusion achieves. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the bidirectional cross-attention fusion block: a text-to-image attention pass (text as query, image as key and value) followed by an image-to-text pass (image as query, refined text-guided features as key and value), with normalization layers in between. This two-step mechanism lets the textual report reshape the image feature map before decoding. Equally central is the templated report formulation, where location phrases and feature phrases are presented in a fixed structure; the ablations show these templated inputs, rather than raw text, are what make the fusion effective.
What would settle it
Re-run the exact test procedure after regenerating the reports from the MRI images alone, without seeing the segmentation masks, either by prompting the multimodal model directly on the 3D Flair volumes or by having radiologists write reports blinded to the labels; if the average Dice advantage over the image-only baseline disappears or shrinks below the t-test threshold, the gain is attributable to mask-derived text rather than to text-guided fusion.
Extended reading notes
Core claim
The paper's central claim is that a text-guided segmentation network can outperform vision-only methods on volumetric brain tumor segmentation when trained and tested with paired radiological reports. The authors introduce TextBraTS, which they describe as the first publicly available volume-level text-image brain MRI dataset, with 369 cases; each report is produced by a large multimodal model and then corrected by expert radiologists against the images and segmentation labels. The proposed architecture combines a transformer-based image encoder with a biomedical language model, and a bidirectional cross-attention fusion block first uses text to refine image features, then uses those refined features to re-attend to the image features, producing a fused representation fed to the decoder. Experiments report a mean Dice of 85.3% (ET 83.3, WT 89.9, TC 82.8) and a 95th-percentile Hausdorff distance of 5.13 mm, improving over all compared baselines, and a t-test (p=0.0077) against NestedFormer over ten runs. Ablations show that templated text containing both location and feature phrases outperforms raw or partial text, and that bidirectional cross-attention outperforms single-direction cross-attention and dot-sum fusion.
Load-bearing premise
The load-bearing premise is that the volume-level text reports used at test time, which were generated with access to the ground-truth tumor masks and then refined by experts against those labels (Section 2), are valid inputs for measuring segmentation accuracy; if they leak mask-derived location information, the reported gain over image-only models would be inflated.
Editorial extensions
If this is right
- If text-guided fusion holds, adding volume-level radiology reports to brain tumor segmentation yields a Dice gain of about 1.2 points on average and a substantially lower edge error (HD95 of 5.13 mm vs. 7.07 mm for the image-only SwinUNETR baseline), which matters for boundary-sensitive downstream tasks like surgical planning.
- TextBraTS provides a public benchmark with paired MRI volumes and expert-refined textual reports, enabling direct comparisons of text-image fusion strategies on volumetric brain tumor data.
- Fully templated reports containing both location and feature information are the recommended input format; raw text or partially templated reports measurably reduce the segmentation benefit.
- Bidirectional cross-attention is a better fusion strategy than one-directional cross-attention or dot-sum fusion, implying that the text and image branches both need to be iteratively aligned.
- The reported improvement is consistent across the three tumor sub-regions (whole tumor, enhancing tumor, tumor core), not just on one region.
Reading between the lines
- The evaluation protocol uses reports derived from ground-truth masks, so the reported gain likely reflects how well the model can read location information that is already implicit in the segmentation labels; a test with reports written from images alone would separate the value of genuine radiological description from label leakage.
- The bidirectional fusion design could transfer to other volumetric medical imaging tasks that have structured reports, such as chest CT or ultrasound, where location-feature templates are available.
- A direct extension would be to train the same image-only backbone with the identical 200-epoch schedule and report Dice; the paper compares against published baselines rather than an equally trained SwinUNETR without text, so part of the gain could be training configuration rather than text guidance.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces TextBraTS, a volume-level dataset pairing BraTS2020 MRI volumes with expert-refined textual reports, and proposes a text-guided 3D brain tumor segmentation model that fuses text and image features with bidirectional cross-attention. The authors report that their method achieves an average Dice of 85.3 on their test split, outperforming several image-only baselines (e.g., 84.1 for nnU-Net and NestedFormer), and they conclude that text guidance significantly improves segmentation accuracy. The central claim is that integrating textual reports with imaging data improves volumetric brain tumor segmentation.
Significance. If the claimed result were valid, the public TextBraTS dataset, code, and pretrained models would be a useful contribution to multimodal medical image segmentation, and the proposed fusion baseline could inform future work on text-guided volumetric segmentation. The authors deserve credit for releasing these resources. However, the experimental design makes the headline claim circular: the text reports were constructed from the ground-truth segmentation masks, so at test time the text input encodes the answer. The reported gains over image-only baselines are therefore evidence of target leakage rather than of effective cross-modal fusion. The dataset may still be valuable for other purposes, such as radiological report generation or segmentation under a protocol where reports are produced without access to the masks, but the central segmentation improvement claimed in the paper is not established by the current evidence.
major comments (3)
- [Section 4, Table 1] The text reports are generated directly from the ground-truth masks and then used as test-time inputs. Section 2 states that 'the corresponding tumor ground truth labels were sliced and converted into videos. These were then input into the GPT-4o model,' and that experts refined the reports 'based on the images and segmentation labels.' Section 4 then feeds these same reports into the model at test time. This is target leakage: the text is a compressed description of the target mask, so the model's task reduces to decoding the answer from the text rather than learning to segment from images. The Table 1 improvement over image-only baselines is therefore not evidence of multimodal fusion. The location-only row in Table 2 (Dice 84.6) nearly matching the full-text row (Dice 85.3) is consistent with this interpretation, since the location phrases explicitly encode tumor position from the ground truth.
- [Section 4, Table 1] The comparison with SOTA methods is not adequately controlled. The paper does not state whether the baselines (3D-UNet, nnU-Net, SegResNet, SwinUNETR, NestedFormer) were retrained on the same 220/55/94 split with the same fixed 128x128x128 resampling, the same checkpoint selection, and the same post-processing protocol. If the baseline numbers are taken from other publications or trained under different conditions, the comparison is not valid. The reported t-test (p=0.0077) is against NestedFormer only, and no confidence intervals are given for any method. Most importantly, there is no ablation of the proposed architecture without the text branch, which would be the necessary image-only control for measuring the contribution of text.
- [Section 3 and Table 3] The fusion-module ablations are confounded by the same leakage. Since every condition in Table 3 receives text derived from the ground-truth mask, the comparison among dot-sum, one-step cross-attention, and bidirectional cross-attention measures how well each module can decode the answer-bearing text, not how well it fuses independent radiological information. The conclusion that bidirectional cross-attention is 'effective' is thus not supported as a general claim about multimodal fusion. A valid evaluation would require text that is generated without access to the masks, or an explicit comparison where the text is a genuine radiological report available at inference time.
minor comments (5)
- [Abstract and Section 3] The abstract refers to a 'sequential cross-attention method,' while Section 3 presents a 'bidirectional cross-attention' mechanism; please align the terminology throughout the paper.
- [Tables 1 and 2] Table 2 has the typo 'Ablaion study' and Table 1 contains an extraneous space in 'T able'; these should be corrected.
- [Section 2, last paragraph] The sentence 'in the field of brain tumor segmentation, no similar text-image dataset and text-image multimodal segmentation methods' is incomplete; please revise to a full sentence.
- [Figure 3] The caption states that template words, location descriptions, and features are colored red, green, and blue, but the report text in the figure is not visibly color-coded in the manuscript; please colorize the example or adjust the caption.
- [Equations (4) and (6)] Equation (4) has a mismatched parenthesis in the Softmax expression, and the shape of the intermediate feature f'_i (token_num x 768) versus the claimed output f_joint (H/32 x W/32 x D/32 x 768) is not explicitly derived; please clarify the tensor dimensions.
Circularity Check
Test-time text reports are generated from ground-truth masks, so the headline segmentation gain reflects target leakage rather than independent cross-modal fusion.
-
self definitional
[Section 2 (TextBraTS Dataset Creation) and Section 4 (Table 1 comparison)]
"For each case, we sliced the 3D Flair images into individual frames and converted them into video format. Similarly, the corresponding tumor ground truth labels were sliced and converted into videos. These were then input into the GPT-4o model, guided by a carefully designed template prompt to generate analyses of the images. ... Subsequently, experts reviewed the generated textual reports based on the images and segmentation labels, correcting the content to produce a final overall description."
The text modality used as the conditioning input at test time is constructed directly from the target segmentation labels. The reported comparison in Table 1 therefore does not measure whether independent radiological text improves segmentation; it measures how well the model decodes a textual encoding of the ground-truth mask. The location-only ablation in Table 2 (84.6 average Dice vs. 85.3 for full text) shows that location phrases alone nearly reproduce the full gain, consistent with the text acting as a compressed mask annotation. Thus the central claim of improved segmentation from text guidance is forced by the data construction.
full rationale
The paper's derivation chain for the central claim—text-guided fusion improves brain tumor segmentation—breaks at the dataset-construction step. Section 2 states explicitly that GPT-4o generated pseudo-reports from ground-truth tumor labels and that experts corrected the reports using the segmentation labels. Section 4 then feeds those same reports into the model at test time. Since the text input is a function of the target mask, the model's improvement over image-only baselines is largely attributable to target leakage, not to a generalizable multimodal fusion advantage. The location-only ablation reinforces this reading: removing feature descriptions still leaves 84.6 Dice versus 85.3 with full templated text, indicating that location phrasing alone carries most of the mask information. This is a substantial circularity in the evaluation of the proposed model. The dataset itself may still be a useful resource for other purposes, and the fusion module could be sound, but the headline experimental evidence does not support the stated conclusion. No other load-bearing circular steps were found: the authors' self-citations (e.g., refs. 22, 23, 25) are not used to justify the central fusion claim, and no uniqueness theorem or ansatz is imported from prior author work in a way that determines the result. The dominant issue is the self-definitional link between the test-time text input and the target masks.
Assumptions & free parameters
free parameters (1)
- token_num =
110
assumptions (4)
- domain assumption GPT-4o pseudo-reports refined by expert radiologists faithfully represent tumor location, signal, edema, necrosis, and mass effect as stated in the reports.
- domain assumption Ground truth segmentation labels of BraTS2020 are correct.
- ad hoc to paper Text generated from ground truth masks is a valid input for evaluating segmentation at test time.
- domain assumption The random split is representative and the baseline results in Table 1 are comparable.
Cite this review
Pith. "Pith review of TextBraTS: Text-Guided Volumetric Brain Tumor Segmentation with Innovative Dataset Development and Fusion Module Exploration." pith.science (2026). https://pith.science/paper/4YGQTLCQ
@misc{pith2026250616784,
author = {Pith},
title = {Pith review of: TextBraTS: Text-Guided Volumetric Brain Tumor Segmentation with Innovative Dataset Development and Fusion Module Exploration},
year = {2026},
howpublished = {\url{https://pith.science/paper/4YGQTLCQ}},
note = {Machine review of arXiv:2506.16784}
}
read the original abstract
Deep learning has demonstrated remarkable success in medical image segmentation and computer-aided diagnosis. In particular, numerous advanced methods have achieved state-of-the-art performance in brain tumor segmentation from MRI scans. While recent studies in other medical imaging domains have revealed that integrating textual reports with visual data can enhance segmentation accuracy, the field of brain tumor analysis lacks a comprehensive dataset that combines radiological images with corresponding textual annotations. This limitation has hindered the exploration of multimodal approaches that leverage both imaging and textual data. To bridge this critical gap, we introduce the TextBraTS dataset, the first publicly available volume-level multimodal dataset that contains paired MRI volumes and rich textual annotations, derived from the widely adopted BraTS2020 benchmark. Building upon this novel dataset, we propose a novel baseline framework and sequential cross-attention method for text-guided volumetric medical image segmentation. Through extensive experiments with various text-image fusion strategies and templated text formulations, our approach demonstrates significant improvements in brain tumor segmentation accuracy, offering valuable insights into effective multimodal integration techniques. Our dataset, implementation code, and pre-trained models are publicly available at https://github.com/Jupitern52/TextBraTS.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Scientific data 4(1), 1–13 (2017)
Bakas, S., Akbari, H., Sotiras, A., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data 4(1), 1–13 (2017)
work page 2017
-
[2]
Cardoso, M.J., Li, W., Brown, R., et al.: MONAI: An open-source framework for deep learning in healthcare (Nov 2022).https://doi.org/10.48550/arXiv.2211. 02701
-
[3]
In: Linguraru, M.G., Dou, Q., Fer- agen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A
Chai, S., Jain, R.K., Mo, S., et al.: A Novel Adaptive Hypergraph Neural Network for Enhancing Medical Image Segmentation. In: Linguraru, M.G., Dou, Q., Fer- agen, A., Giannarou, S., Glocker, B., Lekadir, K., Schnabel, J.A. (eds.) Medical ImageComputingandComputerAssistedIntervention–MICCAI2024,vol.15009, pp. 23–33. Cham (2024).https://doi.org/10.1007/978...
-
[4]
In: 2022 IEEE International Conference on Image Processing (ICIP)
Degerli, A., Kiranyaz, S., Chowdhury, M.E., et al.: Osegnet: Operational segmen- tation network for covid-19 detection using chest x-ray images. In: 2022 IEEE International Conference on Image Processing (ICIP). pp. 2306–2310 (2022)
work page 2022
-
[5]
Dice,L.R.:Measuresoftheamountofecologicassociationbetweenspecies.Ecology 26(3), 297–302 (1945)
1945
-
[6]
Hatamizadeh, A., Nath, V., Tang, Y., et al.: Swin UNETR: Swin Transform- ers for Semantic Segmentation of Brain Tumors in MRI Images. In: Crimi, A., Bakas, S. (eds.) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, vol. 12962, pp. 272–284. Cham (2022).https://doi.org/10.1007/ 978-3-031-08999-2_22
work page 2022
-
[7]
Hsu, C., Chang, C., Chen, T.W., et al.: Brain Tumor Segmentation (BraTS) Challenge Short Paper: Improving Three-Dimensional Brain Tumor Segmenta- tion Using SegResnet and Hybrid Boundary-Dice Loss. In: Crimi, A., Bakas, S. (eds.) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, vol. 12963, pp. 334–344. Cham (2022). https://doi...
work page 2022
-
[8]
Hu, J., Li, Y., Sun, H., et al.: LGA: A Language Guide Adapter for Advanc- ing the SAM Model’s Capabilities in Medical Image Segmentation. In: Lingu- raru, M.G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., Schn- abel, J.A. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, vol. 15012, pp. 610–620. Cham (202...
work page 2024
Show all 26 references
-
[9]
IEEE Transactions on pattern analysis and machine intelligence 15(9), 850–863 (1993)
Huttenlocher, D.P., Klanderman, G.A., Rucklidge, W.J.: Comparing images us- ing the Hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence 15(9), 850–863 (1993)
1993
-
[10]
Nature methods 18(2), 203–211 (2021)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
2021
-
[11]
Bioinformatics36(4), 1234–1240 (2020)
Lee, J., Yoon, W., Kim, S., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics36(4), 1234–1240 (2020)
2020
-
[12]
IEEE transactions on medical imaging (2023) 10 Shi et al
Li, Z., Li, Y., Li, Q., et al.: Lvit: language meets vision transformer in medical image segmentation. IEEE transactions on medical imaging (2023) 10 Shi et al
2023
-
[13]
arXiv preprint arXiv:1711.05101 (2017)
Loshchilov, I.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
2017 arXiv
-
[14]
Neuro-oncology23(8), 1231–1251 (2021)
Louis, D.N., Perry, A., Wesseling, P., et al.: The 2021 WHO classification of tumors of the central nervous system: a summary. Neuro-oncology23(8), 1231–1251 (2021)
2021
-
[15]
IEEE transactions on medical imaging34(10), 1993–2024 (2014)
Menze, B.H., Jakab, A., Bauer, S., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging34(10), 1993–2024 (2014)
2014
- [16]
-
[17]
OpenAI: Hello GPT-4o. Tech. rep. (2024)
2024
-
[18]
Paszke, A., Gross, S., Chintala, S., et al.: Automatic differentiation in pytorch (2017)
2017
-
[19]
WSEAS Transactions on Circuits and Systems8(7), 579–588 (2009)
Popescu, M.C., Balas, V.E., Perescu-Popescu, L., et al.: Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems8(7), 579–588 (2009)
2009
-
[20]
Journal of the American College of Radiology 4(5), 313–319 (2007)
Reiner, B.I., Knight, N., Siegel, E.L.: Radiology reporting, past, present, and fu- ture: the radiologist’s perspective. Journal of the American College of Radiology 4(5), 313–319 (2007)
2007
-
[21]
In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomed- ical Image Segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MIC- CAI 2015, vol. 9351, pp. 234–241. Cham (2...
2015
-
[22]
Journal of Image and Graphics11(4) (2023)
Shi, X., Li, Y., Chen, Y.W., et al.: An intra-and inter-modality fusion model with invariant-and specific-constraints using mr images for prediction of glioma isoci- trate dehydrogenase mutation status. Journal of Image and Graphics11(4) (2023)
2023
-
[23]
ACM Transactions on Computing for Healthcare (2025)
Xiaoyu, S., Rahul, Kumar, J., Yinhao, L., et al.: Multi-modal medical sam: An adaptationmethodofsegmentanythingmodel(sam)forgliomasegmentationusing multi-modal mr images. ACM Transactions on Computing for Healthcare (2025)
2025
-
[24]
In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S
Xing, Z., Yu, L., Wan, L., et al.: NestedFormer: Nested Modality-Aware Trans- former for Brain Tumor Segmentation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) Medical Image Computing and Computer Assisted In- tervention – MICCAI 2022, vol. 13435, pp. 140–...
2022
-
[25]
The Visual Computer39(6), 2367– 2379 (2023)
Zhang, X., Shi, X., Iwamoto, Y., et al.: Idh mutation status prediction by a ra- diomics associated modality attention network. The Visual Computer39(6), 2367– 2379 (2023)
2023
-
[26]
Medical image analysis43, 98–111 (2018)
Zhao, X., Wu, Y., Song, G., et al.: A deep learning model integrating fcnns and crfs for brain tumor segmentation. Medical image analysis43, 98–111 (2018)
2018
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.