REVIEW 3 major objections 4 minor 30 references
Learning Segmentation from Radiology Reports
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Radiology reports can directly supervise tumor segmentation, lifting F1 by up to 16 percentage points.
desk verdict Novel report-to-voxel supervision with a real external Dice gain, but the headline numbers are confounded by adding thousands of unlabeled CTs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the pair of report-supervision losses. Volume Loss is applied as deep supervision: it sums predicted tumor probabilities inside pre-saved organ or sub-segment masks, multiplies by voxel volume to obtain a segmented volume, and penalizes the relative difference from the report-estimated volume with a 10% tolerance and a background cross-entropy term. Ball Loss is applied to the final layer: a fixed spherical kernel matching the reported tumor diameter is convolved over the output to find the highest-probability ball, the top-N voxels inside it (N derived from the report-estimated volume) are maximized, and all voxels not assigned to any reported tumor are minimized, repeating from largest to smallest tumor. Together these losses transform text-wise report information into voxel-wise labels.
What would settle it
Train R-Super on a dataset where ground-truth masks and paired reports are both available and compare report-estimated volumes against mask-derived volumes; if the losses improve segmentation only when the two agree, then on a cohort where reported diameters systematically underestimate true tumor volume (for example, single-slice measurements of elongated tumors), R-Super should underperform mask-only training. Alternatively, construct test reports that omit one of several visible tumors: if R-Super's false negatives match the omitted tumor, that confirms the losses enforce report completeness rather than true tumor burden.
Extended reading notes
Core claim
The central claim is that radiology reports contain enough quantitative detail—tumor count, organ or sub-segment location, and one-to-three diameters—to act as voxel-level training signal for tumor segmentation. The Volume Loss compares, per organ or sub-segment, the total segmented tumor volume against a volume estimated from report diameters using ball or ellipsoid formulas, with a 10% tolerance and a background penalty; the Ball Loss localizes each reported tumor with a spherical convolution whose kernel matches the reported diameter, then maximizes the top-N most probable voxels inside the highest-probability ball and minimizes all unassigned voxels, iterating from largest to smallest tumor. In internal and external validation, R-Super surpassed mask-only segmentation and four report-using baselines, with F1 gains up to 16 points with 344 masks, about 10 points with 50 masks, and 4.3 points with 1.7K masks, including gains for small (diameter at most 2 cm) and large tumors.
Load-bearing premise
The report-derived tumor attributes—count, location, and diameters converted to ellipsoid volumes—are accurate and complete enough to serve as voxel-level supervision, and the paper excludes the small fraction (under 11%) of reported tumors without size rather than modeling them.
Editorial extensions
If this is right
- Hospitals can contribute existing CT-report archives to segmentation training without manual mask annotation, increasing data scale and diversity across centers, scanners, and contrast phases.
- Tumor types with scarce public masks, including pancreatic and kidney tumors, can gain large F1 improvements from report supervision even when only 50 masks exist.
- The method is architecture-agnostic and does not require perfect report-derived labels: the tolerance in Volume Loss and the dynamic mask refinement in Ball Loss accommodate imperfect diameter estimates.
- Gains persist when mask counts are large (1.7K), so report supervision complements rather than replaces manual masks.
- External validation on an unseen hospital shows the report-supervised model generalizes across institutions.
Reading between the lines
- If report diameters are systematically recorded from a single axial slice, the ellipsoid volume estimates may be biased low, and the losses could steer the segmenter toward under-segmentation; a calibration study comparing report-derived volumes with mask-derived volumes on a paired dataset would test this.
- The LLM extraction step is a potential bottleneck: prompt errors in diameter or location propagate directly into the losses, so end-to-end performance is upper-bounded by extraction accuracy; perturbing extracted attributes in a sensitivity analysis would quantify this.
- The approach may transfer to other imaging modalities and lesion types where reports state dimensions and location, such as MRI or ultrasound, and to other anatomical regions beyond the abdomen.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces R-Super, a training loss that converts radiology reports into voxel-level supervision for tumor segmentation. An LLM extracts tumor count, organ/sub-segment location, and diameters from reports; the Volume Loss and Ball Loss enforce agreement between the predicted segmentation and tumor volumes, locations, and counts estimated from reports. Training combines public CT-mask pairs (AbdomenAtlas) with a large private UCSF CT-report dataset. The method is evaluated for pancreatic and kidney tumor segmentation, internally on UCSF-Test (no masks, so detection metrics) and externally on JHH-Test (with manual masks). R-Super reports F1 gains up to +16% internally and Dice gains of +8 to +11 points externally over mask-only training, and it outperforms several report-based and self-supervised baselines.
Significance. If the central claim holds, the contribution is significant: it offers a practical way to scale tumor segmentation by exploiting routinely available radiology reports, and it includes external validation with real masks. The paper provides ablations of the two proposed losses, comparisons with five state-of-the-art methods, and public code. The main caveats are that the few-mask result lacks a no-report control using the same extra CT volumes, and the internal headline metric measures tumor detection rather than voxel-level segmentation.
major comments (3)
- [Section 3, Fig. 4, Table 2] The few-mask experiments (50 masks) compare R-Super trained with 2.2K/2.7K additional CT-report pairs against a segmentation baseline trained with only 50 masks. No baseline uses the same additional CT volumes without report supervision at the 50-mask setting; Models Genesis, the only method that learns from CTs without reports, is reported only at 344/1.7K masks. Consequently, the +9.7% F1 gain at 50 masks could be caused by the additional CT volumes rather than by the report-derived losses. Please add a no-report baseline at 50 masks using the same UCSF-Train CTs (e.g., Models Genesis pretraining or a pseudo-label semi-supervised method), or clearly restrict the few-mask claim to acknowledge this confound.
- [Abstract, Section 3, Table 2] The headline '+16% F1' is measured on UCSF-Test, which has no segmentation masks, so F1 and AUC there evaluate tumor detection, not voxel-level segmentation. The abstract's phrase 'strongly improved tumor segmentation in internal and external validation' conflates detection and segmentation. Please rephrase the abstract to state that internal gains are in detection F1 while segmentation gains are demonstrated on JHH-Test via DSC/NSD, or obtain manual masks on a subset of UCSF-Test and report a segmentation metric there.
- [Training R-Super, Table 1] The comparison to Models Genesis is intended to control for learning from CT volumes without reports, but the paper does not state explicitly whether the Models Genesis baseline was pre-trained on the same UCSF-Train CT volumes used by R-Super. Since the central claim is that report text, not merely extra CT data, drives the improvement, please specify the exact unlabeled CT data used for the Models Genesis baseline and, if it does not include UCSF-Train, add a control that does.
minor comments (4)
- [Figure 1] The right panel of Figure 1 would be easier to read with labeled axes (for example, segmented volume V_s on the x-axis and loss value on the y-axis) and a clear indication of the tolerance region.
- [Section 2.1 and Section 3] No sensitivity analysis is provided for the tolerance parameter τ, the stabilization constant E, or the relative loss weight of 0.1; a brief ablation or discussion of these choices would help readers understand the robustness of the method to hyperparameter settings.
- [Table 2] The text reports that pancreas sub-segment masks improve R-Super and gives numerical results without them, but this comparison is not shown as a table row; consider adding a row or a supplementary table for this ablation.
- [Section 2] The model name is written as 'LLama' in the text; it should be 'LLaMA' for consistency with the reference.
Circularity Check
No significant circularity; external manual-mask validation independently supports the central claim.
full rationale
The R-Super derivation chain is not circular. Reports are parsed into tumor count, location, and diameters; Volume Loss (Eq. 2) and Ball Loss then penalize segmentations that disagree with these report-derived attributes, and the resulting model is evaluated on two test sets. The external JHH-Test uses manual segmentation masks and reports DSC/NSD/F1, so the claim that report supervision improves tumor segmentation is tested against independent anatomical ground truth rather than against the training objective. The internal UCSF-Test has no masks (as the table note states: 'DSC and NSD are not available in UCSF-Test, it does not have masks'), so its F1/AUC are report-based; this is a proxy limitation, not a circular identity, because the UCSF-Test reports are held out and the model must generalize to new patients. The cited RadGPT parser accuracy and organ nnU-Net are component tools from overlapping authors, but the external JHH-Test improvements empirically validate that the report-supervision signal is informative, so these self-citations are not load-bearing in a circular way. A separate concern, that the comparison lacks a no-report control on the added UCSF-Train CTs, is a confound about attributing gains to report text versus extra data; it is a correctness risk, not a by-construction equivalence.
Assumptions & free parameters
free parameters (5)
- Volume loss tolerance tau =
10%
- Stabilization constant E =
500 mm^3
- Weight of R-Super losses relative to segmentation loss =
0.1 vs 1.0
- Ball kernel Gaussian standard deviation =
0.75 times ball diameter
- Third diameter estimate for two-diameter tumors =
d3 = (d1 + d2)/2
assumptions (7)
- domain assumption Radiology report descriptions of tumor count, location, and size are accurate and complete enough to serve as voxel-level training targets.
- domain assumption Tumor volume can be estimated from reported diameters using ball and ellipsoid formulas.
- domain assumption Summing predicted tumor probabilities over organ voxels gives a usable estimate of segmented tumor volume.
- domain assumption The highest response of the spherical convolution identifies the true tumor center.
- domain assumption Organ and organ sub-segment masks from a separately trained nnU-Net are accurate enough for tumor localization.
- domain assumption The LLM extraction of tumor attributes has 96% accuracy for presence and location.
- ad hoc to paper Tumors mentioned in reports without size information can be ignored during training.
Cite this review
Pith. "Pith review of Learning Segmentation from Radiology Reports." pith.science (2026). https://pith.science/paper/RVEEZGBP
@misc{pith2026250705582,
author = {Pith},
title = {Pith review of: Learning Segmentation from Radiology Reports},
year = {2026},
howpublished = {\url{https://pith.science/paper/RVEEZGBP}},
note = {Machine review of arXiv:2507.05582}
}
read the original abstract
Tumor segmentation in CT scans is key for diagnosis, surgery, and prognosis, yet segmentation masks are scarce because their creation requires time and expertise. Public abdominal CT datasets have from dozens to a couple thousand tumor masks, but hospitals have hundreds of thousands of tumor CTs with radiology reports. Thus, leveraging reports to improve segmentation is key for scaling. In this paper, we propose a report-supervision loss (R-Super) that converts radiology reports into voxel-wise supervision for tumor segmentation AI. We created a dataset with 6,718 CT-Report pairs (from the UCSF Hospital), and merged it with public CT-Mask datasets (from AbdomenAtlas 2.0). We used our R-Super to train with these masks and reports, and strongly improved tumor segmentation in internal and external validation--F1 Score increased by up to 16% with respect to training with masks only. By leveraging readily available radiology reports to supplement scarce segmentation masks, R-Super strongly improves AI performance both when very few training masks are available (e.g., 50), and when many masks were available (e.g., 1.7K). Project: https://github.com/MrGiovanni/R-Super
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2106.05735 (2021)
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., van Ginneken, B., et al.: The medical segmentation decathlon. arXiv preprint arXiv:2106.05735 (2021)
arXiv 2021
-
[2]
Bassi, P.R., Li, W., Tang, Y., Isensee, F., Wang, Z., Chen, J., Chou, Y.C., Kirch- hoff, Y., Rokuss, M., Huang, Z., Ye, J., He, J., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K.H., Jaeger, P., Ye, Y., Xie, Y., Zhang, J., Chen, Z., Xia, Y., Xing, Z., Zhu, L., Sadegheih, Y., Bozorgpour, A., Kumari, P., Azad, R., Mer- hof, D., Shi, P., Ma, T...
work page 2024
-
[3]
arXiv preprint arXiv:2501.04678 (2025), https://github.com/ MrGiovanni/RadGPT
Bassi, P.R., Yavuz, M.C., Wang, K., Chen, X., Li, W., Decherchi, S., Cav- alli, A., Yang, Y., Yuille, A., Zhou, Z.: Radgpt: Constructing 3d image-text tumor datasets. arXiv preprint arXiv:2501.04678 (2025), https://github.com/ MrGiovanni/RadGPT
arXiv 2025
-
[4]
arXiv preprint arXiv:1901.04056 (2019)
Bilic, P., Christ, P.F., Vorontsov, E., Chlebus, G., Chen, H., Dou, Q., Fu, C.W., Han, X., Heng, P.A., Hesser, J., et al.: The liver tumor segmentation benchmark (lits). arXiv preprint arXiv:1901.04056 (2019)
arXiv 2019
-
[5]
Research Square pp
Blankemeier, L., Cohen, J.P., Kumar, A., Van Veen, D., Gardezi, S.J.S., Paschali, M., Chen, Z., Delbrouck, J.B., Reis, E., Truyts, C., et al.: Merlin: A vision language foundation model for 3d computed tomography. Research Square pp. rs–3 (2024)
2024
-
[6]
Radiology: Artificial Intelligence 5(5), e230031 (2023)
Bosma, J.S., Saha, A., Hosseinzadeh, M., Slootweg, I., de Rooij, M., Huisman, H.: Semisupervised learning with report-guided pseudo labels for deep learning–based prostate cancer detection using biparametric mri. Radiology: Artificial Intelligence 5(5), e230031 (2023)
work page 2023
-
[7]
In: 2019 IEEE 16th In- ternational Symposium on Biomedical Imaging (ISBI 2019)
Chen, E.Z., Dong, X., Li, X., Jiang, H., Rong, R., Wu, J.: Lesion attributes seg- mentation for melanoma detection with multi-task u-net. In: 2019 IEEE 16th In- ternational Symposium on Biomedical Imaging (ISBI 2019). pp. 485–488. IEEE (2019)
work page 2019
-
[8]
Machine Intelligence Research pp
Chou, Y.C., Li, B., Fan, D.P., Yuille, A., Zhou, Z.: Acquiring weak annotations for tumor localization in temporal and volumetric data. Machine Intelligence Research pp. 1–13 (2024),https://github.com/johnson111788/Drag-Drop
work page 2024
Show all 30 references
-
[9]
Chou,Y.C.,Zhou,Z.,Yuille,A.:Embracingmassivemedicaldata.In:International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 24–35. Springer (2024),https://github.com/MrGiovanni/OnlineLearning
2024
-
[10]
arXiv preprint arXiv:2407.21783 (2024)
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)
2024 arXiv
-
[11]
arXiv preprint arXiv:2203.00131 (2022)
Gao, Y., Zhou, M., Liu, D., Yan, Z., Zhang, S., Metaxas, D.N.: A data-scalable transformer for medical image segmentation: architecture, model efficiency, and benchmark. arXiv preprint arXiv:2203.00131 (2022)
2022 arXiv
-
[12]
Hamamci, I.E., Er, S., Menze, B.: Ct2rep: Automated radiology report generation for 3d medical imaging (2024),https://arxiv.org/abs/2403.06801
2024 arXiv
-
[13]
arXiv preprint arXiv:1904.00445 (2019) 12 P
Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., et al.: The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv prep...
2019 arXiv
-
[14]
Nature Methods18(2), 203–211 (2021)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods18(2), 203–211 (2021)
2021
-
[15]
arXiv preprint arXiv:2404.09556 (2024)
Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jaeger, P.F.: nnu-net revisited: A call for rigorous validation in 3d medical image segmen- tation. arXiv preprint arXiv:2404.09556 (2024)
2024 arXiv
-
[16]
arXiv preprint arXiv:2307.13375 (2023)
Jaus, A., Seibold, C., Hermann, K., Walter, A., Giske, K., Haubold, J., Kleesiek, J., Stiefelhagen, R.: Towards unifying anatomy segmentation: Automated generation of a full-body ct dataset via knowledge aggregation and anatomical guidelines. arXiv preprint arXiv:2307.13375 (2023)
2023 arXiv
-
[17]
arXiv preprint arXiv:2206.08023 (2022)
Ji, Y., Bai, H., Yang, J., Ge, C., Zhu, Y., Zhang, R., Li, Z., Zhang, L., Ma, W., Wan, X., et al.: Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206.08023 (2022)
2022 arXiv
-
[18]
arXiv preprint arXiv:2501.03410 (2025),https://github
Li, W., Bassi, P.R., Lin, T., Chou, Y.C., Zhou, X., Tang, Y., Isensee, F., Wang, K., Chen, Q., Xu, X., et al.: Scalemai: Accelerating the development of trusted datasets and ai models. arXiv preprint arXiv:2501.03410 (2025),https://github. com/MrGiovanni/ScaleMAI
2025
-
[19]
Medical Image Analysis p
Li, W., Qu, C., Chen, X., Bassi, P.R., Shi, Y., Lai, Y., Yu, Q., Xue, H., Chen, Y., Lin, X., et al.: Abdomenatlas: A large-scale, detailed-annotated, & multi- center dataset for efficient transfer learning and open algorithmic benchmark- ing. Medical Image Analysis p. 103285 (...
2024
-
[20]
Li, W., Yuille, A., Zhou, Z.: How well do supervised models transfer to 3d image segmentation? In: International Conference on Learning Representations (2024), https://github.com/MrGiovanni/SuPreM
2024
-
[21]
arXiv preprint arXiv:2507.01291 (2025),https://github.com/MrGiovanni/PanTS
Li, W., Zhou, X., Chen, Q., Lin, T., Bassi, P.R., Plotka, S., Cwikla, J.B., Chen, X., Ye, C., Zhu, Z., et al.: Pants: The pancreatic tumor segmentation dataset. arXiv preprint arXiv:2507.01291 (2025),https://github.com/MrGiovanni/PanTS
2025 arXiv
-
[22]
Ma, J., Zhang, Y., Gu, S., Ge, C., Wang, E., Zhou, Q., Huang, Z., Lyu, P., He, J., Wang, B.: Automatic organ and pan-cancer segmentation in abdomen ct: the flare 2023 challenge (2024),https://arxiv.org/abs/2408.12534
2024 arXiv
-
[23]
Cancer47(1), 207–214 (1981)
Miller, A., Hoogstraten, B., Staquet, M., Winkler, A.: Reporting results of cancer treatment. Cancer47(1), 207–214 (1981)
1981
-
[24]
Diagnostic and interventional imaging101(1), 35–44 (2020)
Park, S., Chu, L., Fishman, E., Yuille, A., Vogelstein, B., Kinzler, K., Horton, K., Hruban, R., Zinreich, E., Fouladi, D.F., et al.: Annotated normal ct data of the ab- domen for deep learning: Challenges and strategies for implementation. Diagnostic and interventional imagin...
2020
-
[25]
In: Conference on Neural Information Processing Systems
Qu, C., Zhang, T., Qiao, H., Liu, J., Tang, Y., Yuille, A., Zhou, Z.: Abdomenatlas- 8k: Annotating 8,000 abdominal ct volumes for multi-organ segmentation in three weeks. In: Conference on Neural Information Processing Systems. vol. 21 (2023), https://github.com/MrGiovanni/Abd...
2023
-
[26]
Radiology: Artificial Intelligence 5(5) (2023)
Wasserthal, J., Breit, H.C., Meyer, M.T., Pradella, M., Hinck, D., Sauter, A.W., Heye, T., Boll, D.T., Cyriac, J., Yang, S., et al.: Totalsegmentator: robust segmen- tation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence 5(5) (2023)
2023
-
[27]
medRxiv (2022)
Xia, Y., Yu, Q., Chu, L., Kawamoto, S., Park, S., Liu, F., Chen, J., Zhu, Z., Li, B., Zhou, Z., et al.: The felix project: Deep networks to detect pancreatic neoplasms. medRxiv (2022)
2022
-
[28]
IEEE transactions on medical imaging40(6), 1618–1631 (2021)
Zhang, Y., Li, H., Du, J., Qin, J., Wang, T., Chen, Y., Liu, B., Gao, W., Ma, G., Lei, B.: 3d multi-attention guided multi-task learning network for automatic Learning Segmentation from Radiology Reports 13 gastric tumor segmentation and lymph node classification. IEEE transac...
2021
-
[29]
Zhou,Z.,Sodha,V.,Pang,J.,Gotway,M.B.,Liang,J.:Modelsgenesis.MedicalIm- ageAnalysis 67,101840(2021), https://github.com/MrGiovanni/ModelsGenesis
2021
-
[30]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Zhou, Z., Sodha, V., Siddiquee, M.M.R., Feng, R., Tajbakhsh, N., Gotway, M.B., Liang, J.: Models genesis: Generic autodidactic models for 3d medical image analysis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 384–393. Springe...
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.