REVIEW 4 major objections 5 minor 21 references
Stress-testing cross-cancer generalizability of 3D nnU-Net for PET-CT tumor segmentation: multi-cohort evaluation with novel oesophageal and lung cancer datasets
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Diverse training data, not architectural novelty, is the main driver of robust PET-CT tumor segmentation.
desk verdict A genuinely useful pair of new PET-CT datasets and a clean, honest benchmark, but the headline claim about diversity versus architecture is not actually tested and the cross-cancer interpretation is confounded by site and population. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the 3D nnU-Net, a self-configuring 3D U-Net for biomedical image segmentation that automatically determines patch size, architecture, and training schedule. The carrying argument is the three-way training comparison: a target-only model trained on oesophageal cancer, a public-only model trained on AutoPET, and a combined model trained on both, with all three evaluated on the same oesophageal test set, the independent lung cohort, and the reserved AutoPET test set. This setup isolates what each training-data composition contributes to in-domain accuracy and cross-domain transfer.
What would settle it
A direct test would be to train the oesophageal-only model on additional oesophageal scans acquired with the same scanner and protocol used for the Indian lung cohort; if the model still scores near zero on the lung scans, cancer type is implicated, whereas a large improvement would show that site and acquisition differences, not cancer type, drove the failure.
Extended reading notes
Core claim
The central claim is that a 3D nnU-Net trained on a combined dataset covering multiple cancers, centers, and patient populations generalizes more reliably across PET-CT domains than either a target-domain-only model or a public-only model. The oesophageal-only model shows severe cancer-type and site-specific overfitting: it reaches a mean DSC of 57.8 on its own test set but falls to 3.4 or below on the independent Indian lung cohort. The public-only model generalizes to 51.6 on the lung cohort but drops to 26.7 on oesophageal cases. The combined OC+AutoPET model delivers the most balanced outcome, with mean DSC 52.9 on lung, 40.7 on oesophageal, and 60.9 on AutoPET, along with lower Hausdorff boundary errors. The paper interprets this as quantitative evidence that multi-center, multi-cancer data diversity, rather than model complexity, is the key driver of clinically robust generalization.
Load-bearing premise
The load-bearing premise is that the oesophageal-only model's collapse on the lung cohort is caused by cancer type and anatomical site, not by the scanner, protocol, population, and annotation differences that also separate the two cohorts.
Editorial extensions
If this is right
- A single-domain model should not be deployed outside its own cohort; the oesophageal-only model's near-zero lung performance is a concrete failure mode.
- Combined training is the only tested configuration that keeps mean DSC above 40 on all three cohorts, making it a safer default for multi-cancer PET-CT workflows.
- Architectural novelty is not the primary lever for robust generalization; effort spent on collecting and curating diverse data is likely to yield larger gains.
- Small, low-contrast lesions remain a failure mode in every configuration, so human-in-the-loop review remains necessary for clinical use.
- Improved boundary metrics under combined training matter for tasks such as radiotherapy planning, where edge accuracy determines dose delivery.
Reading between the lines
- Inference: because the two new cohorts differ in center, scanner, protocol, and annotation style as well as cancer type, the cleanest reading is that training-data diversity of any kind helps; separating cancer type from domain shift would require a dataset with all four combinations of cancer type and acquisition site.
- Inference: a testable extension is to add oesophageal scans from the Indian site and lung scans from the Australian site; if combined training still balances performance, the conclusion is about diversity per se rather than the specific cancers.
- Inference: the same three-way training protocol could be applied to other PET-avid cancers such as head-and-neck, prostate, or lymphoma to see whether one combined model can hold a portfolio of cancers above a clinical threshold.
- Inference: the balanced model could serve as automated triage or region proposal, with radiologist review reserved for the small or low-contrast lesions identified in the error analysis.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces two private PET-CT datasets (Australian oesophageal cancer, n=279; Indian lung cancer, n=54) and uses 3D nnU-Net v2 to compare three training configurations: target-only (oesophageal), public-only (AutoPET), and combined training. The models are evaluated on the oesophageal test set, the AutoPET test set, and the Indian lung cohort using DSC, precision, recall, and HD95. The reported results show that the oesophageal-only model is best in-domain but collapses on external cohorts, the AutoPET-only model generalizes moderately but underperforms on oesophageal cases, and the combined model gives the most balanced performance across cohorts. The paper concludes that dataset diversity, rather than architectural novelty, is the primary driver of robust generalization.
Significance. If the results hold, the two expert-annotated datasets and the three-way training comparison are a useful contribution to benchmarking PET-CT lesion segmentation under domain shift. The combined-training result provides a reproducible baseline for future work on oesophageal and lung cancer segmentation. However, the headline claim that dataset diversity 'outweighs architectural novelty' is not tested by the experimental design, and the cross-cancer interpretation is confounded by site and protocol differences. The paper would benefit from uncertainty quantification and more careful scoping of its claims.
major comments (4)
- [Section 2.1 and Section 3.1 (Table 2)] The inference that the OC-only model's collapse on the lung cohort (DSC 1.25) is due to 'cancer-type and site-specific overfitting' is confounded. Cancer type is entangled with scanner hardware, acquisition protocol, reconstruction settings, patient demographics, and annotation practice between the Australian oesophageal cohort and the Indian lung cohort. The manuscript reports no scanner metadata, no per-center stratification, and no statistical adjustment, so the observed failure could be a generic domain-shift or data-composition effect rather than evidence about cross-cancer specificity. Please provide acquisition metadata and per-center analyses that isolate cancer type, or explicitly narrow the conclusion to cross-cohort generalization.
- [Abstract and Section 4 (Discussion)] The claim that 'dataset diversity ... outweighs architectural novelty' is not supported by the experimental design. No alternative architecture is evaluated; all models use the same 3D nnU-Net and differ only in training-data composition. The results can at most support the conclusion that, within nnU-Net, training-data composition affects cross-domain generalization. An architectural comparison or a revised, accurately scoped claim is required.
- [Section 3.1, Table 2] The 'most balanced' conclusion rests on aggregate point estimates without confidence intervals, significance tests, or per-cohort variance summaries. For example, the combined model's lung DSC (52.9) is only 1.3 points above the AutoPET-only model (51.6), a difference likely within sampling variability given the small external cohort (n=54). Please report bootstrap confidence intervals and paired tests where appropriate, or explicitly state that the observed differences are not statistically assessed.
- [Table 1 and Section 2.1] The data quantities are internally inconsistent. A 70:30 split of 279 oesophageal cases gives approximately 195 training and 84 test cases, but Table 1 lists 210 train and 69 test (while its caption says 200 train). Similarly, the introduction describes AutoPET as 1,014 studies, but Table 1 uses 324 training and 139 test cases, whose sum (463) is not 70% of 1,014. Please clarify the actual inclusion criteria, selection procedure, and exact split for both the private and public datasets, and explain any discrepancy with the cited AutoPET corpus.
minor comments (5)
- [Abstract] The phrase 'mean DSC = lung (52.9); oesophageal (40.7); AutoPET (60.9)' is awkward; please reword as 'lung: 52.9; oesophageal: 40.7; AutoPET: 60.9'.
- [Section 2.1 and Section 2.2] There are several typos and grammar issues: 'commitee', 'annoataions', 'sybtypes', and the sentence 'we have used two RTX A6000 NVIDIA GPU card with 24 GB memory each for training' is a fragment.
- [Section 2.1 and Figure 2] The text refers to 'Figure 2 describes the number of images used for training and testing', but Figure 2 is captioned 'Representation of the datasets' and the split counts appear in Table 1; please correct the cross-reference. Also, the demographic and radiomics panels described in Section 2.1 are not clearly labeled in the figure caption.
- [Abstract and Table 2] The abstract reports the OC-only model's external performance as 'mean DSC ≤ 3.4', but Table 2 gives 1.25 for the lung cohort and 3.4 for AutoPET; please use exact values or clarify the bound.
- [Section 3.2] Quantitative example values in the qualitative analysis are given as proportions (e.g., DSC = 0.89) while Table 2 reports DSC as percentages (e.g., 57.8); please use a consistent scale throughout.
Circularity Check
No circularity: the paper is an empirical multi-cohort benchmark with independent held-out test sets; no prediction reduces to a fitted input or self-citation chain.
full rationale
The paper reports a standard supervised segmentation benchmark: three nnU-Net training configurations (OC-only, AutoPET-only, OC+AutoPET) are evaluated on held-out OC, AutoPET, and external lung test cohorts. The central claim that dataset diversity drives robust generalization is a comparative empirical conclusion drawn from Table 2 and the qualitative cases, not a quantity derived from its own inputs. There are no fitted parameters renamed as predictions, no uniqueness theorems imported from the authors' prior work, and no equations in which the output is identical to the input by construction. The only self-citations are supporting references (e.g., nnU-Net methods, AI agents in nuclear medicine), and none carries a load-bearing argument that reduces to the authors' own unverified result. The reviewer's concern about confounded cancer type versus site/protocol differences is a validity or correctness limitation, not circularity: it does not show that the reported numbers were forced by the paper's assumptions. Similarly, the absence of an alternative architecture means the 'diversity outweighs architectural novelty' framing is not fully stress-tested, but that is an incompleteness concern, not a circular one. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption AutoPET 70:30 split (324 train, 139 test) is representative and comparable to the official benchmark.
- domain assumption Expert annotations are accurate and consistent enough to serve as ground truth across all three cohorts.
- domain assumption Differences between oesophageal (Australian) and lung (Indian) cohorts are attributed to cancer type, not to scanner or population.
- ad hoc to paper nnU-Net v2's self-configuration is an adequate representation of the architectural baseline; no alternative architecture is compared.
Cite this review
Pith. "Pith review of Stress-testing cross-cancer generalizability of 3D nnU-Net for PET-CT tumor segmentation: multi-cohort evaluation with novel oesophageal and lung cancer datasets." pith.science (2026). https://pith.science/paper/JKLOBDWJ
@misc{pith2026250818612,
author = {Pith},
title = {Pith review of: Stress-testing cross-cancer generalizability of 3D nnU-Net for PET-CT tumor segmentation: multi-cohort evaluation with novel oesophageal and lung cancer datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/JKLOBDWJ}},
note = {Machine review of arXiv:2508.18612}
}
read the original abstract
Robust generalization is essential for deploying deep learning based tumor segmentation in clinical PET-CT workflows, where anatomical sites, scanners, and patient populations vary widely. This study presents the first cross cancer evaluation of nnU-Net on PET-CT, introducing two novel, expert-annotated whole-body datasets. 279 patients with oesophageal cancer (Australian cohort) and 54 with lung cancer (Indian cohort). These cohorts complement the public AutoPET dataset and enable systematic stress-testing of cross domain performance. We trained and tested 3D nnUNet models under three paradigms. Target only (oesophageal), public only (AutoPET), and combined training. For the tested sets, the oesophageal only model achieved the best in-domain accuracy (mean DSC, 57.8) but failed on external Indian lung cohort (mean DSC less than 3.4), indicating severe overfitting. The public only model generalized more broadly (mean DSC, 63.5 on AutoPET, 51.6 on Indian lung cohort) but underperformed in oesophageal Australian cohort (mean DSC, 26.7). The combined approach provided the most balanced results (mean DSC, lung (52.9), oesophageal (40.7), AutoPET (60.9)), reducing boundary errors and improving robustness across all cohorts. These findings demonstrate that dataset diversity, particularly multi demographic, multi center and multi cancer integration, outweighs architectural novelty as the key driver of robust generalization. This work presents the demography based cross cancer deep learning segmentation evaluation and highlights dataset diversity, rather than model complexity, as the foundation for clinically robust segmentation.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Anatomy guided modality fusion for cancer segmentation in pet ct vol- umes and images
Ibtihaj Ahmad, Sadia Jabbar Anwar, Bagh Hussain, Atiq ur Rehman, and Amine Bermak. Anatomy guided modality fusion for cancer segmentation in pet ct vol- umes and images. Scientific Reports, 15(1):12153, 2025
work page 2025
-
[2]
Montserrat Carles, Dejan Kuhn, Tobias Fechter, Dimos Baltas, Michael Mix, Ur- sula Nestle, Anca L Grosu, Luis Mart´ ı-Bonmat´ ı, Gianluca Radicioni, and Eleni Gkika. Development and evaluation of two open-source nnu-net models for auto- matic segmentation of lung tumors on pet and ct images with and without respi- ratory motion compensation. European radi...
work page 2024
-
[3]
Deep learning tech- niques in pet/ct imaging: A comprehensive review from sinogram to image space
Maryam Fallahpoor, Subrata Chakraborty, Biswajeet Pradhan, Oliver Faust, Pra- bal Datta Barua, Hossein Chegeni, and Rajendra Acharya. Deep learning tech- niques in pet/ct imaging: A comprehensive review from sinogram to image space. Computer methods and programs in biomedicine , 243:107880, 2024
work page 2024
-
[4]
S. Gatidis and T. Kuestner. A whole-body fdg-pet/ct dataset with manually an- notated tumor lesions (fdg-pet-ct-lesions). The Cancer Imaging Archive [Dataset], 2022
work page 2022
-
[5]
The autopet challenge: towards fully automated lesion segmentation in oncologic pet/ct imag- ing
Sergios Gatidis, Marcel Fr¨ uh, Matthias Fabritius, Sijing Gu, Konstantin Nikolaou, Christian La Foug` ere, Jin Ye, Junjun He, Yige Peng, Lei Bi, et al. The autopet challenge: towards fully automated lesion segmentation in oncologic pet/ct imag- ing. 2023
work page 2023
-
[6]
Sergios Gatidis, Marcel Fr¨ uh, Matthias P Fabritius, Sijing Gu, Konstantin Niko- laou, Christian La Foug` ere, Jin Ye, Junjun He, Yige Peng, Lei Bi, et al. Results from the autopet challenge on fully automated lesion segmentation in oncologic pet/ct imaging. Nature Machine Intelligence , 6(11):1396–1405, 2024
work page 2024
-
[7]
A whole-body fdg-pet/ct dataset with manually anno- tated tumor lesions
Sergios Gatidis, Tobias Hepp, Marcel Fr¨ uh, Christian La Foug` ere, Konstantin Nikolaou, Christina Pfannenberg, Bernhard Sch¨ olkopf, Thomas K¨ ustner, Clemens Cyran, and Daniel Rubin. A whole-body fdg-pet/ct dataset with manually anno- tated tumor lesions. Scientific Data, 9(1):601, 2022
work page 2022
-
[8]
Automatic segmentation of pet/ct lymphoma using an nnu-net model, 2023
Navid Hasani, Liangchen Liu, Faraz Farhadi, Taylor Delie, Benjamin Hou, Michael Morris, Ronald Summers, and Babak Saboury. Automatic segmentation of pet/ct lymphoma using an nnu-net model, 2023
work page 2023
Show all 21 references
-
[9]
nnu-net: a self-configuring method for deep learning-based biomedical image segmentation
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier- Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):203–211, 2021. Title Suppressed Due to Excessive Length 13
2021
-
[10]
nnu-net revisited: A call for rigorous vali- dation in 3d medical image segmentation
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. nnu-net revisited: A call for rigorous vali- dation in 3d medical image segmentation. In International Conference on Medical Image Computing and Computer-Assi...
2024
-
[11]
Appropriate use criteria for 18f-fdg pet/ct in restaging and treatment response assessment of malignant disease
Hossein Jadvar, Patrick M Colletti, Roberto Delgado-Bolton, Giuseppe Esposito, Bernd J Krause, Andrei H Iagaru, Helen Nadel, David I Quinn, Eric Rohren, Rathan M Subramaniam, et al. Appropriate use criteria for 18f-fdg pet/ct in restaging and treatment response assessment of m...
2026
-
[12]
Jeblick et al
K. Jeblick et al. A whole-body psma-pet/ct dataset with manually annotated tumor lesions (psma-pet-ct-lesions) (version 1). The Cancer Imaging Archive [Dataset], 2024
2024
-
[13]
Towards a guideline for evaluation metrics in medical image segmentation.BMC Research Notes, 15(1):210, 2022
Dominik M¨ uller, I˜ naki Soto-Rey, and Frank Kramer. Towards a guideline for evaluation metrics in medical image segmentation.BMC Research Notes, 15(1):210, 2022
2022
-
[14]
From fdg to psma: A hitchhiker’s guide to multitracer, multicenter lesion segmentation in pet/ct imaging
Maximilian Rokuss, Balint Kovacs, Yannick Kirchhoff, Shuhan Xiao, Constantin Ulrich, Klaus H Maier-Hein, and Fabian Isensee. From fdg to psma: A hitchhiker’s guide to multitracer, multicenter lesion segmentation in pet/ct imaging. arXiv preprint arXiv:2409.09478, 2024
2024 arXiv
-
[15]
U-net and its variants for medical image segmentation: theory and applications
Nahian Siddique, Paheding Sidike, Colin Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: theory and applications. arXiv preprint arXiv:2011.01118, 2020
2011 arXiv
-
[16]
Computational radiomics system to decode the radiographic phenotype
Joost JM Van Griethuysen, Andriy Fedorov, Chintan Parmar, Ahmed Hosny, Nicole Aucoin, Vivek Narayan, Regina GH Beets-Tan, Jean-Christophe Fillion- Robin, Steve Pieper, and Hugo JWL Aerts. Computational radiomics system to decode the radiographic phenotype. Cancer research, 77(...
2017
-
[17]
The potential role of ai agents in transforming nuclear medicine research and cancer management in india
Rajat Vashistha, Arif Gulzar, Parveen Kundu, Punit Sharma, Mark Brunstein, and Viktor Vegh. The potential role of ai agents in transforming nuclear medicine research and cancer management in india. arXiv preprint arXiv:2503.07673, 2025
2025 arXiv
-
[18]
Modular gan: positron emission tomography image reconstruction using two generative adversarial networks
Rajat Vashistha, Viktor Vegh, Hamed Moradi, Amanda Hammond, Kieran O’Brien, and David Reutens. Modular gan: positron emission tomography image reconstruction using two generative adversarial networks. Frontiers in Radiology, 4:1466498, 2024
2024
-
[19]
Dual channel cw nnu-net for 3d pet-ct lesion segmentation in 2024 autopet iii challenge
Ching-Wei Wang, Ting-Sheng Su, and Keng-Wei Liu. Dual channel cw nnu-net for 3d pet-ct lesion segmentation in 2024 autopet iii challenge. arXiv preprint arXiv:2409.07144, 2024
2024 arXiv
-
[20]
Robust and generaliz- able artificial intelligence for multi-organ segmentation in ultra-low-dose total-body pet imaging: a multi-center and cross-tracer study
Hanzhong Wang, Xiaoya Qiao, Wenxiang Ding, Gaoyu Chen, Ying Miao, Rui Guo, Xiaohua Zhu, Zhaoping Cheng, Jiehua Xu, Biao Li, et al. Robust and generaliz- able artificial intelligence for multi-organ segmentation in ultra-low-dose total-body pet imaging: a multi-center and cross...
2025
-
[21]
Quantitative analysis of pet studies
Wolfgang A Weber. Quantitative analysis of pet studies. Radiotherapy and Oncol- ogy, 96(3):308–310, 2010
2010
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.