Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Stress-testing cross-cancer generalizability of 3D nnU-Net for PET-CT tumor segmentation: multi-cohort evaluation with novel oesophageal and lung cancer datasets

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read Diverse training data, not architectural novelty, is the main driver of robust PET-CT tumor segmentation.

desk verdict A genuinely useful pair of new PET-CT datasets and a clean, honest benchmark, but the headline claim about diversity versus architecture is not actually tested and the cross-cancer interpretation is confounded by site and population. read the letter →

arxiv 2508.18612 v1 pith:JKLOBDWJ submitted 2025-08-26 eess.IV cs.LG

classification eess.IVcs.LG
keywords PET-CTtumorsegmentationnnU-Netcross-cancergeneralizationoesophagealcancerlungmulti-centerdataAutoPET
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that, for 3D PET-CT tumor segmentation, dataset diversity matters more than architectural novelty. It introduces two new expert-annotated whole-body datasets, 279 oesophageal cancer scans from an Australian cohort and 54 lung cancer scans from an Indian cohort, and uses them with the public AutoPET dataset to stress-test a 3D nnU-Net under three training regimes. The oesophageal-only model scores high on its own domain (mean DSC 57.8) but collapses on the external lung cohort (mean DSC below 3.4); the AutoPET-only model generalizes better but underperforms on oesophageal cases (mean DSC 26.7). The combined model gives the most balanced result (mean DSC 52.9 lung, 40.7 oesophageal, 60.9 AutoPET), with better boundary metrics. If the claim holds, clinical deployment should prioritize assembling diverse multi-center, multi-cancer training data over designing bespoke architectures.

What carries the argument

The central object is the 3D nnU-Net, a self-configuring 3D U-Net for biomedical image segmentation that automatically determines patch size, architecture, and training schedule. The carrying argument is the three-way training comparison: a target-only model trained on oesophageal cancer, a public-only model trained on AutoPET, and a combined model trained on both, with all three evaluated on the same oesophageal test set, the independent lung cohort, and the reserved AutoPET test set. This setup isolates what each training-data composition contributes to in-domain accuracy and cross-domain transfer.

What would settle it

A direct test would be to train the oesophageal-only model on additional oesophageal scans acquired with the same scanner and protocol used for the Indian lung cohort; if the model still scores near zero on the lung scans, cancer type is implicated, whereas a large improvement would show that site and acquisition differences, not cancer type, drove the failure.

Watch

Extended reading notes

Core claim

The central claim is that a 3D nnU-Net trained on a combined dataset covering multiple cancers, centers, and patient populations generalizes more reliably across PET-CT domains than either a target-domain-only model or a public-only model. The oesophageal-only model shows severe cancer-type and site-specific overfitting: it reaches a mean DSC of 57.8 on its own test set but falls to 3.4 or below on the independent Indian lung cohort. The public-only model generalizes to 51.6 on the lung cohort but drops to 26.7 on oesophageal cases. The combined OC+AutoPET model delivers the most balanced outcome, with mean DSC 52.9 on lung, 40.7 on oesophageal, and 60.9 on AutoPET, along with lower Hausdorff boundary errors. The paper interprets this as quantitative evidence that multi-center, multi-cancer data diversity, rather than model complexity, is the key driver of clinically robust generalization.

Load-bearing premise

The load-bearing premise is that the oesophageal-only model's collapse on the lung cohort is caused by cancer type and anatomical site, not by the scanner, protocol, population, and annotation differences that also separate the two cohorts.

Editorial extensions

If this is right

  • A single-domain model should not be deployed outside its own cohort; the oesophageal-only model's near-zero lung performance is a concrete failure mode.
  • Combined training is the only tested configuration that keeps mean DSC above 40 on all three cohorts, making it a safer default for multi-cancer PET-CT workflows.
  • Architectural novelty is not the primary lever for robust generalization; effort spent on collecting and curating diverse data is likely to yield larger gains.
  • Small, low-contrast lesions remain a failure mode in every configuration, so human-in-the-loop review remains necessary for clinical use.
  • Improved boundary metrics under combined training matter for tasks such as radiotherapy planning, where edge accuracy determines dose delivery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the two new cohorts differ in center, scanner, protocol, and annotation style as well as cancer type, the cleanest reading is that training-data diversity of any kind helps; separating cancer type from domain shift would require a dataset with all four combinations of cancer type and acquisition site.
  • Inference: a testable extension is to add oesophageal scans from the Indian site and lung scans from the Australian site; if combined training still balances performance, the conclusion is about diversity per se rather than the specific cancers.
  • Inference: the same three-way training protocol could be applied to other PET-avid cancers such as head-and-neck, prostate, or lymphoma to see whether one combined model can hold a portfolio of cancers above a clinical threshold.
  • Inference: the balanced model could serve as automated triage or region proposal, with radiologist review reserved for the small or low-contrast lesions identified in the error analysis.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces two private PET-CT datasets (Australian oesophageal cancer, n=279; Indian lung cancer, n=54) and uses 3D nnU-Net v2 to compare three training configurations: target-only (oesophageal), public-only (AutoPET), and combined training. The models are evaluated on the oesophageal test set, the AutoPET test set, and the Indian lung cohort using DSC, precision, recall, and HD95. The reported results show that the oesophageal-only model is best in-domain but collapses on external cohorts, the AutoPET-only model generalizes moderately but underperforms on oesophageal cases, and the combined model gives the most balanced performance across cohorts. The paper concludes that dataset diversity, rather than architectural novelty, is the primary driver of robust generalization.

Significance. If the results hold, the two expert-annotated datasets and the three-way training comparison are a useful contribution to benchmarking PET-CT lesion segmentation under domain shift. The combined-training result provides a reproducible baseline for future work on oesophageal and lung cancer segmentation. However, the headline claim that dataset diversity 'outweighs architectural novelty' is not tested by the experimental design, and the cross-cancer interpretation is confounded by site and protocol differences. The paper would benefit from uncertainty quantification and more careful scoping of its claims.

major comments (4)
  1. [Section 2.1 and Section 3.1 (Table 2)] The inference that the OC-only model's collapse on the lung cohort (DSC 1.25) is due to 'cancer-type and site-specific overfitting' is confounded. Cancer type is entangled with scanner hardware, acquisition protocol, reconstruction settings, patient demographics, and annotation practice between the Australian oesophageal cohort and the Indian lung cohort. The manuscript reports no scanner metadata, no per-center stratification, and no statistical adjustment, so the observed failure could be a generic domain-shift or data-composition effect rather than evidence about cross-cancer specificity. Please provide acquisition metadata and per-center analyses that isolate cancer type, or explicitly narrow the conclusion to cross-cohort generalization.
  2. [Abstract and Section 4 (Discussion)] The claim that 'dataset diversity ... outweighs architectural novelty' is not supported by the experimental design. No alternative architecture is evaluated; all models use the same 3D nnU-Net and differ only in training-data composition. The results can at most support the conclusion that, within nnU-Net, training-data composition affects cross-domain generalization. An architectural comparison or a revised, accurately scoped claim is required.
  3. [Section 3.1, Table 2] The 'most balanced' conclusion rests on aggregate point estimates without confidence intervals, significance tests, or per-cohort variance summaries. For example, the combined model's lung DSC (52.9) is only 1.3 points above the AutoPET-only model (51.6), a difference likely within sampling variability given the small external cohort (n=54). Please report bootstrap confidence intervals and paired tests where appropriate, or explicitly state that the observed differences are not statistically assessed.
  4. [Table 1 and Section 2.1] The data quantities are internally inconsistent. A 70:30 split of 279 oesophageal cases gives approximately 195 training and 84 test cases, but Table 1 lists 210 train and 69 test (while its caption says 200 train). Similarly, the introduction describes AutoPET as 1,014 studies, but Table 1 uses 324 training and 139 test cases, whose sum (463) is not 70% of 1,014. Please clarify the actual inclusion criteria, selection procedure, and exact split for both the private and public datasets, and explain any discrepancy with the cited AutoPET corpus.
minor comments (5)
  1. [Abstract] The phrase 'mean DSC = lung (52.9); oesophageal (40.7); AutoPET (60.9)' is awkward; please reword as 'lung: 52.9; oesophageal: 40.7; AutoPET: 60.9'.
  2. [Section 2.1 and Section 2.2] There are several typos and grammar issues: 'commitee', 'annoataions', 'sybtypes', and the sentence 'we have used two RTX A6000 NVIDIA GPU card with 24 GB memory each for training' is a fragment.
  3. [Section 2.1 and Figure 2] The text refers to 'Figure 2 describes the number of images used for training and testing', but Figure 2 is captioned 'Representation of the datasets' and the split counts appear in Table 1; please correct the cross-reference. Also, the demographic and radiomics panels described in Section 2.1 are not clearly labeled in the figure caption.
  4. [Abstract and Table 2] The abstract reports the OC-only model's external performance as 'mean DSC ≤ 3.4', but Table 2 gives 1.25 for the lung cohort and 3.4 for AutoPET; please use exact values or clarify the bound.
  5. [Section 3.2] Quantitative example values in the qualitative analysis are given as proportions (e.g., DSC = 0.89) while Table 2 reports DSC as percentages (e.g., 57.8); please use a consistent scale throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical multi-cohort benchmark with independent held-out test sets; no prediction reduces to a fitted input or self-citation chain.

full rationale

The paper reports a standard supervised segmentation benchmark: three nnU-Net training configurations (OC-only, AutoPET-only, OC+AutoPET) are evaluated on held-out OC, AutoPET, and external lung test cohorts. The central claim that dataset diversity drives robust generalization is a comparative empirical conclusion drawn from Table 2 and the qualitative cases, not a quantity derived from its own inputs. There are no fitted parameters renamed as predictions, no uniqueness theorems imported from the authors' prior work, and no equations in which the output is identical to the input by construction. The only self-citations are supporting references (e.g., nnU-Net methods, AI agents in nuclear medicine), and none carries a load-bearing argument that reduces to the authors' own unverified result. The reviewer's concern about confounded cancer type versus site/protocol differences is a validity or correctness limitation, not circularity: it does not show that the reported numbers were forced by the paper's assumptions. Similarly, the absence of an alternative architecture means the 'diversity outweighs architectural novelty' framing is not fully stress-tested, but that is an incompleteness concern, not a circular one. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numbers were fitted to data beyond standard nnU-Net v2 hyperparameters, which are self-configured. The main assumptions are about dataset representativeness and annotation quality. The most fragile assumption is that cohort differences reflect cancer type rather than scanner, population, and protocol, which is not controlled in the design.

assumptions (4)
  • domain assumption AutoPET 70:30 split (324 train, 139 test) is representative and comparable to the official benchmark.
    Section 2.1 and Table 1; official AutoPET uses 1,014 studies with a fixed external test set, so the split details matter.
  • domain assumption Expert annotations are accurate and consistent enough to serve as ground truth across all three cohorts.
    Section 2.1; no inter-observer variability or consensus metrics are reported.
  • domain assumption Differences between oesophageal (Australian) and lung (Indian) cohorts are attributed to cancer type, not to scanner or population.
    Section 2.1 and Figure 2; the study does not control these confounds.
  • ad hoc to paper nnU-Net v2's self-configuration is an adequate representation of the architectural baseline; no alternative architecture is compared.
    Section 2.2 and Discussion; the claim that data diversity outweighs architectural novelty requires this benchmark assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stress-testing cross-cancer generalizability of 3D nnU-Net for PET-CT tumor segmentation: multi-cohort evaluation with novel oesophageal and lung cancer datasets." pith.science (2026). https://pith.science/paper/JKLOBDWJ

@misc{pith2026250818612,
  author       = {Pith},
  title        = {Pith review of: Stress-testing cross-cancer generalizability of 3D nnU-Net for PET-CT tumor segmentation: multi-cohort evaluation with novel oesophageal and lung cancer datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKLOBDWJ}},
  note         = {Machine review of arXiv:2508.18612}
}
read the original abstract

Robust generalization is essential for deploying deep learning based tumor segmentation in clinical PET-CT workflows, where anatomical sites, scanners, and patient populations vary widely. This study presents the first cross cancer evaluation of nnU-Net on PET-CT, introducing two novel, expert-annotated whole-body datasets. 279 patients with oesophageal cancer (Australian cohort) and 54 with lung cancer (Indian cohort). These cohorts complement the public AutoPET dataset and enable systematic stress-testing of cross domain performance. We trained and tested 3D nnUNet models under three paradigms. Target only (oesophageal), public only (AutoPET), and combined training. For the tested sets, the oesophageal only model achieved the best in-domain accuracy (mean DSC, 57.8) but failed on external Indian lung cohort (mean DSC less than 3.4), indicating severe overfitting. The public only model generalized more broadly (mean DSC, 63.5 on AutoPET, 51.6 on Indian lung cohort) but underperformed in oesophageal Australian cohort (mean DSC, 26.7). The combined approach provided the most balanced results (mean DSC, lung (52.9), oesophageal (40.7), AutoPET (60.9)), reducing boundary errors and improving robustness across all cohorts. These findings demonstrate that dataset diversity, particularly multi demographic, multi center and multi cancer integration, outweighs architectural novelty as the key driver of robust generalization. This work presents the demography based cross cancer deep learning segmentation evaluation and highlights dataset diversity, rather than model complexity, as the foundation for clinically robust segmentation.

Figures

Figures reproduced from arXiv: 2508.18612 by the authors.

Figure 1
Figure 1. Schematic visualizes the dataset curation, model training, and evaluation pro￾cess of 3D nnU-Net. (Drawn in canvas using Gemini 2.5 pro) Datasets Train Test Curated Oesophagus (OC) 210 69 Lung 0 54 Public Melanoma, Lung and Lymphoma (AutoPET) 324 139 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Representation of the datasets (green: PET, orange: CT). Radiomics features are computed using Pyradiomics package [16]. Its eval￾uation ( [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Boxplots of segmentation performance (DSC, Precision, Recall, HD95) across oesophageal cancer (OC) and lung cancer (LC) test cohorts. Models were trained on OC-only (baseline), public-only, and combined (baseline+public) datasets. The base￾line achieves strong in-domain performance but fails cross-domain, the public model generalizes better but underperforms on OC, while combined training provides the most balanced … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Coronal PET-CT views present a whole-body comparison of segmentation out￾comes for two oesophageal cancer test cases (OC 1 and OC 2) and one independent lung cancer test case (LC 1) across three training configurations: Baseline (target-domain only), Public-only, and c…
Figure 5
Figure 5. Figure 5: The axial PET-CT slices in the figure illustrate qualitative and quantitative differences in segmentation performance across three training configurations For OC 1, which represents a large, high-contrast oesophageal lesion, both the Baseline and combined models achiev…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

  1. [1]

    Anatomy guided modality fusion for cancer segmentation in pet ct vol- umes and images

    Ibtihaj Ahmad, Sadia Jabbar Anwar, Bagh Hussain, Atiq ur Rehman, and Amine Bermak. Anatomy guided modality fusion for cancer segmentation in pet ct vol- umes and images. Scientific Reports, 15(1):12153, 2025

  2. [2]

    Development and evaluation of two open-source nnu-net models for auto- matic segmentation of lung tumors on pet and ct images with and without respi- ratory motion compensation

    Montserrat Carles, Dejan Kuhn, Tobias Fechter, Dimos Baltas, Michael Mix, Ur- sula Nestle, Anca L Grosu, Luis Mart´ ı-Bonmat´ ı, Gianluca Radicioni, and Eleni Gkika. Development and evaluation of two open-source nnu-net models for auto- matic segmentation of lung tumors on pet and ct images with and without respi- ratory motion compensation. European radi...

  3. [3]

    Deep learning tech- niques in pet/ct imaging: A comprehensive review from sinogram to image space

    Maryam Fallahpoor, Subrata Chakraborty, Biswajeet Pradhan, Oliver Faust, Pra- bal Datta Barua, Hossein Chegeni, and Rajendra Acharya. Deep learning tech- niques in pet/ct imaging: A comprehensive review from sinogram to image space. Computer methods and programs in biomedicine , 243:107880, 2024

  4. [4]

    Gatidis and T

    S. Gatidis and T. Kuestner. A whole-body fdg-pet/ct dataset with manually an- notated tumor lesions (fdg-pet-ct-lesions). The Cancer Imaging Archive [Dataset], 2022

  5. [5]

    The autopet challenge: towards fully automated lesion segmentation in oncologic pet/ct imag- ing

    Sergios Gatidis, Marcel Fr¨ uh, Matthias Fabritius, Sijing Gu, Konstantin Nikolaou, Christian La Foug` ere, Jin Ye, Junjun He, Yige Peng, Lei Bi, et al. The autopet challenge: towards fully automated lesion segmentation in oncologic pet/ct imag- ing. 2023

  6. [6]

    Results from the autopet challenge on fully automated lesion segmentation in oncologic pet/ct imaging

    Sergios Gatidis, Marcel Fr¨ uh, Matthias P Fabritius, Sijing Gu, Konstantin Niko- laou, Christian La Foug` ere, Jin Ye, Junjun He, Yige Peng, Lei Bi, et al. Results from the autopet challenge on fully automated lesion segmentation in oncologic pet/ct imaging. Nature Machine Intelligence , 6(11):1396–1405, 2024

  7. [7]

    A whole-body fdg-pet/ct dataset with manually anno- tated tumor lesions

    Sergios Gatidis, Tobias Hepp, Marcel Fr¨ uh, Christian La Foug` ere, Konstantin Nikolaou, Christina Pfannenberg, Bernhard Sch¨ olkopf, Thomas K¨ ustner, Clemens Cyran, and Daniel Rubin. A whole-body fdg-pet/ct dataset with manually anno- tated tumor lesions. Scientific Data, 9(1):601, 2022

  8. [8]

    Automatic segmentation of pet/ct lymphoma using an nnu-net model, 2023

    Navid Hasani, Liangchen Liu, Faraz Farhadi, Taylor Delie, Benjamin Hou, Michael Morris, Ronald Summers, and Babak Saboury. Automatic segmentation of pet/ct lymphoma using an nnu-net model, 2023

Show all 21 references
  1. [9]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier- Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):203–211, 2021. Title Suppressed Due to Excessive Length 13

  2. [10]

    nnu-net revisited: A call for rigorous vali- dation in 3d medical image segmentation

    Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. nnu-net revisited: A call for rigorous vali- dation in 3d medical image segmentation. In International Conference on Medical Image Computing and Computer-Assi...

  3. [11]

    Appropriate use criteria for 18f-fdg pet/ct in restaging and treatment response assessment of malignant disease

    Hossein Jadvar, Patrick M Colletti, Roberto Delgado-Bolton, Giuseppe Esposito, Bernd J Krause, Andrei H Iagaru, Helen Nadel, David I Quinn, Eric Rohren, Rathan M Subramaniam, et al. Appropriate use criteria for 18f-fdg pet/ct in restaging and treatment response assessment of m...

  4. [12]

    Jeblick et al

    K. Jeblick et al. A whole-body psma-pet/ct dataset with manually annotated tumor lesions (psma-pet-ct-lesions) (version 1). The Cancer Imaging Archive [Dataset], 2024

  5. [13]

    Towards a guideline for evaluation metrics in medical image segmentation.BMC Research Notes, 15(1):210, 2022

    Dominik M¨ uller, I˜ naki Soto-Rey, and Frank Kramer. Towards a guideline for evaluation metrics in medical image segmentation.BMC Research Notes, 15(1):210, 2022

  6. [14]

    From fdg to psma: A hitchhiker’s guide to multitracer, multicenter lesion segmentation in pet/ct imaging

    Maximilian Rokuss, Balint Kovacs, Yannick Kirchhoff, Shuhan Xiao, Constantin Ulrich, Klaus H Maier-Hein, and Fabian Isensee. From fdg to psma: A hitchhiker’s guide to multitracer, multicenter lesion segmentation in pet/ct imaging. arXiv preprint arXiv:2409.09478, 2024

  7. [15]

    U-net and its variants for medical image segmentation: theory and applications

    Nahian Siddique, Paheding Sidike, Colin Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: theory and applications. arXiv preprint arXiv:2011.01118, 2020

  8. [16]

    Computational radiomics system to decode the radiographic phenotype

    Joost JM Van Griethuysen, Andriy Fedorov, Chintan Parmar, Ahmed Hosny, Nicole Aucoin, Vivek Narayan, Regina GH Beets-Tan, Jean-Christophe Fillion- Robin, Steve Pieper, and Hugo JWL Aerts. Computational radiomics system to decode the radiographic phenotype. Cancer research, 77(...

  9. [17]

    The potential role of ai agents in transforming nuclear medicine research and cancer management in india

    Rajat Vashistha, Arif Gulzar, Parveen Kundu, Punit Sharma, Mark Brunstein, and Viktor Vegh. The potential role of ai agents in transforming nuclear medicine research and cancer management in india. arXiv preprint arXiv:2503.07673, 2025

  10. [18]

    Modular gan: positron emission tomography image reconstruction using two generative adversarial networks

    Rajat Vashistha, Viktor Vegh, Hamed Moradi, Amanda Hammond, Kieran O’Brien, and David Reutens. Modular gan: positron emission tomography image reconstruction using two generative adversarial networks. Frontiers in Radiology, 4:1466498, 2024

  11. [19]

    Dual channel cw nnu-net for 3d pet-ct lesion segmentation in 2024 autopet iii challenge

    Ching-Wei Wang, Ting-Sheng Su, and Keng-Wei Liu. Dual channel cw nnu-net for 3d pet-ct lesion segmentation in 2024 autopet iii challenge. arXiv preprint arXiv:2409.07144, 2024

  12. [20]

    Robust and generaliz- able artificial intelligence for multi-organ segmentation in ultra-low-dose total-body pet imaging: a multi-center and cross-tracer study

    Hanzhong Wang, Xiaoya Qiao, Wenxiang Ding, Gaoyu Chen, Ying Miao, Rui Guo, Xiaohua Zhu, Zhaoping Cheng, Jiehua Xu, Biao Li, et al. Robust and generaliz- able artificial intelligence for multi-organ segmentation in ultra-low-dose total-body pet imaging: a multi-center and cross...

  13. [21]

    Quantitative analysis of pet studies

    Wolfgang A Weber. Quantitative analysis of pet studies. Radiotherapy and Oncol- ogy, 96(3):308–310, 2010

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.