REVIEW 5 major objections 5 minor 25 references
Empirical investigation of 3D CT Foundation Models and Unsupervised Adaptation for Head and Neck Cancer Recurrence Prediction
T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Across 3,644 head-and-neck patients, no single 3D CT foundation model consistently predicts recurrence on both internal and external cohorts, and the best external results come from a vision-language pretrained model combined with gating-ba
desk verdict Useful large-scale benchmark with a credible negative result, but the model-ranking claims are undermined by a mixed apples-to-oranges comparison (image-only VISTA3D vs multimodal others) and by missing confidence intervals. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a frozen-feature benchmark pipeline: each CT is center-cropped around the expert-delineated gross tumor volume, passed through one of four pretrained 3D foundation models (CT-FM, CT-CLIP, VISTA3D, SUPREM), and the output is global-average-pooled into an embedding vector. Three fusion functions—concatenation, gating, and attention—combine these imaging embeddings with clinical variables, and two training strategies are compared: training the predictor from scratch, or first applying an unsupervised few-shot adaptation based on ProtoNet. Their proposed Cox-like ProtoNet variant replaces the class centroid of the negative class with a sum over individual negative sample
What would settle it
If rerunning the pipeline with standardized preprocessing—same contour expansion, resampling, and cropping protocol for both cohorts—eliminates or reverses the external-cohort differences between VISTA3D and CT-CLIP, the claim that model choice drives generalization would be falsified.
Extended reading notes
Core claim
The authors claim that when four 3D CT foundation models are used as frozen feature extractors for two-year recurrence prediction in head-and-neck cancer, no model is reliably transferable across imaging distributions. VISTA3D achieves the best internal performance (AUC 0.668), while CT-CLIP, an image-text contrastively trained model, generalizes best to an external cohort (AUC 0.658 after Cox-like ProtoNet adaptation); the other models drop substantially, with AUC differences between internal and external cohorts ranging from 0.035 to 0.155. They further claim that a gating fusion of imaging and clinical features outperforms concatenation and attention, and that unsupervised few-shot pre-tr
Load-bearing premise
The benchmark assumes the feature-extraction pipeline is identical across datasets because every scan is center-cropped around an expert-drawn tumor contour and pooled the same way; if contouring or preprocessing conventions differ between the two cohorts, the generalization gap could be an artifact of extraction rather than a property of the models.
Editorial extensions
If this is right
- When building prognostic models from frozen 3D CT foundation models, gating-based fusion of imaging and clinical features should be preferred over concatenation or attention.
- Unsupervised few-shot pre-training of the predictor, particularly the Cox-like ProtoNet variant, improves external generalization for most foundation models tested.
- CT-CLIP's image-text pretraining appears to yield embeddings that transfer better across acquisition protocols than purely vision-based pretraining, though the authors leave the mechanism as an open question.
- Current VFM embeddings alone are insufficient for recurrence prediction: clinical variables consistently outperform imaging features, and most external F1-scores remain below 0.5.
- ROC AUC alone is insufficient for comparing models on imbalanced survival data, and balanced accuracy and F1-score should be reported alongside it.
Reading between the lines
- A direct test of the paper's implicit hypothesis would compare CT-CLIP against a vision-only model trained on the same text-paired data; if text-derived supervision is what drives external robustness, removing it should shrink the generalization advantage.
- The Cox-like ProtoNet variant approximates a Cox partial likelihood restricted to a binary endpoint, so extending it to continuous survival times with actual event ordering is a natural next step that could further improve risk stratification.
- The large external-cohort drops for VISTA3D and CT-FM may reflect predictor overfitting to RADCURE-specific feature statistics; a domain-adversarial head or test-time normalization could test whether the embeddings themselves are actually transferable.
- Because the pipeline assumes identical contouring, cropping, and pooling across datasets, harmonizing preprocessing between the two cohorts would isolate whether the measured generalization gap is a property of the models or an artifact of feature extraction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks four 3D CT foundation models (CT-FM, CT-CLIP, VISTA3D, SUPREM) for two-year recurrence prediction in head and neck cancer, using RADCURE (n=3,346) for training/validation/testing and Head-Neck-PET-CT (n=298) for external validation. The authors compare three fusion functions (concatenation, gating, attention) and two predictor training strategies (supervised from scratch and a proposed 'Cox-like ProtoNet' variant of few-shot prototype learning, which they label 'unsupervised'). Main claims are: (1) no single VFM dominates across all settings—VISTA3D achieves the best internal performance while CT-CLIP generalizes best externally; (2) external validation yields generally low AUCs (mostly <0.6), indicating limited generalization of current VFM embeddings; (3) gating is the best fusion function; and (4) prototype-based pre-training improves external performance for most VFMs. The manuscript includes two main result tables and reports AUC, balanced accuracy, and F1 averaged over 10 bootstraps.
Significance. If the findings are robust, this would be a useful empirical benchmark for the medical-imaging foundation-model community: it directly compares four recent VFMs on a clinically relevant task, uses two public datasets totaling 3,644 patients, and proposes a practical few-shot adaptation variant. The paper also commits to releasing code and extracted features, which would aid reproducibility. However, the central comparative claims about cross-dataset generalization and VFM ranking are currently undermined by a confounding in Table 1, by the absence of statistical uncertainty measures, by unverified comparability of the feature-extraction pipeline across datasets, and by several numeric inconsistencies between text and tables. These issues need to be resolved before the main conclusions can be accepted.
major comments (5)
- [Section 3.1 / Section 4] Cross-dataset comparability of VFM embeddings is not established, yet it is load-bearing for the claim that 'significant performance drops' reflect poor VFM generalization. The manuscript states that CTs are center-cropped around the GTV 'while respecting the input size requirements of each VFM' and that features are global-average-pooled. If the crop size, resampling, or GTV contouring conventions differ between RADCURE and HN-PETCT, the external-validation gap could be an artifact of preprocessing mismatch rather than a property of the VFM. Please report the exact crop sizes/resampling for each model and each dataset, and ideally perform a sensitivity analysis (e.g., common crop size or contour dilation) to demonstrate that the internal–external differences are not produced by the pipeline.
- [Table 1 / Section 5.1] The comparison of VFMs is confounded by input modality. In Table 1, VISTA3D is evaluated image-only (no 'multimodal' checkmark) while all other VFMs are evaluated with clinical data fused. The conclusion that 'VISTA3D features clearly outperform the other VFMs' on RADCURE is therefore not a like-for-like comparison of VFM embeddings. Moreover, Section 5.1 states that for VISTA3D 'adding clinical data decreases performance by 0.029 AUC, 0.026 BA and 0.029 F1', but no such multimodal VISTA3D row appears in Table 1. Please provide a fully crossed design (all VFMs × image-only/multimodal) or restrict ranking statements to matched settings.
- [Sections 5.1 and 5.3 / Table 1] Several numeric claims in the text are inconsistent with Table 1. In Section 5.3, the gains for CT-FM on HN-PETCT are reported as AUC +0.017 and F-score +0.015, but Table 1 shows the best pretrained AUC is 0.541 vs 0.539 without pretraining (+0.002) and F1 is 0.348 vs 0.349 (negative). Section 5.1 reports internal–external AUC differences of 0.021 (CT-CLIP), 0.035 (SUPREM), 0.123 (VISTA3D), and 0.155 (CT-FM); these appear to select different pretraining rows for different models (e.g., VISTA3D 'none' gives 0.123, CT-FM 'ProtoNet' gives 0.155, while SUPREM 'ProtoNet' gives 0.035). The reported differences are not systematically tied to a single strategy and are inconsistent with the table when a consistent choice is made. Please correct the text or clarify exactly which rows are used for each computed difference.
- [Section 3.3 / Abstract / Title] The method labeled 'unsupervised adaptation' is not unsupervised. Equations (1)–(3) explicitly use class labels to define positive/negative sets and to compute prototypes; the text itself says 'the class is used to group patients together.' This is a supervised few-shot/prototype-based adaptation method. The contradiction is material because 'unsupervised' appears in the title, abstract, and Section 5.3, and this is a core claimed contribution. Please rename the method (e.g., 'prototype-based adaptation' or 'few-shot adaptation') and adjust the title/abstract accordingly.
- [Section 4 / Results] No confidence intervals or significance tests are reported for any comparative claim. All metrics are 'averaged over 10 bootstraps,' but the paper does not report the bootstrap variance. Given that the key differences are often small (e.g., CT-CLIP external AUC 0.658 vs SUPREM 0.584; gating vs concatenation AUC +0.078), the conclusions 'clearly outperform,' 'better generalization,' and 'gating is clearly the most effective' are not statistically supported. Please provide 95% bootstrap intervals or paired significance tests for the central comparisons (VFM ranking, fusion function ranking, pretraining gains).
minor comments (5)
- [Figure 2] Figure 2 is garbled/unreadable in the manuscript text provided, with sequences such as '/uni00000013...' instead of readable labels. This makes it impossible to verify the visual comparison. Please regenerate the figure.
- [Section 2 / Table 1] The term 'multimodal' is used without explicit definition in Table 1. Clarify that the checkmark indicates fusion of imaging features with the listed clinical variables (age, sex, treatment, TNM, GTV dose). Without this, readers cannot interpret the row differences.
- [Section 4] The sentence 'Training patients are used for hyperparameters search and test patients are held out for evaluation' does not explain the role of the 499 validation samples. Please clarify how the validation set is used and whether any model selection was performed using it.
- [Section 2] The related work states that VFMs 'leverage unsupervised pretext tasks,' but VISTA3D and SUPREM are supervised segmentation models. Please phrase this more precisely to avoid conflating self-supervised and supervised pretraining.
- [Equation (3)] The term 'Cox-like' is an analogy rather than a formal derivation. If the surrogate loss is claimed to approximate the Cox partial likelihood, a short justification or reference would help; otherwise, the name may overstate the connection.
Circularity Check
No circularity found: the benchmark trains predictors on held-out data and evaluates them on internal test and external cohorts; no construction-equivalence or self-citation chain.
full rationale
The paper is an empirical benchmarking study with no circular derivation chain. VFMs are used as frozen feature extractors (Section 3.1), so their parameters are not fitted to the recurrence label. The downstream fusion module and classification head are trained on the RADCURE training fold and evaluated on the held-out RADCURE test split and the HN-PETCT external cohort (Section 4), which is a standard supervised evaluation design. The proposed 'Cox-like ProtoNet' variant (Eq. 3) is a modification of ProtoNet and is trained with the same training labels used to construct prototypes; calling it 'unsupervised' is a terminological overclaim, but it is not a logical circularity because the method does not derive the evaluation result from itself or from the test labels. The choice of CT-CLIP as the base configuration for Table 2 after observing its external performance in Table 1 introduces a selection bias concern, but this is a statistical validity issue rather than construction-equivalence. Similarly, possible preprocessing mismatch between RADCURE and HN-PETCT (center-crop around GTV, input-size requirements, pooling) is a threat to the generalizability conclusion, not a circularity. The paper's claims are empirical observations made after evaluation, not predictions forced by definition or by self-citation; all cited foundation models are external works, and no load-bearing argument reduces to authors' prior results. Therefore no significant circularity is present.
Assumptions & free parameters
free parameters (4)
- 2-year recurrence threshold =
2 years
- Majority undersampling ratio =
1:1
- Distance metric d in the prototype losses =
unstated (presumably Euclidean)
- Head/fusion hyperparameters =
lr=5e-5, dropout=0.5, 200 epochs
assumptions (5)
- domain assumption Global average pooling of the VFM encoder's last layer produces an embedding that preserves outcome-relevant information.
- domain assumption The two public datasets are comparable after applying inclusion criteria, and GTV contours are reliable across datasets.
- domain assumption A binarized 2-year recurrence endpoint, with censored patients excluded, adequately approximates recurrence-free survival.
- ad hoc to paper The Cox-like ProtoNet loss (Eq. 3) with instance-level negative distances approximates the Cox partial likelihood.
- domain assumption Frozen VFM features contain task-relevant information without fine-tuning.
Cite this review
Pith. "Pith review of Empirical investigation of 3D CT Foundation Models and Unsupervised Adaptation for Head and Neck Cancer Recurrence Prediction." pith.science (2026). https://pith.science/paper/J6LRP4VW
@misc{pith2026260800071,
author = {Pith},
title = {Pith review of: Empirical investigation of 3D CT Foundation Models and Unsupervised Adaptation for Head and Neck Cancer Recurrence Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/J6LRP4VW}},
note = {Machine review of arXiv:2608.00071}
}
read the original abstract
The rapid emergence of 3D CT foundation models has opened new avenues for predictive modeling from CT imaging, offering a compelling alternative to traditional radiomics which is known to suffer from reproducibility issues and sensitivity to acquisition protocol variations. Yet, as these models grow in availability, a critical need arises to evaluate how well their learned representations generalize across diverse clinical settings and whether adaptation to specific downstream tasks is necessary to unlock their full potential. To address these questions, we benchmarked several 3D CT foundation models for predicting recurrence-free survival in head and neck cancer across two public datasets totaling 3,644 patients, evaluating various adaptation strategies and modality fusion mechanisms. Our findings reveal persistent difficulty in identifying features that generalize consistently across different imaging distributions, as evidenced by significant performance drops on external validation cohorts. Ultimately, the integration of imaging features with clinical data remains the most accurate approach for prognostic prediction, though achieving universal generalization across varied clinical contexts continues to represent a substantial challenge for the current generation of models.
Figures
Reference graph
Works this paper leans on
-
[1]
Journal of Clinical Oncology37(18), 1578–1589 (2019) Empirical investigation of 3D CT Foundation Models 13
Adelstein, D.J., Ismaila, N., Ku, J.A., Burtness, B., Swiecicki, P.L., Mell, L., Beitler, J.J., Gross, N., Jones, C.U., Kaufman, M., et al.: Role of treatment dein- tensification in the management of p16+ oropharyngeal cancer: Asco provisional clinical opinion. Journal of Clinical Oncology37(18), 1578–1589 (2019) Empirical investigation of 3D CT Foundatio...
2019
-
[2]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Bae, J., Kapse, S., Zhou, L., Mani, K., Prasanna, P.: Hog-net: Hierarchical multi- organgraphnetworkforheadandneckcancerrecurrencepredictionfromctimages. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 317–327. Springer (2024)
2024
-
[3]
Physics and Imaging in Radiation Oncology25, 100404 (2023)
Berger, T., Noble, D.J., Yang, Z., Shelley, L.E., McMullan, T., Bates, A., Thomas, S., Carruthers, L.J., Beckett, G., Duffton, A., et al.: Assessing the generalisability of radiomics features previously identified as predictive of radiation-induced sticky saliva and xerostomia. Physics and Imaging in Radiation Oncology25, 100404 (2023)
2023
-
[4]
In: International conference on machine learning
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PmLR (2020)
2020
-
[5]
In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)
Cho, K., Van Merriënboer, B., Gulçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder–decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1724–1734 (2014)
2014
-
[6]
bmj 384(2024)
Collins, G.S., Dhiman, P., Ma, J., Schlussel, M.M., Archer, L., Van Calster, B., Harrell, F.E., Martin, G.P., Moons, K.G., Van Smeden, M., et al.: Evaluation of clinical prediction models (part 1): from development to external validation. bmj 384(2024)
2024
-
[7]
Trans- lational oncology14(1), 100906 (2021)
Haider, S.P., Sharaf, K., Zeevi, T., Baumeister, P., Reichel, C., Forghani, R., Kann, B.H., Petukhova, A., Judson, B.L., Prasad, M.L., et al.: Prediction of post- radiotherapy locoregional progression in hpv-associated oropharyngeal squamous cell carcinoma using machine-learning analysis of baseline pet/ct radiomics. Trans- lational oncology14(1), 100906 (2021)
2021
-
[8]
Nature Biomedical Engineering pp
Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Dogan, I., Durugol, O.F., Hou, B., Shit, S., et al.: Generalist foundation models from a multimodal dataset for 3d computed tomography. Nature Biomedical Engineering pp. 1–19 (2026)
2026
Show all 25 references
-
[9]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
He, Y., Guo, P., Tang, Y., Myronenko, A., Nath, V., Xu, Z., Yang, D., Zhao, C., Si- mon, B., Belue, M., et al.: Vista3d: A unified segmentation foundation model for 3d medical imaging. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 20863–20873 (2025)
2025
-
[10]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Hu, S.X., Li, D., Stühmer, J., Kim, M., Hospedales, T.M.: Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9068–9077 (2022)
2022
-
[11]
Nature reviews Clinical oncology14(12), 749–762 (2017)
Lambin, P., Leijenaar, R.T., Deist, T.M., Peerlings, J., De Jong, E.E., Van Tim- meren, J., Sanduleanu, S., Larue, R.T., Even, A.J., Jochems, A., et al.: Radiomics: the bridge between medical imaging and personalized medicine. Nature reviews Clinical oncology14(12), 749–762 (2017)
2017
-
[12]
Scientific reports12(1), 3183 (2022)
Le, W.T., Vorontsov, E., Romero, F.P., Seddik, L., Elsharief, M.M., Nguyen-Tan, P.F., Roberge, D., Bahig, H., Kadoury, S.: Cross-institutional outcome prediction for head and neck cancer patients using self-attention neural networks. Scientific reports12(1), 3183 (2022)
2022
-
[13]
Li, W., Yuille, A., Zhou, Z.: How well do supervised 3d models transfer to medical imaging tasks? arXiv preprint arXiv:2501.11253 (2025)
2025 arXiv
-
[14]
Cancers14(7), 1674 (2022) 14 B
Lv, W., Xu, H., Han, X., Zhang, H., Ma, J., Rahmim, A., Lu, L.: Context-aware saliency guided radiomics: Application to prediction of outcome and hpv-status from multi-center pet/ct images of head and neck cancer. Cancers14(7), 1674 (2022) 14 B. Guetarni et al
2022
-
[15]
arXiv preprint arXiv:2501.09001 (2025)
Pai, S., Hadzic, I., Bontempi, D., Bressem, K., Kann, B.H., Fedorov, A., Mak, R.H., Aerts, H.J.: Vision foundation models for computed tomography. arXiv preprint arXiv:2501.09001 (2025)
2025 arXiv
-
[16]
Radiation Oncology15(1), 81 (2020)
Rühle, A., Sprave, T., Kalckreuth, T., Stoian, R., Haehl, E., Zamboglou, C., Laszig, R., Knopf, A., Grosu, A.L., Nicolay, N.H.: The value of moderate dose escalation for re-irradiation of recurrent or second primary head-and-neck cancer. Radiation Oncology15(1), 81 (2020)
2020
-
[17]
Cancer Medicine 14(6), e70706 (2025)
Shannon, N.B., Lyer, N.G., Chua, M.L.K.: Leveraging artificial intelligence and ra- diomics for improved nasopharyngeal carcinoma prognostication. Cancer Medicine 14(6), e70706 (2025)
2025
-
[18]
Ad- vances in neural information processing systems30(2017)
Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. Ad- vances in neural information processing systems30(2017)
2017
-
[19]
Health Economics Review15(1), 84 (2025)
Sun, S., Lu, M., Wei, S., Liang, Y., Zhang, Z., Wang, H., Si, L.: Global burden and cross-country inequalities in head and neck cancer from 1992 to 2021: results from the global burden of disease study. Health Economics Review15(1), 84 (2025)
1992
-
[20]
Frontiers in oncology12, 974467 (2022)
Teng, X., Zhang, J., Ma, Z., Zhang, Y., Lam, S., Li, W., Xiao, H., Li, T., Li, B., Zhou, T., et al.: Improving radiomic model reliability using robust features from perturbations for head-and-neck carcinoma. Frontiers in oncology12, 974467 (2022)
2022
-
[21]
Scientific reports7(1), 10117 (2017)
Vallières, M., Kay-Rivest, E., Perrin, L.J., Liem, X., Furstoss, C., Aerts, H.J., Khaouam, N., Nguyen-Tan, P.F., Wang, C.S., Sultanem, K., et al.: Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Scientific reports7(1), 10117 (2017)
2017
-
[22]
Advances in neural information pro- cessing systems30(2017)
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)
2017
-
[23]
Medical Physics51(4), 3101–3109 (2024)
Welch, M.L., Kim, S., Hope, A.J., Huang, S.H., Lu, Z., Marsilla, J., Kazmierski, M., Rey-McIntyre, K., Patel, T., O’Sullivan, B., et al.: Radcure: An open-source head and neck cancer ct dataset for clinical radiation therapy insights. Medical Physics51(4), 3101–3109 (2024)
2024
-
[24]
The Cancer Imaging Archive4(2023)
Welch, M., Kim, S., Hope, A., Huang, S., Lu, Z., Marsilla, J., Kazmierski, M., Rey-McIntyre, K., Patel, T., O’Sullivan, B., et al.: Computed tomography images from large head and neck cohort (radcure). The Cancer Imaging Archive4(2023). https://doi.org/https://doi.org/10.7937/...
2023 doi
-
[25]
Scientific reports 6(1), 23428 (2016)
Zhao, B., Tan, Y., Tsai, W.Y., Qi, J., Xie, C., Lu, L., Schwartz, L.H.: Reproducibil- ity of radiomics for deciphering tumor phenotype with imaging. Scientific reports 6(1), 23428 (2016)
2016
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.