REVIEW 3 major objections 6 minor 31 references
Comparing the Performance of Foundation Model Derived Embeddings with Traditional Approaches for Distant Metastasis Prediction in Head and Neck Cancer
T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A pretrained whole-volume CT foundation model predicts distant metastasis in head and neck cancer as accurately as a combination of radiomics and a deep-learning network, without requiring any tumor contours.
desk verdict A useful, honestly reported benchmark showing CT Foundation embeddings roughly match radiomics for HNC distant-metastasis prediction, but the 'contour-free' claim is undercut by the axial tumor-location cropping applied to the foundation input. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the CT Foundation model, a contrastive captioner (CoCa) that encodes CT volumes paired with radiology reports into a shared embedding space, outputting a 1408-dimensional vector per volume. This embedding is fed to a 4-layer MLP with clinical features concatenated after the third layer; the same MLP architecture is used for the comparison arms, which extract radiomics features (pyradiomics on GTVp masks, 1316 features) and ViT features (a from-scratch 10-layer Vision Transformer on 80x80x80 tumor-centered crops). The architecture isolates the feature-source comparison, since only the input features change while the classifier and training protocol stay fixed.
What would settle it
Run the identical foundation-embedding pipeline on the same dataset but with axial slices reduced to a fixed anatomical region that does not depend on the tumor contour (e.g., a standard mid-neck slab), and compare the AUC to the reported 0.791. If the AUC drops substantially toward chance or toward the ViT-only result, the conclusion that whole-volume, contour-free embeddings are responsible for the performance would be falsified.
Extended reading notes
Core claim
The paper's central claim is that CT Foundation embeddings, computed from full CT volumes with no manual GTVp contour, carry enough prognostic information to predict distant metastasis in head and neck cancer on par with the best contour-dependent pipelines. In their experiments, the foundation-embedding MLP achieved an ensemble test AUC of 0.791 [0.736, 0.846], higher than radiomics alone (0.772) and ViT features alone (0.753), and statistically indistinguishable from the radiomics+ViT combination (0.794, corrected p=0.558). The authors interpret this as evidence that a large-scale pretrained foundation model can serve as a drop-in feature extractor that bypasses segmentation, reduces train
Load-bearing premise
The load-bearing premise is that the axial reduction of the CT volumes following the tumor location does not leak tumor-location information to the foundation model, and that the cloud API's undisclosed pretraining data do not overlap with the RADCURE test set in a way that inflates performance.
Editorial extensions
If this is right
- If the result holds, outcome-prediction models for head and neck cancer can be built without manual segmentation, shortening the modeling pipeline and removing contouring variability as a source of bias.
- The near-equivalence with the radiomics+ViT combination suggests that a single foundation-embedding feature set may replace the need to blend handcrafted and learned features.
- Embedding-based features reduce the computational burden of training a 3D network from scratch, enabling CPU-only downstream training and broader deployment in resource-limited settings.
- The approach may extend to other tumor sites and other outcome endpoints where tumor contours are currently a bottleneck, provided a suitable CT foundation model is available.
- The reported performance on a large public cohort provides a baseline for future segmentation-free prognostic models.
Reading between the lines
- The paper's comparison is not perfectly clean: the foundation-model input was reduced axially following the tumor volume, which may implicitly encode tumor-location information that the radiomics and ViT arms receive explicitly through masks or crops; if so, some of the observed parity could be attributed to location leakage rather than to generic whole-volume processing.
- A direct test of this would be to run the foundation model on axially reduced volumes cropped away from the tumor, or on slices that have no tumor-location reference, and compare AUCs; a large drop would indicate location information is doing much of the work.
- The authors note that the CT Foundation API is a black-box cloud service whose pretraining data are not fully disclosed; if those data include head-and-neck CTs from a similar population or from the same institution era, the reported performance could reflect dataset overlap rather than generalizable feature quality.
- An external validation on a separate multi-institutional dataset, with and without tumor-location masking, would materially strengthen the claim that contour-free foundation embeddings are a general substitute for traditional feature engineering.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript compares CT Foundation embeddings against radiomics and a ViT trained from scratch for 2-year distant metastasis prediction in 2327 HNC patients from RADCURE. Three MLP classifiers are trained on four feature sets (foundation, radiomics, ViT, radiomics+ViT) plus SVM baselines; all are evaluated on the fixed RADCURE challenge test split. The reported test AUCs are 0.791 for foundation, 0.772 radiomics, 0.753 ViT, and 0.794 for radiomics+ViT; the corrected DeLong p-value between foundation and radiomics+ViT is 0.558. The authors conclude that contour-free foundation embeddings are a promising alternative to ROI-dependent feature extraction.
Significance. If the result holds, it is practically significant: it would suggest that a generic CT foundation model, with no contour input and a small downstream MLP, can match a combined radiomics+ViT model that requires GTVp contours. The study has strengths: public data, predefined external test split, 5-fold CV for model selection, per-fold results, calibration, DeLong tests with FDR correction, and subgroup analyses. However, the central comparison is currently confounded by an input-preparation asymmetry: the foundation arm uses a tumor-location-based axial reduction while the prose claims 'no cropping'. The central 'contour-free' conclusion is therefore not established by the presented experiments. With a controlled re-analysis, the manuscript would make a useful contribution.
major comments (3)
- [II.B, Table S.1, Fig. 1, Discussion §IV] The foundation arm is not actually contour-free. §II.B says volumes were 'reduced axially following the location of the tumor volume' before upload to the CT Foundation API, and Fig. 1 states 'the only input is an axial section of the CT volume'. This is a tumor-localized z-crop. Yet Discussion §IV says 'axial slices had no cropping applied' and Table S.1 lists 'Cropping: None' for Foundation. The ViT arm receives a GTVp-centered 80×80×80 crop, so the comparison conflates feature extractor with field-of-view. The reported parity (0.791 vs 0.794, p=0.558) could be driven by the foundation model receiving a tumor-focused sub-volume rather than by generic pretrained embeddings. Since the scalability/expertise-reduction conclusions depend on whole-volume, unlocalized input, this must be fixed: rerun the foundation arm on full CT volumes (or a z-slab chosen without GTVp location), and report
- [II.B / IV] The CT Foundation embeddings are produced by a black-box cloud API. The paper reports only that the model was trained on >500,000 multi-institutional CT images; there is no API version, checkpoint date, data card, or analysis of possible overlap of pretraining data with RADCURE. Because the test set is public and the API is closed, one cannot rule out that the reported AUC reflects pretraining on patients from the same population. This is a correctness risk for the 'generic foundation embeddings' claim. Please provide API version/data-provenance information, and/or replicate with a locally available foundation model (e.g., CT-FM) whose training data are known, or perform a near-duplicate/overlap check against the pretraining corpus.
- [III, Table 5] The statement that the foundation model had 'similar performance' to radiomics+ViT is based on a corrected one-sided DeLong p=0.558. Non-significance does not establish parity. Report a two-sided confidence interval for the AUC difference, or use an equivalence test with a pre-specified margin. Also, the claim that the foundation model 'outperformed' the ViT model should be softened, since the corrected p-value is 0.090 (not <0.05).
minor comments (6)
- [Figures 2–5] The captions appear swapped relative to the text: the text says MLP curves are in Figs. 2–3 and SVM curves in Figs. 4–5, while the captions label Figs. 2–3 as SVM and Figs. 4–5 as MLP.
- [Table 1] The '2-year Distant Failure' row reports 375 (12%), but 375/2327 is approximately 16%. Please correct the percentage.
- [II.B vs Table S.1] The text says the ViT produces 512 features, while Table S.1 lists 'Features to classifier' as 256. Clarify which is correct.
- [II.C / II.D] Minor typos: 'MLMP' should be 'MLP' in II.C; 'gridgrid search' in II.D; 'where comparble' and 'clariy' in the Discussion.
- [IV] The Discussion phrase 'axial slices had no cropping applied' directly contradicts §II.B's axial reduction step. Reword to describe the actual preprocessing or correct the method.
- [Table S.7] Subgroup AUCs are reported without confidence intervals. Add CIs or at least event counts per subgroup to support the claim of similar performance across sexes and contrast status.
Circularity Check
AUC comparison is not circular, but the 'contour-free/minimal preprocessing' claim is self-definitional because the foundation input was axially cropped using tumor location.
-
self definitional
[II.B Data preparation and feature extraction; cf. IV Discussion and Table S.1]
"The individual images were reduced axially, in order to save space when uploading the images to the cloud server. This reduction was done following the location of the tumor volume to prevent clipping off parts of the tumor, while keeping the 512x512 slices unprocessed."
The paper's central conclusion that CT Foundation embeddings are a 'minimally processed', contour-free alternative is made true by definition: the input is called 'whole volume'/'no cropping' even though the axial field of view was selected using the tumor volume's location, which is an ROI prior. The comparison against ViT/radiomics then attributes the result to generic foundation embeddings, while the foundation arm received a tumor-localized slab. The claimed advantage is therefore not derived from the method; it is built into the way 'no cropping' is defined.
full rationale
The MLP prediction itself is not circular: the CT Foundation embeddings are frozen, the MLP is trained on outcome labels, and the reported AUC is on a held-out test split, so the performance is not a refit of the labels. The self-citation to the RADCURE challenge test set is not load-bearing; it is a public benchmark choice. The genuine circularity-adjacent issue is the definition of the foundation input: §II.B states volumes were axially reduced following tumor location, while the Discussion and Table S.1 describe the foundation input as uncropped/whole volume. This makes the 'contour-free' and 'minimally processed' claims tautological rather than demonstrated, and it confounds the comparison because the foundation arm and ViT arm differ in field-of-view as well as feature extractor. That is a conceptual circularity in the claim, not a statistical refit of the prediction.
Assumptions & free parameters
assumptions (3)
- domain assumption The CT Foundation API's embeddings are an unbiased, well-calibrated representation of the CT volume, with no leakage of RADCURE test information into its pretraining.
- ad hoc to paper The axial reduction of CT volumes 'following the location of the tumor volume' does not itself leak the GTVp location into the foundation embeddings in a way that the ViT/radiomics arms do not also receive.
- domain assumption Clinical features (14 variables) are properly encoded and their inclusion does not conceal a missing-data bias; the <1% missing claim is taken at face value.
Cite this review
Pith. "Pith review of Comparing the Performance of Foundation Model Derived Embeddings with Traditional Approaches for Distant Metastasis Prediction in Head and Neck Cancer." pith.science (2026). https://pith.science/paper/B36QA6PL
@misc{pith2026260726276,
author = {Pith},
title = {Pith review of: Comparing the Performance of Foundation Model Derived Embeddings with Traditional Approaches for Distant Metastasis Prediction in Head and Neck Cancer},
year = {2026},
howpublished = {\url{https://pith.science/paper/B36QA6PL}},
note = {Machine review of arXiv:2607.26276}
}
read the original abstract
Background: Early prediction of distant metastasis (DM) risk in head and neck cancer (HNC) can enable timely interventions that may improve treatment outcomes. Many current machine learning methods rely on prior knowledge of the region of interest such as tumor segmentations, which require expert knowledge, is time-consuming and introduces user-dependent variability. Medical image-based foundation models have recently been developed for specific imaging modalities to streamline down-stream prediction tasks by extracting modality-relevant features. Purpose: In this study, we evaluate the effectiveness of using a foundation model as the feature extractor to predict DM risk in HNC patients and compare its performance with traditional approaches that require prior knowledge on the regions of interest. Methods: Preoperative CT images of 2327 patients from the RADCURE dataset were used. Three features-sets were created including radiomics, deep-learning based features, and CT Foundation derived features. The feature-sets were used individually in a multi-layer perceptron (MLP) to predict DM risk. Results: The model using CT Foundation embeddings outperformed the radiomics and deep learning-based models, achieving a Receiver Operating Characteristic Area Under the Curve (AUC) of 0.791, compared to AUC values of 0.772 and 0.753 for the radiomics and deep learning-based models, respectively. The CT Foundation based model had similar performance to a model that combined the use of radiomics and deep learning-based features that achieved an AUC of 0.794. Conclusions: Features based on foundation models offer a promising alternative to traditional radiomics while reducing the need for domain expertise and extensively annotated datasets. Their minimal preprocessing requirements also make them a more accessible and scalable option.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
and Burtness, Barbara and Leemans, C
Johnson, Daniel E. and Burtness, Barbara and Leemans, C. René and Lui, Vivian Wai Yan and Bauman, Julie E. and Grandis, Jennifer R. , title =. Nature Reviews Disease Primers , volume =. 2020 , type =. doi:10.1038/s41572-020-00224-3 , url =
-
[2]
and Airoldi, M
Pisani, P. and Airoldi, M. and Allais, A. and Aluffi Valletti, P. and Battista, M. and Benazzo, M. and Briatore, R. and Cacciola, S. and Cocuzza, S. and Colombo, A. and Conti, B. and Costanzo, A. and Della Vecchia, L. and Denaro, N. and Fantozzi, C. and Galizia, D. and Garzaro, M. and Genta, I. and Iasi, G. A. and Krengli, M. and Landolfo, V. and Lanza, G...
2020
-
[3]
Artificial Intelligence Review , volume =
Wiegrebe, Simon and Kopper, Philipp and Sonabend, Raphael and Bischl, Bernd and Bender, Andreas , title =. Artificial Intelligence Review , volume =. 2024 , type =. doi:10.1007/s10462-023-10681-3 , url =
-
[4]
Alabi, R. O. and Elmusrati, M. and Leivo, I. and Almangush, A. and Mäkitie, A. A. , title =. Int J Med Inform , volume =. doi:10.1016/j.ijmedinf.2024.105464 , year =
arXiv 2024
-
[5]
and Chang, Peter and Khatri, Deepak and Boockvar, John A
Zlochower, Avraham and Chow, Daniel S. and Chang, Peter and Khatri, Deepak and Boockvar, John A. and Filippi, Christopher G. , title =. Topics in Magnetic Resonance Imaging , volume =. 2020 , type =. doi:10.1097/rmr.0000000000000237 , url =
-
[6]
Mayerhoefer, M. E. and Materka, A. and Langs, G. and Häggström, I. and Szczypiński, P. and Gibbs, P. and Cook, G. , title =. J Nucl Med , volume =. doi:10.2967/jnumed.118.222893 , year =
-
[7]
Bogowicz, Marta and Tanadini-Lang, Stephanie and Guckenberger, Matthias and Riesterer, Oliver , title =. Scientific Reports , volume =. 2019 , type =. doi:10.1038/s41598-019-51599-7 , url =
-
[8]
and Wang, K
Chen, M. and Wang, K. and Wang, J. , title =. ArXiv , abstract =. 2024 , type =
2024
Show all 31 references
-
[9]
and Shiri, I
Toosi, A. and Shiri, I. and Zaidi, H. and Rahmim, A. , title =. Cancers (Basel) , volume =. doi:10.3390/cancers16142538 , year =
-
[10]
Medical Image Analysis , volume =
Zhang, Shaoting and Metaxas, Dimitris , title =. Medical Image Analysis , volume =. 2024 , type =. doi:https://doi.org/10.1016/j.media.2023.102996 , url =
2024
- [11]
- [12]
-
[13]
and Bernatz, Simon and Hosny, Ahmed and Mak, Raymond H
Pai, Suraj and Bontempi, Dennis and Hadzic, Ibrahim and Prudente, Vasco and Sokač, Mateo and Chaunzwa, Tafadzwa L. and Bernatz, Simon and Hosny, Ahmed and Mak, Raymond H. and Birkbak, Nicolai J. and Aerts, Hugo J. W. L. , title =. Nature Machine Intelligence , volume =. 2024 ,...
2024 doi
-
[14]
and Leem, S
Khan, W. and Leem, S. and See, K. B. and Wong, J. K. and Zhang, S. and Fang, R. , title =. IEEE Reviews in Biomedical Engineering , pages =. doi:10.1109/RBME.2025.3531360 , year =
2025
-
[15]
2024 , type =
Atilla Kiraly, Madeleine Traverse , title =. 2024 , type =
2024
-
[16]
Journal of Digital Imaging , volume =
Clark, Kenneth and Vendt, Bruce and Smith, Kirk and Freymann, John and Kirby, Justin and Koppel, Paul and Moore, Stephen and Phillips, Stanley and Maffitt, David and Pringle, Michael and Tarbox, Lawrence and Prior, Fred , title =. Journal of Digital Imaging , volume =. 2013 , ...
2013 doi
-
[17]
and Kim, Sejin and Hope, Andrew J
Welch, Mattea L. and Kim, Sejin and Hope, Andrew J. and Huang, Shao Hui and Lu, Zhibin and Marsilla, Joseph and Kazmierski, Michal and Rey-McIntyre, Katrina and Patel, Tirth and O'Sullivan, Brian and Waldron, John and Bratman, Scott and Haibe-Kains, Benjamin and Tadic, Tony an...
2024 doi
-
[18]
van Griethuysen, J. J. M. and Fedorov, A. and Parmar, C. and Hosny, A. and Aucoin, N. and Narayan, V. and Beets-Tan, R. G. H. and Fillion-Robin, J. C. and Pieper, S. and Aerts, Hjwl , title =. Cancer Res , volume =. doi:10.1158/0008-5472.Can-17-0339 , year =
-
[19]
Lin, C. Y. and Guo, S. M. and Lien, J. J. and Lin, W. T. and Liu, Y. S. and Lai, C. H. and Hsu, I. L. and Chang, C. C. and Tseng, Y. L. , title =. Radiol Med , volume =. doi:10.1007/s11547-023-01730-6 , year =
-
[20]
and Zhang, L
Wang, X. and Zhang, L. and Yang, X. and Tang, L. and Zhao, J. and Chen, G. and Li, X. and Yan, S. and Li, S. and Yang, Y. and Kang, Y. and Li, Q. and Wu, N. , title =. Eur J Radiol , volume =. doi:10.1016/j.ejrad.2020.109150 , year =
2020
-
[21]
Physics in Medicine & Biology , volume =
Chen, Liyuan and Zhou, Zhiguo and Sher, David and Zhang, Qiongwen and Shah, Jennifer and Pham, Nhat-Long and Jiang, Steve and Wang, Jing , title =. Physics in Medicine & Biology , volume =. 2019 , type =. doi:10.1088/1361-6560/ab083a , url =
2019 doi
-
[22]
2019 , type =
Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Köpf, Andreas and Yang, Edward and DeVito, Zach and Raison, Martin and...
-
[23]
2019 , type =
William Falcon, The PyTorch Lightning Team , title =. 2019 , type =. doi:10.5281/zenodo.3828935 , url =
2019 doi
-
[24]
DeLong, E. R. and DeLong, D. M. and Clarke-Pearson, D. L. , title =. Biometrics , volume =. 1988 , type =
1988
-
[25]
and Banerjee, A
Patel, M. and Banerjee, A. and Erickson, B. A. and Bovi, J. A. , title =. International Journal of Radiation Oncology, Biology, Physics , volume =. 2023 , type =. doi:10.1016/j.ijrobp.2023.06.1588 , url =
2023 doi
-
[26]
and Lo, Joseph Y
Draelos, Rachel Lea and Dov, David and Mazurowski, Maciej A. and Lo, Joseph Y. and Henao, Ricardo and Rubin, Geoffrey D. and Carin, Lawrence , title =. Medical Image Analysis , volume =. 2021 , type =. doi:https://doi.org/10.1016/j.media.2020.101857 , url =
2021
-
[27]
BMC Medical Imaging , volume =
Zhao, Wenzhe and Huang, Xin and Xu, Lei , title =. BMC Medical Imaging , volume =. 2025 , type =. doi:10.1186/s12880-025-01961-1 , url =
2025 doi
-
[28]
European Radiology , volume =
Fu, Yuanyuan and Hou, Runping and Qian, Liqiang and Feng, Wen and Zhang, Qin and Yu, Wen and Cai, Xuwei and Liu, Jun and Wang, Yu and Ding, Zhengping and Xu, Yujin and Zhao, Jun and Fu, Xiaolong , title =. European Radiology , volume =. 2025 , type =. doi:10.1007/s00330-025-11...
2025 doi
-
[29]
2024 , type =
Google-Health , title =. 2024 , type =
2024
-
[30]
, title =
Lemaître, Guillaume and Nogueira, Fernando and Aridas, Christos K. , title =. J. Mach. Learn. Res. , volume =. 2017 , type =
2017
-
[31]
and Peng, H
Ding, C. and Peng, H. , title =. J Bioinform Comput Biol , volume =. doi:10.1142/s0219720005001004 , year =
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.