CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction

{\AA}se Johnsson; Andreas Hallqvist; Ida H\"aggstr\"om; Jennifer Alv\'en; Jonas Andersson; Mikael Johansson; Sofie Allg\"ower

REVIEW 2 major objections 4 minor 19 references

CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction

T0 review · 2 major / 4 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen CT-CLIP backbone plus a small survival head beats clinical baselines for lung-cancer prognosis on a 242-patient cohort.

desk verdict Solid, careful application paper showing frozen CT-CLIP embeddings work for scarce-data lung-cancer survival; the N=50 test set makes the superiority claim fragile but does not erase the practical value. read the letter →

arxiv 2607.08503 v1 pith:5EMAOXFB submitted 2026-07-09 cs.CV

Sofie Allgöwer , Mikael Johansson , Andreas Hallqvist , Jonas Andersson , {AA}se Johnsson , Ida Häggström , Jennifer Alvén This is my paper · ORCID

classification cs.CV

keywords multimodalsurvivalanalysislungcancerCT-CLIPfoundationmodelsfrozenencodersDeepSurvprognosisprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Lung-cancer survival models are hard to train because large, well-curated imaging cohorts with reliable outcome labels are scarce. This paper shows that a domain-specific vision-language foundation model, CT-CLIP, already encodes useful prognostic information in its frozen image and text representations. By turning clinical variables into short template notes and feeding both the pretreatment CT volume and the note through the frozen CT-CLIP encoders, then training only a lightweight DeepSurv head on the concatenated embeddings, the authors obtain better discrimination than a pure clinical Cox model and match or exceed several task-specific multimodal baselines. Freezing the backbone also prevents the overfitting that appears when the same model is fully fine-tuned or adapted with LoRA on only 192 training cases. The resulting risk scores cleanly separate patients into high- and low-risk groups on Kaplan–Meier curves. The practical message is that pretrained medical foundation-model features can be used off-the-shelf for survival modelling when labelled data are limited.

What carries the argument

Frozen CT-CLIP dual encoders (CT-ViT image branch + CXR-BERT text branch) that map a preprocessed CT volume and a template-generated clinical note into a shared 1024-dimensional space; only the subsequent lightweight DeepSurv head is trained with the Cox partial log-likelihood.

What would settle it

On a larger independent lung-cancer cohort, replace the template notes with actual free-text clinical reports (or ablate the text branch entirely) and check whether the frozen multimodal C-index still significantly exceeds the pure clinical baseline.

Watch

Extended reading notes

Core claim

On a real-world cohort of 242 lung-cancer patients, a completely frozen CT-CLIP model whose 512-dimensional image and text embeddings are simply concatenated and passed through a two-layer DeepSurv head yields a held-out Harrell’s C-index of 0.755, outperforming the clinical CoxPH baseline (0.721) and matching or exceeding several trained multimodal CNN and fusion models, while also producing statistically significant high- versus low-risk separation.

Load-bearing premise

The short template sentences built from tabular clinical variables are close enough in style and content to the real radiology reports used to pre-train the text encoder that the frozen embeddings remain informative for survival.

Editorial extensions

If this is right

Hospitals with only a few hundred labelled CT cases can obtain competitive multimodal survival models by freezing a medical foundation model and training a small head.
Full fine-tuning or LoRA of large CT encoders is often unnecessary and can degrade performance under severe data scarcity.
Template-based conversion of tabular variables into natural-language notes is a viable way to reuse radiology-pretrained text encoders for survival tasks.
Risk scores from the frozen model can be used directly for clinically meaningful high/low-risk stratification without additional calibration steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same frozen-feature recipe may transfer to other scarce-label oncology tasks (response prediction, recurrence) that already possess CT-CLIP-compatible inputs.
If template notes already work, modest domain-adaptive pre-training of the text encoder on clinical notes rather than pure radiology reports could further close the remaining gap to larger multimodal models.
The observed superiority of frozen over fine-tuned configurations suggests that many medical foundation models are currently under-utilised as pure feature extractors in low-resource settings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing worth knowing is that a frozen CT-CLIP (image + templated clinical notes) plus a tiny DeepSurv head beats CoxPH and matches or edges the usual multimodal CNN baselines on a real 242-patient single-center cohort. That is new empirical evidence, not a new method.

What they do well: they actually run the adaptation ablations (frozen / full FT / LoRA), modality ablations, and several published multimodal baselines with the same protocol. The frozen configuration wins cleanly on their internal numbers (Harrell C 0.755 vs 0.721 for CoxPH), the Kaplan–Meier separation is clean, and they are honest that full fine-tuning overfits. The math is standard Cox partial likelihood; no circularity. Citations are appropriate and they correctly note that Xing et al. cannot be re-run.

Soft spots, in proportion. The held-out test set is only 50 patients (~39 events). The reported stds come from the three training folds, not from resampling the test set, so the Bonferroni stars and the “outperforms / comparable” language are sensitive to that particular split. That is a real limitation for any general claim about foundation-model representations, but it is the same limitation every scarce-data clinical paper faces; they do not hide it. The template-note assumption (that CXR-BERT still works on “Male lung cancer patient, stage IIA…”) is untested and secondary. No code, no external validation.

This is useful for groups sitting on 100–300-patient CT + tabular cohorts who want a drop-in feature extractor rather than training a 3-D ViT from scratch. It is not a methods advance and will not change how anyone designs foundation models. I would still send it to peer review: the experiment is clean enough and the practical question is real enough that referees should see it. Minor revision for clearer uncertainty quantification and a stronger caveat on the test-set size would be enough. Worth a look if you work in this niche; otherwise skim the tables and move on.

Referee Report

2 major / 4 minor

Summary. The paper evaluates CT-CLIP (a vision-language foundation model pretrained on chest CT volumes and radiology reports) as a feature extractor for multimodal overall-survival prediction on a private single-center cohort of 242 lung-cancer patients (pretreatment CT + clinical variables). Clinical variables are converted into template-generated notes so that the frozen CXR-BERT text encoder can be used; image and text embeddings are concatenated and fed to a lightweight DeepSurv head. Adaptation strategies (frozen, full fine-tuning, LoRA) and modality ablations are compared; the best configuration is a fully frozen CT-CLIP backbone plus trainable survival head. On a held-out test set of 50 patients this model reports Harrell’s C = 0.755 ± 0.006, Uno’s C = 0.744 ± 0.006 and 5-year AUC = 0.853 ± 0.016, outperforming CoxPH (0.721 ± 0.007) and matching or exceeding several published multimodal baselines, while also producing statistically significant high/low-risk Kaplan–Meier separation.

Significance. If the result holds under external validation, the work supplies a practical, low-parameter recipe for leveraging domain-specific foundation-model representations in data-scarce clinical survival settings: freeze the pretrained encoders, train only a small Cox head. This is directly relevant to real-world lung-cancer cohorts that rarely exceed a few hundred cases with reliable outcome labels. Strengths include a clear experimental design (adaptation ablations, modality ablations, multiple published multimodal baselines, Bonferroni-corrected tests) and an explicit demonstration that freezing outperforms full fine-tuning on this small cohort. The contribution is therefore of genuine applied interest even if the absolute performance numbers remain provisional.

major comments (2)

Table 3 and §3.3: the central claim that frozen CT-CLIP “outperforms the clinical baseline and achieves comparable or improved performance relative to other multimodal approaches” rests on a single held-out test set of N = 50 (≈39 events). The reported standard deviations are obtained from three training folds rather than from bootstrap or repeated test-set resampling; consequently the Bonferroni-corrected significance markers and the ranking versus ResNet+Tabular (0.757) and Interactive-Model (0.741) are highly sensitive to the particular 50-patient draw. At minimum the authors should supply bootstrap confidence intervals on the test set and, ideally, an external multi-center cohort before the superiority claim can be regarded as robust.
§2.3 and the modality-ablation results (Table 2): the text branch relies on the untested assumption that template-generated clinical notes (“Male lung cancer patient. The patient is diagnosed with adenocarcinoma, overall stage IIA …”) lie sufficiently close in distribution to the real radiology reports used to pre-train CXR-BERT. No quantitative check of embedding-space shift or ablation that replaces the templated notes with a conventional tabular encoder is provided. Because the multimodal gain over the image-only branch is modest (0.755 vs 0.741), this distributional assumption is load-bearing for the claim that the frozen text encoder contributes survival-informative features.

minor comments (4)

Figure 1 caption and §2.3: the exact template used to generate clinical notes is only partially illustrated; a complete example (including handling of missing variables) should be supplied in the appendix for reproducibility.
Table 1: the validation C-indices for frozen versus LoRA configurations are numerically close and overlapping; a short discussion of why full fine-tuning systematically under-performs would strengthen the interpretation of the frozen-backbone result.
§3.1: the precise definition of overall survival (time origin, censoring rules) and the distribution of follow-up times should be stated more explicitly, as they affect interpretation of the 5-year AUC.
References: the preprint arXiv:2403.17834 (CT-CLIP) and the PEFT library citation are appropriate, but a brief note on the exact checkpoint (CT-CLIP_v2.pt) and any preprocessing differences from the original CT-RATE pipeline would aid replication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical evaluation of an external foundation model with standard Cox survival training and held-out metrics.

full rationale

The paper is a standard machine-learning evaluation study. CT-CLIP (vision and text encoders plus projections) is taken as a fixed external pretrained model (Hamamci et al., CT-RATE). Representations are extracted, optionally adapted via freeze/full/LoRA, concatenated, and fed to a lightweight DeepSurv head whose parameters are optimized by the ordinary Cox partial log-likelihood on the training folds. Discrimination (Harrell’s C, Uno’s C, 5-year AUC) and Kaplan–Meier risk-group separation are then measured on a held-out test set of 50 patients and compared with independent baselines (CoxPH, ResNet+Tabular, DAFT, FiLM, Interactive-Model). No quantity is defined in terms of a later-reported prediction; no uniqueness theorem or ansatz is imported from the present authors; no self-citation is load-bearing for the central claim. The only modeling choice that could be questioned (template-generated clinical notes) is an explicit distributional assumption, not a circular reduction. Consequently the derivation chain is self-contained against external benchmarks and contains no circular steps.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard survival-analysis machinery, the transferability of a publicly released foundation model, and a handful of modelling choices (template notes, Cox partial likelihood, frozen encoders). No new physical entities or free parameters are invented to force the result; the free parameters are ordinary ML hyper-parameters selected by validation C-index.

free parameters (4)

learning rate
Chosen by grid search over five values on the three validation folds; final value not reported but directly affects the selected checkpoint.
LoRA rank r=8 and alpha=16
Fixed by reference to prior LoRA papers rather than re-tuned; used in all LoRA runs.
DeepSurv head widths (512, 256) and dropout 0.25
Architectural choices fixed a priori; only the head is trained in the best model.
median risk threshold for high/low stratification
Used to produce Kaplan–Meier curves; any other quantile would change the visual separation.

assumptions (4)

domain assumption Cox proportional-hazards partial log-likelihood is an appropriate training objective for overall survival
Invoked throughout §2.1 and used as the sole loss; alternatives (DeepHit, etc.) are mentioned but not compared.
ad hoc to paper Template-generated clinical notes are distributionally close enough to CT-RATE radiology reports for the frozen CXR-BERT encoder to remain informative
Stated explicitly in §2.3; never empirically verified against real reports.
domain assumption CT-CLIP vision and text encoders pretrained on CT-RATE transfer to pretreatment diagnostic CTs of a Swedish lung-cancer cohort
Core premise of the frozen-encoder experiments (§3.2–3.3).
standard math Stratified 80/20 split plus 3-fold CV on 192 training patients yields unbiased estimates of generalization
Standard ML practice; small absolute N still leaves residual optimism risk.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction." pith.science (2026). https://pith.science/paper/5EMAOXFB

@misc{pith2026260708503,
  author       = {Pith},
  title        = {Pith review of: CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EMAOXFB}},
  note         = {Machine review of arXiv:2607.08503}
}

read the original abstract

Accurate prognosis prediction is important for treatment planning in lung cancer, but deep learning-driven survival modelling is often limited by the scarcity of curated imaging cohorts with reliable outcome data. This study evaluates whether representations from a domain-specific foundation model can be used for multimodal survival prediction in data-constrained clinical settings. We assess the foundation model CT-CLIP as a feature extractor for pretreatment computed tomography images and clinical variables from 242 diagnosed lung cancer patients. The evaluation includes adaptation strategies based on frozen encoders, full fine-tuning, and low-rank adaptation, together with modality ablations and comparisons with clinical and multimodal baselines. The results show that a frozen CT-CLIP model combined with a trainable lightweight survival head outperforms the clinical baseline and achieves comparable or improved performance relative to other multimodal approaches, and separates patients into clinically meaningful high- and low-risk groups.

Figures

Figures reproduced from arXiv: 2607.08503 by the authors.

**Figure 1.** The CT-CLIP survival framework. ⊕ represents concatenation. 3 Experiments and Results 3.1 Dataset We curate a dataset comprising pretreatment CT scans and associated clinical variables from 242 patients diagnosed with lung cancer during the years 2008- 2018, collected at the University Hospital of Umeå, Sweden. The study was approved by the Swedish Ethical Review Authority (Dnr: 2021-06549-01). Outcome data are col… view at source ↗

**Figure 2.** Kaplan-Meier curves for CoxPH and CT-CLIP from the best fold. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

[1]

In: European conference on computer vision

Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: European conference on computer vision. pp. 1–21. Springer (2022)

work page 2022
[2]

CA: a cancer journal for clinicians 74(3), 229–263 (2024)

Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R.L., Soerjomataram, I., Je- mal, A.: Global cancer statistics 2022: Globocan estimates of incidence and mor- tality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 74(3), 229–263 (2024)

work page 2022
[3]

Journal of the Royal Statistical So- ciety: Series B (Methodological)34(2), 187–202 (12 2018)

Cox, D.R.: Regression models and life-tables. Journal of the Royal Statistical So- ciety: Series B (Methodological)34(2), 187–202 (12 2018)

work page 2018
[4]

In: International conference on medical image com- puting and computer-assisted intervention

Duanmu, H., Huang, P.B., Brahmavar, S., Lin, S., Ren, T., Kong, J., Wang, F., Duong, T.Q.: Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using deep learning with integrative imaging, molec- ular and demographic data. In: International conference on medical image com- puting and computer-assisted intervention. pp...

work page 2020
[5]

In: European Conference on Com- puter Vision

Hamamci, I.E., Er, S., Sekuboyina, A., Simsar, E., Tezcan, A., Simsek, A.G., Esirgun, S.N., Almas, F., Doğan, I., Dasdelen, M.F., et al.: Generatect: Text- conditional generation of 3d chest ct volumes. In: European Conference on Com- puter Vision. pp. 126–143. Springer (2024)

work page 2024
[6]

E., Er, S., Almas, F., Simsek, A

Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Do- gan, I., Durugol, O.F., Hou, B., Shit, S., et al.: Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834 (2024)

work page arXiv 2024
[7]

Iclr1(2), 3 (2022)

Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr1(2), 3 (2022)

work page 2022
[8]

BMC medical research methodology18(1), 24 (2018)

Katzman, J.L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., Kluger, Y.: Deep- surv:personalizedtreatmentrecommendersystemusingacoxproportionalhazards deep neural network. BMC medical research methodology18(1), 24 (2018)

work page 2018

Show all 19 references

[9]

In: International Conference on Medical Image Computing and Computer- Assisted Intervention

Kim, K., Lee, Y., Park, D., Eo, T., Youn, D., Lee, H., Hwang, D.: Llm-guided multi-modal multiple instance learning for 5-year overall survival prediction of lung cancer. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 239–249. ...

2024
[10]

In: Proceedings of the AAAI conference on artificial intelligence

Lee, C., Zame, W., Yoon, J., Van Der Schaar, M.: Deephit: A deep learning ap- proach to survival analysis with competing risks. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

2018
[11]

https://github.com/huggingface/peft (2022)

Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., Bossan, B., Tietz, M.: PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft (2022)

2022
[12]

Nature machine intelligence2(5), 274–282 (2020)

Mukherjee, P., Zhou, M., Lee, E., Schicht, A., Balagurunathan, Y., Napel, S., Gillies, R., Wong, S., Thieme, A., Leung, A., et al.: A shallow convolutional neural network predicts prognosis of lung cancer patients in multi-institutional computed tomography image datasets. Natu...

2020
[13]

In: Proceedings of the AAAI conference on artificial intelligence

Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: Visual rea- soning with a general conditioning layer. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32, pp. 3942–3951 (2018) 10 F. Author et al

2018
[14]

In: International conference on machine learning

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

2021
[15]

NeuroImage260, 119505 (2022)

Wolf, T.N., Pölsterl, S., Wachinger, C., Initiative, A.D.N., et al.: Daft: A universal module to interweave tabular data and 3d images in cnns. NeuroImage260, 119505 (2022)

2022
[16]

In: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

Wu, Y., Ma, J., Huang, X., Ling, S.H., Su, S.W.: Deepmmsa: A novel multimodal deep learning method for non-small cell lung cancer survival analysis. In: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC). pp. 1468–1472. IEEE (2021)

2021
[17]

In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention

Xing, Q., Song, Z., Gong, B., Yang, L., Yu, J., Yang, W.: Cross-modality masked learning for survival prediction in ici treated nsclc patients. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 136–146. Springer (2025)

2025
[18]

Radiotherapy and oncology180, 109483 (2023)

Zheng, S., Guo, J., Langendijk, J.A., Both, S., Veldhuis, R.N., Oudkerk, M., van Ooijen, P.M., Wijsman, R., Sijtsema, N.M.: Survival prediction for stage i-iiia non- small cell lung cancer using deep learning. Radiotherapy and oncology180, 109483 (2023)

2023
[19]

In: 2016 IEEE international conference on bioinformatics and biomedicine (BIBM)

Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE international conference on bioinformatics and biomedicine (BIBM). pp. 544–547. IEEE (2016)

2016

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.

[1] [1]

In: European conference on computer vision

Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: European conference on computer vision. pp. 1–21. Springer (2022)

work page 2022

[2] [2]

CA: a cancer journal for clinicians 74(3), 229–263 (2024)

Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R.L., Soerjomataram, I., Je- mal, A.: Global cancer statistics 2022: Globocan estimates of incidence and mor- tality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 74(3), 229–263 (2024)

work page 2022

[3] [3]

Journal of the Royal Statistical So- ciety: Series B (Methodological)34(2), 187–202 (12 2018)

Cox, D.R.: Regression models and life-tables. Journal of the Royal Statistical So- ciety: Series B (Methodological)34(2), 187–202 (12 2018)

work page 2018

[4] [4]

In: International conference on medical image com- puting and computer-assisted intervention

Duanmu, H., Huang, P.B., Brahmavar, S., Lin, S., Ren, T., Kong, J., Wang, F., Duong, T.Q.: Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using deep learning with integrative imaging, molec- ular and demographic data. In: International conference on medical image com- puting and computer-assisted intervention. pp...

work page 2020

[5] [5]

In: European Conference on Com- puter Vision

Hamamci, I.E., Er, S., Sekuboyina, A., Simsar, E., Tezcan, A., Simsek, A.G., Esirgun, S.N., Almas, F., Doğan, I., Dasdelen, M.F., et al.: Generatect: Text- conditional generation of 3d chest ct volumes. In: European Conference on Com- puter Vision. pp. 126–143. Springer (2024)

work page 2024

[6] [6]

E., Er, S., Almas, F., Simsek, A

Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Do- gan, I., Durugol, O.F., Hou, B., Shit, S., et al.: Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834 (2024)

work page arXiv 2024

[7] [7]

Iclr1(2), 3 (2022)

Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr1(2), 3 (2022)

work page 2022

[8] [8]

BMC medical research methodology18(1), 24 (2018)

Katzman, J.L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., Kluger, Y.: Deep- surv:personalizedtreatmentrecommendersystemusingacoxproportionalhazards deep neural network. BMC medical research methodology18(1), 24 (2018)

work page 2018