Pith. sign in

REVIEW 4 major objections 6 minor 60 references

Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multimodal transformer that fuses longitudinal blood tests, medication records, and CT-derived organ volumes predicts 3-, 6-, 9-, and 12-month mortality in immunotherapy patients, with the best variant reaching an AUC of 0.84 at 3 months.

desk verdict Solid incremental application paper whose headline improvement rests on test-set variant selection and a single-center cohort; worth a careful read, not a practice-changer. read the letter →

arxiv 2411.18253 v2 pith:QOBZTKD2 submitted 2024-11-27 cs.LG cs.AIq-bio.QM

Melda Yeghaian , Zuhir Bodalal , Daan van den Broek , John B A G Haanen , Regina G H Beets-Tan , Stefano Trebeschi , Marcel A J van Gerven This is my paper · ORCID
classification cs.LGcs.AIq-bio.QM
keywords artificialintelligencedeeplearningimmunotherapylongitudinalstudymultimodaldataintegrationsurvivalpredictiontemporalattentiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a deep learning model trained on routinely collected, noninvasive longitudinal data—blood tests, prescribed medications, and CT-derived organ volumes—can predict short-term mortality in cancer patients treated with immunotherapy. The proposed architecture, called MMTSimTA, embeds a simple temporal attention module inside a transformer encoder so that each modality is encoded separately and then fused at the feature level. In a pan-cancer cohort of 694 patients, the best variant reaches area under the receiver operating characteristic curve (AUC) values of 0.84, 0.83, 0.82, and 0.81 for 3-, 6-, 9-, and 12-month survival prediction, outperforming the baseline multimodal model at most endpoints and the best unimodal blood-based model at the earlier endpoints. If this holds, it would mean that data already collected in routine care could support personalized prognosis for immunotherapy patients without additional invasive procedures.

What carries the argument

The load-bearing object is the MMTSimTA network, which replaces the self-attention layer of a transformer encoder with the simple temporal attention (SimTA) module—an attention mechanism that linearly encodes the time intervals between examinations and assumes the most recent examination is the most informative. Each modality is processed by its own TSimTA block, the resulting representations are concatenated, optionally passed through a multi-head self-attention block, and then fed to a multilayer perceptron that jointly predicts all four survival endpoints; multimodal dropout lets the network train on patients with missing modalities.

What would settle it

Run the same MMTSimTA concat+SA and blood-only TSimTA models on an external cohort that includes patients with shorter follow-up, and compare their 3-month mortality AUCs; if the multimodal model does not beat the blood-only model, or its AUC drops substantially below 0.84, the claimed advantage of multimodal longitudinal integration fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that fusing longitudinal noninvasive modalities with a transformer-based temporal attention network yields better short-term mortality prediction than the baseline multimodal architecture or the best single modality. Concretely, the best variant—MMTSimTA with concatenated modality representations followed by a self-attention block—reaches AUCs of $0.84 \pm 0.04$, $0.83 \pm 0.02$, $0.82 \pm 0.02$, and $0.81 \pm 0.03$ for 3-, 6-, 9-, and 12-month survival prediction in a cohort of 694 immunotherapy-treated cancer patients. The paper interprets this as evidence that the added nonlinearity and skip connections of the transformer extension let the model exploit feature-level interactions between blood, medication, and imaging data, particularly for near-term endpoints.

Load-bearing premise

The model comparison assumes that the 280 patients excluded for having less than three months of follow-up—who differed significantly from the included 694 on several variables—would not overturn the observed ranking of models if they were kept in the study.

Editorial extensions

If this is right

  • The best MMTSimTA variant significantly outperforms the baseline MMSimTA at the 6-, 9-, and 12-month endpoints, so the transformer extension itself adds prognostic value beyond the original temporal attention module.
  • The multimodal model beats the best unimodal blood model at the 3- and 6-month endpoints, indicating that the added imaging and medication modalities contribute early prognostic signal rather than only noise.
  • Intermediate fusion outperforms late fusion for the extended transformer models, while the simpler baseline models do better with late fusion, suggesting that the right fusion strategy depends on the capacity of the unimodal encoders.
  • Feeding the model 6 months of on-treatment data instead of 3 months improves AUCs on the same patients, so longer observation windows strengthen mortality prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these results replicate in an external cohort, the architecture could support early mortality-risk stratification in routine oncology practice using data already collected for every patient, with no added invasive tests.
  • Because the SimTA module is hard-wired to weight the most recent examinations most heavily, a learnable temporal-attention variant might find informative patterns in earlier time points; testing that variant on the same cohort would clarify how much the recency prior matters.
  • The large gap between blood markers and organ-volume features suggests that replacing crude volume summaries with learned imaging features could change the size of the multimodal gain, a question the paper leaves open.
  • A time-to-event formulation that uses all available follow-up, rather than excluding patients with short follow-up, could directly test whether the acknowledged selection bias affects the reported model ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes MMTSimTA, a multimodal architecture that embeds SimTA temporal-attention blocks inside transformer encoder blocks, and evaluates it on a pan-cancer cohort of 694 immunotherapy-treated patients using longitudinal blood tests, CT-derived organ volumes, and medication records. The models predict 3-, 6-, 9-, and 12-month mortality as a multitask binary classification problem. Using 3-fold cross-validation, the authors compare unimodal SimTA/TSimTA, multimodal MMSimTA/MMTSimTA under intermediate fusion (Concat and Concat+SA variants), and a model-agnostic late-fusion strategy (MA-MMSimTA and MA-MMTSimTA). The headline result is that the best MMTSimTA variant (Concat+SA) attains mean AUCs of 0.84, 0.83, 0.82, and 0.81, with reported DeLong improvements over the best MMSimTA variant at 6-, 9-, and 12-month endpoints, and over late-fusion MMTSimTA at early endpoints. The authors acknowledge the exclusion of 280 patients and the lack of external validation as limitations.

Significance. If the reported improvements survive a properly controlled model-selection protocol, the paper would be a useful contribution to multimodal longitudinal prognosis: it extends SimTA with a transformer-style block, integrates three noninvasive modalities in a large real-world cohort, and compares intermediate versus late fusion in a systematic way. Strong points include the cohort size and modality coverage (11,249 blood tests, 14,849 medication records, 1,337 CT scans), the honest discussion of limitations, and the stated intention to release code. The significance is currently limited by evaluation design: the best variant appears to be selected on the same test folds used to report AUCs and p-values, and at least one table entry appears to contradict the abstract's 'strongest prognostic performance' claim. These issues need to be resolved before the conclusions can be taken at face value.

major comments (4)
  1. [Abstract / Results, Table 3] The abstract's claim that 'the strongest prognostic performance was demonstrated using a variant of the MMTSimTA model' is not supported by the results as reported. Table 3 lists MA-MMSimTA (late-fusion baseline) with mean AUCs of 0.85, 0.81, 0.81, and 0.81, which is higher than the MMTSimTA-Concat+SA values of 0.84, 0.83, 0.82, and 0.81 at the 3-month endpoint and equal at 12 months. The results text compares MMTSimTA with MMSimTA and with MA-MMTSimTA but does not report a pairwise test of MMTSimTA-Concat+SA versus MA-MMSimTA at 3 months; if that test is reported in Table S8, it should be moved to the main text, and the abstract should be qualified accordingly.
  2. [Model training and validation strategies / Results, Table 3] The selection of the 'best' variant appears to be made from test-fold AUCs. The same three test folds are used to choose among Concat, Concat+SA, and late-fusion configurations and then to compute the reported AUCs and DeLong p-values. No nested cross-validation, separate validation set, or multiple-comparison correction is described. Because the improvement of MMTSimTA-Concat+SA over MMSimTA-Concat+SA is small (0.01 at 3 months, 0.03 at 6 and 9 months, 0.03 at 12 months), part or all of the reported advantage could be due to selection. Please provide a nested variant-selection procedure or corrected p-values covering all variants and endpoints, and report the results for all variants rather than only the best.
  3. [Study cohort and ethical approval / Results, Patient characteristics] The exclusion of 280 patients with insufficient follow-up is acknowledged as a possible source of selection bias, and Table S2 shows significant differences between included and excluded patients on several variables. Because the task requires a 12-month follow-up window, the excluded patients are likely to have shorter survival and may be non-ignorably missing. The manuscript should include a sensitivity analysis (for example, inverse-probability weighting or a survival-analysis formulation that can censor the excluded patients) or, at minimum, a quantitative bound on how much the model ranking could shift under plausible outcome assumptions for the excluded group.
  4. [Statistical analysis] The statistical analysis section pools three per-fold DeLong p-values using Fisher's method but does not report effect sizes or confidence intervals for the AUC differences, and the 3-fold design has limited power. At 3 months the headline difference is not significant (P>.05), while at 6, 9, and 12 months the p-values (0.01, 0.001, 0.01) are obtained after the same test data have been used for variant selection. Reporting per-fold AUCs with confidence intervals and adjusting for the four endpoints (and the number of variants) would make the strength of the evidence easier to assess.
minor comments (6)
  1. [Abstract] The phrase 'with area under the curves (AUCs)' should be 'with areas under the receiver operating characteristic curves'; the plural form should be used consistently.
  2. [Statistical analysis] The relationship between the Mann-Whitney U test and the DeLong test is unclear; please state explicitly which test is used for which comparison and why both are needed.
  3. [Table 4] The 3-month row is labeled 'No death' and reports only three AUC values; the absence of a three-month value and the meaning of 'No death' should be explained in the table or text.
  4. [Figure 1] The abbreviations 'T' and 'N' in the figure are not defined in the figure legend; the legend should define these symbols.
  5. [Data availability] The manuscript states the code 'will be publicly accessible,' but the provided GitHub link is not resolved in the current version; please ensure the availability statement is accurate at the time of publication.
  6. [Tables and text] The notation 'Concat+SA' and 'Concat þ SA' is used interchangeably; please standardize the notation across the text, tables, and figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical evaluation is self-contained, and self-citations serve only as context and baselines.

full rationale

The paper makes no first-principles derivation claim; it reports an empirical evaluation of a neural architecture. MMTSimTA is constructed by combining the published SimTA module (ref 31) with a transformer encoder block (ref 43), and neither component is defined in terms of the survival endpoints. Survival labels are not used in the architecture definition, and reported AUCs are obtained from 3-fold stratified cross-validation on held-out folds with DeLong tests combined via Fisher's method; no reported performance measure is algebraically implied by the training objective or by any fitted parameter. The self-citations (refs 3, 34, 40, 41) provide methodological context and a prior late-fusion strategy, but the late-fusion baseline is implemented and evaluated rather than assumed, and ref 40 is a separate published study; no load-bearing premise rests solely on a self-citation. The Discussion explicitly acknowledges the main non-circular limitations: selection bias from excluding 280 patients (Table S2) and single-center data. A statistical concern exists that the 'best variant' labels in Table 3 and Figure 2 may be chosen post hoc from test-fold AUCs without correction for multiple comparisons, but that is an overfitting or selection-bias issue, not a definitional circularity: the reported performance is not equivalent to the model's inputs by construction. No circular step can be exhibited, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The empirical result rests on data preprocessing choices (missing-value thresholds), model hyperparameters, and domain assumptions about imaging segmentation, imputation, and the temporal attention prior. No new physical or biological entities are postulated.

free parameters (2)
  • Missing-value retention cutoffs = 50% for imaging structures, 60% for blood tests
    Structures with >=50% missing values and blood tests with >60% missing values were dropped; these thresholds are chosen by the authors and determine the input feature set.
  • SimTA/transformer hyperparameters = Not reported in main text
    Number of SimTA layers (3), transformer blocks, hidden dimensions, dropout rates, and learning rate are architecture choices that affect performance; details are deferred to supplementary material.
assumptions (4)
  • domain assumption The most recent examinations are the most informative for predicting survival.
    This is the core assumption of the SimTA module (Yang et al. 2020), used in all model variants; if clinically false, the temporal attention design would be suboptimal.
  • domain assumption TotalSegmentator accurately segments 117 anatomical structures in this CT cohort.
    Organ volumes are computed from automated segmentation; segmentation errors propagate directly into the imaging features.
  • domain assumption Missing blood marker values can be imputed from the training distribution.
    Multivariate iterative imputation is applied; informative missingness would bias the imputed features and the resulting predictions.
  • domain assumption Three-fold cross-validation on a single-center cohort provides a reliable estimate of generalization.
    Mean AUC over only 3 folds is used as the performance estimate; no external validation is performed, so between-center variability is unaccounted for.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning." pith.science (2026). https://pith.science/paper/QOBZTKD2

@misc{pith2026241118253,
  author       = {Pith},
  title        = {Pith review of: Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOBZTKD2}},
  note         = {Machine review of arXiv:2411.18253}
}
abstract

Purpose: Immunotherapies have revolutionized the landscape of cancer treatments. However, our understanding of response patterns in advanced cancers treated with immunotherapy remains limited. By leveraging routinely collected noninvasive longitudinal and multimodal data with artificial intelligence, we could unlock the potential to transform immunotherapy for cancer patients, paving the way for personalized treatment approaches. Methods: In this study, we developed a novel artificial neural network architecture, multimodal transformer-based simple temporal attention (MMTSimTA) network, building upon a combination of recent successful developments. We integrated pre- and on-treatment blood measurements, prescribed medications and CT-based volumes of organs from a large pan-cancer cohort of 694 patients treated with immunotherapy to predict mortality at three, six, nine and twelve months. Different variants of our extended MMTSimTA network were implemented and compared to baseline methods incorporating intermediate and late fusion based integration methods. Results: The strongest prognostic performance was demonstrated using a variant of the MMTSimTA model with area under the curves (AUCs) of $0.84 \pm $0.04, $0.83 \pm $0.02, $0.82 \pm $0.02, $0.81 \pm $0.03 for 3-, 6-, 9-, and 12-month survival prediction, respectively. Discussion: Our findings show that integrating noninvasive longitudinal data using our novel architecture yields an improved multimodal prognostic performance, especially in short-term survival prediction. Conclusion: Our study demonstrates that multimodal longitudinal integration of noninvasive data using deep learning may offer a promising approach for personalized prognostication in immunotherapy-treated cancer patients.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 56 canonical work pages

  1. [1]

    Harnessing multimodal data integration to advance precision oncology

    Boehm KM, Khosravi P, Vanguri R, Gao J, Shah SP. Harnessing multimodal data integration to advance precision oncology. Nat Rev Cancer. 2022;22:114-126

    work page 2022

  2. [2]

    Multimodal data fusion for cancer biomarker discovery with deep learning

    Steyaert S, Pizurica M, Nagaraj D, et al. Multimodal data fusion for cancer biomarker discovery with deep learning. Nat Mach Intell. 2023;5:351-362

    work page 2023

  3. [3]

    Dynamic Bayesian net - works as prognostic models for clinical patient management

    van Gerven MAJ, Taal BG, Lucas PJF. Dynamic Bayesian net - works as prognostic models for clinical patient management. J Biomed Inform. 2008;41:515-529

    work page 2008

  4. [4]

    Converg - ing on a cure: the roads to predictive immunotherapy

    Stein-O'Brien GL, Le DT, Jaffee EM, Fertig EJ, Zaidi N. Converg - ing on a cure: the roads to predictive immunotherapy. Cancer Dis - cov. 2023;13:1053-1057

    work page 2023

  5. [5]

    Artificial intelligence for predictive biomarker discovery in immuno-oncology: a systematic review

    Prelaj A, Miskovic V, Zanitti M, et al. Artificial intelligence for predictive biomarker discovery in immuno-oncology: a systematic review. Ann Oncol. 2024;35:29-65

    work page 2024

  6. [6]

    Novel clinical and radiomic predic - tors of rapid disease progression phenotypes among lung cancer patients treated with immunotherapy: an early report

    Tunali I, Gray JE, Qi J, et al. Novel clinical and radiomic predic - tors of rapid disease progression phenotypes among lung cancer patients treated with immunotherapy: an early report. Lung Can - cer. 2019;129:75-79

    work page 2019

  7. [7]

    Radiomics, tumor volume, and blood biomarkers for early prediction of pseudoprogression in patients with metastatic melanoma treated with immune check - point inhibition

    Basler L, Gabry �s HS, Hogan SA, et al. Radiomics, tumor volume, and blood biomarkers for early prediction of pseudoprogression in patients with metastatic melanoma treated with immune check - point inhibition. Clin Cancer Res. 2020;26:4414-4425

    work page 2020

  8. [8]

    Non-invasive decision support for NSCLC treatment using PET/CT radiomics

    Mu W, Jiang L, Zhang J, et al. Non-invasive decision support for NSCLC treatment using PET/CT radiomics. Nat Commun. 2020;11:5228

    work page 2020

Show all 60 references
  1. [9]

    Using machine learn - ing algorithms to predict immunotherapy response in patients with advanced melanoma

    Johannet P, Coudray N, Donnelly DM, et al. Using machine learn - ing algorithms to predict immunotherapy response in patients with advanced melanoma. Clin Cancer Res. 2021;27:131-140

    2021

  2. [10]

    Machine learning for prediction of survival outcomes with immune-checkpoint inhibitors in urothelial cancer

    Abuhelwa AY, Kichenadasse G, McKinnon RA, Rowland A, Hop - kins AM, Sorich MJ. Machine learning for prediction of survival outcomes with immune-checkpoint inhibitors in urothelial cancer. Cancers (Basel). 2021;13:2001

    2021

  3. [11]

    Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non- small cell lung cancer

    Vanguri RS, Luo J, Aukerman AT, MSK MIND Consortium, et al. Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non- small cell lung cancer. Nat Cancer. 2022;3:1151-1164

    2022

  4. [12]

    Combination of whole-body baseline CT radiomics and clinical parameters to predict response and survival in a stage-IV melanoma cohort undergoing immuno - therapy

    Peisen F, H €ansch A, Hering A, et al. Combination of whole-body baseline CT radiomics and clinical parameters to predict response and survival in a stage-IV melanoma cohort undergoing immuno - therapy. Cancers (Basel). 2022;14:2992

    2022

  5. [13]

    Deep learning reveals predictive sequence concepts within immune repertoires to immunotherapy

    Sidhom J-W, Oliveira G, Ross-MacDonald P, et al. Deep learning reveals predictive sequence concepts within immune repertoires to immunotherapy. Sci Adv. 2022;8:eabq5089

    2022

  6. [14]

    Cancer immunotherapy response prediction from multi-modal clinical and image data using semi- supervised deep learning

    Wang X, Jiang Y, Chen H, et al. Cancer immunotherapy response prediction from multi-modal clinical and image data using semi- supervised deep learning. Radiother Oncol. 2023;186:109793

    2023

  7. [15]

    A muti-modal feature fusion method based on deep learning for predicting immunotherapy response

    Li X, Feng X, Zhou J, et al. A muti-modal feature fusion method based on deep learning for predicting immunotherapy response. J Theor Biol. 2024;586:111816

    2024

  8. [16]

    Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data

    Chen Z, Chen Y, Sun Y, et al. Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data. Signal Transduct Target Ther. 2024;9:222

    2024

  9. [17]

    Predicting cancer out - comes from histology and genomics using convolutional networks

    Mobadersany P, Yousefi S, Amgad M, et al. Predicting cancer out - comes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018;115:E2970-E2979

    2018

  10. [18]

    Deep learning with multimodal representa - tion for pancancer prognosis prediction

    Cheerla A, Gevaert O. Deep learning with multimodal representa - tion for pancancer prognosis prediction. Bioinformatics. 2019;35: i446-i454

    2019

  11. [19]

    SALMON: survival analysis learning with multi-omics neural networks on breast cancer

    Huang Z, Zhan X, Xiang S, et al. SALMON: survival analysis learning with multi-omics neural networks on breast cancer. Front Genet. 2019;10:166

    2019

  12. [20]

    Overall sur - vival prediction of non-small cell lung cancer by integrating micro - array and clinical data with deep learning

    Lai Y-H, Chen W-N, Hsu T-C, Lin C, Tsao Y, Wu S. Overall sur - vival prediction of non-small cell lung cancer by integrating micro - array and clinical data with deep learning. Sci Rep. 2020;10:4679

    2020

  13. [21]

    Long-term cancer survival prediction using multimodal deep learning

    Vale-Silva LA, Rohr K. Long-term cancer survival prediction using multimodal deep learning. Sci Rep. 2021;11:13505

    2021

  14. [22]

    Multimodal deep learning for prognosis prediction in renal cancer

    Schulz S, Woerl A-C, Jungmann F, et al. Multimodal deep learning for prognosis prediction in renal cancer. Front Oncol. 2021;11:788740

    2021

  15. [23]

    Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis

    Chen RJ, Lu MY, Wang J, et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans Med Imaging. 2022;41:757-770

    2022

  16. [24]

    Survival prediction of brain cancer with incomplete radiology, pathology, genomic, and demographic data

    Cui C, Liu H, Liu Q, et al. Survival prediction of brain cancer with incomplete radiology, pathology, genomic, and demographic data. In: Medical Image Computing and Computer Assisted Interven - tion—MICCAI 2022. Springer Nature; 2022:626-635

    2022

  17. [25]

    Pan-cancer integrative histology-genomic analysis via multimodal deep learning

    Chen RJ, Lu MY, Williamson DFK, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell. 2022;40:865-878.e6

    2022

  18. [26]

    Early stage NSCLS patients’ prognos - tic prediction with multi-information using transformer and graph neural network model

    Lian J, Deng J, Hui ES, et al. Early stage NSCLS patients’ prognos - tic prediction with multi-information using transformer and graph neural network model. Elife. 2022;11:e80547

    2022

  19. [27]

    Zhou F, Xu Y, Cui Y, et al. 2004. iMD4GC: incomplete multimo - dal data integration to advance precise treatment response predic - tion and survival analysis for gastric cancer. arXiv [eess.IV], preprint: not peer reviewed. https://doi.org/10.48550/arXiv.2404. 01192

    2004 doi

  20. [28]

    Nikolaou N, Salazar D, RaviPrakash H, et al. 2024. Quantifying the advantage of multimodal data fusion for survival prediction in cancer patients, bioRxiv () 2024.01.08.574756, preprint: not peer reviewed

    2024

  21. [29]

    Song AH, Chen RJ, Jaume G, Vaidya AJ, Baras AS, Mahmood F

  22. [30]

    Multimodal data integration using deep learning predicts overall survival of patients with glioma

    Yuan Y, Zhang X, Wang Y, et al. Multimodal data integration using deep learning predicts overall survival of patients with glioma. View. 2024;5:20240001. 8 Journal of the American Medical Informatics Association, 2025, Vol. 00, No. 00 Downloaded from https://academic.oup.com/j...

    2024 doi

  23. [31]

    MIA-prognosis: a deep learning framework to predict therapy response

    Yang J, Chen J, Kuang K, Lin T, He J, Ni B. MIA-prognosis: a deep learning framework to predict therapy response. In: Medical Image Computing and Computer Assisted Intervention—MICCAI

  24. [32]

    Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer

    Khorrami M, Prasanna P, Gupta A, et al. Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer. Cancer Immunol Res. 2020;8:108-119

    2020

  25. [33]

    A multi-omics-based serial deep learning approach to predict clinical outcomes of single-agent anti- PD-1/PD-L1 immunotherapy in advanced stage non-small-cell lung cancer

    Yang Y, Yang J, Shen L, et al. A multi-omics-based serial deep learning approach to predict clinical outcomes of single-agent anti- PD-1/PD-L1 immunotherapy in advanced stage non-small-cell lung cancer. Am. J. Transl. Res. 2021;13:743-756

    2021

  26. [34]

    Prognostic value of deep learning-mediated treatment monitoring in lung cancer patients receiving immunotherapy

    Trebeschi S, Bodalal Z, Boellaard TN, et al. Prognostic value of deep learning-mediated treatment monitoring in lung cancer patients receiving immunotherapy. Front Oncol. 2021;11:609054

    2021

  27. [35]

    Imaging biomarkers to predict and evaluate the effectiveness of immunotherapy in advanced non- small-cell lung cancer

    Liu Y, Wu M, Zhang Y, et al. Imaging biomarkers to predict and evaluate the effectiveness of immunotherapy in advanced non- small-cell lung cancer. Front Oncol. 2021;11:657615

    2021

  28. [36]

    Delta radiomics model for the predic - tion of progression-free survival time in advanced non-small-cell lung cancer patients after immunotherapy

    Xie D, Xu F, Zhu W, et al. Delta radiomics model for the predic - tion of progression-free survival time in advanced non-small-cell lung cancer patients after immunotherapy. Front Oncol. 2022;12:990608

    2022

  29. [37]

    A short-term follow-up CT based radiomics approach to predict response to immunotherapy in advanced non-small-cell lung cancer

    Gong J, Bao X, Wang T, et al. A short-term follow-up CT based radiomics approach to predict response to immunotherapy in advanced non-small-cell lung cancer. Oncoimmunology. 2022;11:2028962

    2022

  30. [38]

    Radiomics and delta-radiomics signatures to predict response and survival in patients with non- small-cell lung cancer treated with immune checkpoint inhibitors

    Cousin F, Louis T, Dheur S, et al. Radiomics and delta-radiomics signatures to predict response and survival in patients with non- small-cell lung cancer treated with immune checkpoint inhibitors. Cancers (Basel). 2023;15:1968

    2023

  31. [39]

    Integration of lon - gitudinal deep-radiomics and clinical data improves the prediction of durable benefits to anti-PD-1/PD-L1 immunotherapy in advanced NSCLC patients

    Farina B, Guerra ADR, Bermejo-Pel �aez D, et al. Integration of lon - gitudinal deep-radiomics and clinical data improves the prediction of durable benefits to anti-PD-1/PD-L1 immunotherapy in advanced NSCLC patients. J Transl Med. 2023;21:174

    2023

  32. [40]

    Integrated non - invasive diagnostics for prediction of survival in immunotherapy

    Yeghaian M, Bodalal Z, Tareco Bucho TM, et al. Integrated non - invasive diagnostics for prediction of survival in immunotherapy. Immunooncol Technol. 2024;24:100723

    2024

  33. [41]

    Can blood- based markers predict RECIST progression in non-small cell lung cancer treated with immunotherapy? J Cancer Res Clin Oncol

    Yeghaian M, Tareco Bucho TM, de Bruin M, et al. Can blood- based markers predict RECIST progression in non-small cell lung cancer treated with immunotherapy? J Cancer Res Clin Oncol. 2024;150:329

    2024

  34. [42]

    Progress in serial imaging for prognostic stratification of lung cancer patients receiving immunotherapy: a systematic review and meta-analysis

    Chiu H-Y, Wang T-W, Hsu M-S, et al. Progress in serial imaging for prognostic stratification of lung cancer patients receiving immunotherapy: a systematic review and meta-analysis. Cancers (Basel). 2024;16:615

    2024

  35. [43]

    Attention is all you need

    Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;31:6000-6010

    2017

  36. [44]

    Multimodal learning with transformers: a survey

    Xu P, Zhu X, Clifton DA. Multimodal learning with transformers: a survey. IEEE Trans Pattern Anal Mach Intell. 2023;45:12113-12132

    2023

  37. [45]

    Transformers in time series: a sur - vey

    Wen Q, Zhou T, Zhang C, et al. Transformers in time series: a sur - vey. arXiv, 2022, preprint: not peer reviewed. https://doi.org/10. 48550/arXiv.2202.07125

    2022 doi arXiv

  38. [46]

    Multimodal data integration for oncology in the era of deep neural networks: a review

    Waqas A, Tripathi A, Ramachandran RP, Stewart PA, Rasool G. Multimodal data integration for oncology in the era of deep neural networks: a review. Front Artif Intell. 2024;7:1408843

    2024

  39. [47]

    TotalSegmentator: robust segmentation of 104 anatomic structures in CT images

    Wasserthal J, Breit H-C, Meyer MT, et al. TotalSegmentator: robust segmentation of 104 anatomic structures in CT images. Radiol Artif Intell. 2023;5:e230024

    2023

  40. [48]

    Scikit-learn: machine learning in python journal of machine learning research

    Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in python journal of machine learning research. J Mach Learn Res. 2011;12:2825-2830

    2011

  41. [49]

    PyTorch: an imperative style, high-performance deep learning library

    Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library. Adv Neural Inf Process Syst. 2019;32:abs/1912.01703

    2019 arXiv

  42. [50]

    Multimodal machine learn - ing: a survey and taxonomy

    Baltrusaitis T, Ahuja C, Morency L-P. Multimodal machine learn - ing: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell. 2019;41:423-443

    2019

  43. [51]

    ModDrop: adaptive multi-modal gesture recognition

    Neverova N, Wolf C, Taylor G, Nebout F. ModDrop: adaptive multi-modal gesture recognition. IEEE Trans Pattern Anal Mach Intell. 2016;38:1692-1706

    2016

  44. [52]

    Deep learning predicts lung can - cer treatment response from serial medical imaging

    Xu Y, Hosny A, Zeleznik R, et al. Deep learning predicts lung can - cer treatment response from serial medical imaging. Clin Cancer Res. 2019;25:3266-3275

    2019

  45. [53]

    Time-distanced gates in long short- term memory networks

    Gao R, Tang Y, Xu K, et al. Time-distanced gates in long short- term memory networks. Med Image Anal. 2020;65:101785

    2020

  46. [54]

    Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography, Proc

    Li TZ, Xu K, Gao R, et al. Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography, Proc. SPIE Int Soc Opt Eng. 2023;12464:1246412

    2023

  47. [55]

    ViSTA: a novel network improv - ing lung adenocarcinoma invasiveness prediction from follow-up CT series

    Zhao W, Sun Y, Kuang K, et al. ViSTA: a novel network improv - ing lung adenocarcinoma invasiveness prediction from follow-up CT series. Cancers (Basel). 2022;14:3675

    2022

  48. [56]

    Digital twins for predictive oncology will be a paradigm shift for precision cancer care

    Hernandez-Boussard T, Macklin P, Greenspan EJ, et al. Digital twins for predictive oncology will be a paradigm shift for precision cancer care. Nat Med. 2021;27:2065-2066

    2021

  49. [57]

    M3Care: learning with missing modalities in multimodal healthcare data

    Zhang C, Chu X, Ma L, et al. M3Care: learning with missing modalities in multimodal healthcare data. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery; 2022:2418-2428

    2022

  50. [58]

    Modality competi - tion: what makes joint training of multi-modal network fail in deep learning? (provably)

    Huang Y, Lin J, Zhou C, Yang H, Huang L. Modality competi - tion: what makes joint training of multi-modal network fail in deep learning? (provably). In: Chaudhuri K, Jegelka S, Song L, Sze - pesvari C, Niu G, Sabato S, eds. Proceedings of the 39th Interna - tional Conference ...

    2022

  51. [2020]

    Springer International Publishing; 2020:211-220

    2020

  52. [2024]

    https://doi.org/10.48550/ arXiv.2407.00224

    Multimodal prototyping for cancer survival prediction, arXiv, preprint: not peer reviewed. https://doi.org/10.48550/ arXiv.2407.00224

    doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.