REVIEW 5 major objections 5 minor 27 references
Can artificial intelligence predict clinical trial outcomes?
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that GPT-4o is the best large language model tested for predicting clinical trial outcomes, yet every LLM tested overlooks most failed trials, while the HINT model uniquely catches negative outcomes.
desk verdict The central comparison is invalidated by a reversed temporal split that lets LLMs memorize outcomes; the abstract and main table also disagree on the headline numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is two families of predictors. The LLM machinery is a prompt: each model receives the trial title, summary, conditions, interventions, primary outcome measures, and trial type, and must answer 'success' or 'failure' in one word. The HINT machinery is a hierarchical interaction network, a dynamic-attention graph neural network that embeds drug SMILES, disease ICD-10 codes, and eligibility criteria and outputs a success probability, with 0.5 as the threshold. The evaluation machinery is the metric set of balanced accuracy, Matthews correlation coefficient, recall, and specificity, chosen because the dataset is imbalanced at roughly 79 percent successful trials. A final piece of machinery is cutoff-based data selection: each model is tested only on trials whose primary completion date precedes that model's training cutoff, which the authors assume prevents outcome memorisation.
What would settle it
Re-run the same evaluations on trials whose outcomes were not public before each model's training cutoff, for example trials completed after the cutoff, and ask whether GPT-4o's balanced-accuracy lead over HINT persists. A cheap consistency check is to rerun the exact 74-trial GPT-4o evaluation: the abstract reports a balanced accuracy of 0.573 and an MCC of 0.212, while Table 1 reports 0.532 and 0.098, so only one set of numbers can reproduce.
Extended reading notes
Core claim
The study's central discovery is that no single model predicts clinical trial outcomes well, but the models fail in opposite directions. GPT-4o, the best LLM, is reported to reach a balanced accuracy of 0.573 and an MCC of 0.212 in the abstract, because it classifies most trials as successful and catches few negative outcomes. HINT, by contrast, has the highest specificity (0.473) and a balanced accuracy of 0.525, making it the only model that can reliably recognise failed trials. The paper further finds that LLMs are strongest on Phase I trials and Overall Survival endpoints, while HINT is more stable across phases and handles complex endpoints such as Objective Response Rate, but underperforms on oncology trials, which make up 30 to 37 percent of the data. The authors conclude that the two families are complementary and advocate hybrid systems rather than either model alone.
Load-bearing premise
The load-bearing premise is that limiting each model's test set to trials whose primary completion date falls before that model's training cutoff prevents data leakage; in reality, outcomes for those trials were public before the cutoff, so the models may have memorised them rather than predicted them.
Editorial extensions
If this is right
- If the complementarity claim holds, a combined system that uses GPT-4o for overall scoring and HINT's negative prediction as a filter would flag failed trials that pure-LLM pipelines miss.
- If the LLMs' near-zero specificity is a general property, any LLM-based trial screening tool needs an explicit negative-class mechanism or calibration before it can support go/no-go decisions.
- If HINT's oncology shortfall is intrinsic, hybrid designs for cancer trials should weight LLM predictions more heavily for neoplasms, the opposite of the overall pattern.
- If the phase and duration trends are real, deployment should be stratified: use LLMs for early-phase and simple-endpoint decisions and HINT for later-phase and complex-endpoint decisions.
Reading between the lines
- The authors stop at advocating hybrids; I would test a concrete two-stage design in which GPT-4o screens every trial and every trial it labels 'success' is re-checked by HINT, using HINT's specificity as a go/no-go gate for the most expensive decisions.
- A prospective test that the paper does not run would collect trials whose results are posted after the paper's data-collection date; if GPT-4o's edge shrinks, the leaderboard here would reflect memorisation more than prediction.
- The abstract-to-Table 1 discrepancy in GPT-4o's numbers (0.573 versus 0.532 balanced accuracy, 0.212 versus 0.098 MCC) is my main reason to treat the exact ranking as provisional until one set of numbers is rerun and confirmed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript compares several large language models (GPT-3.5, GPT-4, GPT-4o, GPT-4mini, Llama3) with the HINT deep learning model on the task of predicting clinical trial success versus failure, using trials drawn from ClinicalTrials.gov. The central claims are that GPT-4o achieves the best overall performance among LLMs, that all LLMs struggle to identify negative outcomes (low specificity), and that HINT has high specificity and therefore complements LLMs, motivating hybrid approaches. Performance is reported through balanced accuracy, MCC, recall, and specificity, with subgroup analyses by trial phase, disease category, endpoint, and trial duration.
Significance. If the evaluation were valid, the paper would provide a practically useful benchmark and an evidence-based argument for hybrid LLM-plus-HINT systems in clinical trial outcome forecasting. The question is important and the paper includes multiple metrics and several clinically relevant stratifications. However, the central comparison rests on a temporal split that is reversed: test trials are selected with primary completion dates before each model's training cutoff, so LLM predictions may reflect memorization of pretraining data rather than out-of-sample forecasting. The paper also does not release code or a complete dataset, and one headline metric is internally inconsistent. The claims are therefore not currently supported.
major comments (5)
- [Section 3.1, Fig. 2A] The evaluation design introduces a reversed temporal split. Section 3.1 states that 'we restricted the primary completion date of the trials according to the cut-off dates corresponding to the training periods of different models,' so the test set consists of trials completed before each model's training cutoff. For such trials, the outcome was publicly known before the cutoff, and the prompt provides the trial title and summary, making it possible for the LLM to retrieve the outcome from pretraining data. This directly threatens the headline comparison: the GPT-4o balanced accuracy of 0.573 and MCC of 0.212 in the Abstract and Section 2.1, the 'LLMs struggle with negative outcomes' pattern, and the HINT specificity advantage could all be artifacts of memorized labels rather than predictive skill. A proper evaluation would use trials whose outcomes were not available at the model's cutoff, e.g., primary completion or results disclosure after the cutoff.
- [Section 2.6 and Section 3.3] Terminated trials are excluded after observing poor LLM performance on them, which is outcome-based sample selection. Section 2.6 reports that LLMs achieved accuracy around 0.1 or lower on terminated trials and then says, 'Consequently, we excluded “terminated” trials from further analyses to avoid biasing the experimental results.' Section 3.3 similarly states, 'In subsequent analyses, we excluded data with a “terminated” status because we found that the LLMs lacked the ability to accurately recognize trial termination.' This post hoc removal of a difficult and clinically important subset inflates the reported metrics and changes the evaluation population. If termination prediction is a goal, these trials should be included; if not, the exclusion criteria should be specified before the analysis, not in response to the results.
- [Abstract, Section 2.1, Table 1] The headline results for GPT-4o are internally inconsistent. The Abstract states GPT-4o achieves a balanced accuracy of 0.573 and MCC of 0.212, and Section 2.1 repeats these values with recall 0.931 and specificity 0.214. However, Table 1 reports GPT-4o balanced accuracy 0.532, MCC 0.098, recall 0.913, and specificity 0.150. These discrepancies mean the reader cannot determine which numbers support the central claim, and the abstract's numbers do not match the paper's own results table.
- [Section 3.1, Table 1] The models are evaluated on non-identical test sets of very different sizes, which confounds the cross-model comparison. Section 3.1 reports 3,811 trials for GPT-3.5, 2,304 for HINT, 725 for Llama3, and only 54 and 74 for GPT-4 and GPT-4o/GPT-4mini, respectively, using different cutoff dates. Consequently, the disease distribution, outcome prevalence, and trial difficulty differ across model columns, as visible in Fig. 1A and Table 3. The claim that 'GPT-4o achieves superior overall performance among LLMs' is not supportable when each model is scored on a different test population.
- [Section 2.3, Table 3] Several subgroup claims are based on extremely small sample sizes. For GPT-4o, disease categories have between 2 and 18 trials, and endpoint categories between 5 and 28 trials, yet the text makes specific claims such as GPT-4o's specificity of 1.000 in respiratory diseases (3 trials) and 0.500 in digestive diseases (3 trials). These estimates have very large uncertainty, and the statement that GPT-4o 'displayed strong recognition of negative samples' in these categories is not statistically grounded. The paper should at least report confidence intervals or avoid strong claims for cells with fewer than, say, 20 trials.
minor comments (5)
- [Section 3.1] The paragraph ends with an incomplete sentence: 'For most of our analyses, we retained clinical trial data with the status' is followed by no completion; the intended status filter should be stated.
- [Title page and Abstract] There are typographical errors: 'Singpaore' on the title page and 'We underscores' in the Abstract should be 'We underscore'.
- [Fig. 2 caption] The Figure 2 caption labels subpanels C, D, E, F but the caption text is incomplete and lists only part of the described content; the description of subpanel D is missing, and subpanel C's description is unclear ('different labels in different study states').
- [Section 3.9 and Table 8] The duration bins are inconsistent: Section 3.9 defines short-term as 0-1000 days, medium as 1001-2000 days, and long-term as 2001+ days, but Table 8 uses categories '0-1000', '1001-3000', and '3000+'. The table also reports different total counts than the rest of the text.
- [Table 7 and Section 3.1] Table 7 lists GPT-4 as 'gpt-4-turbo-2024-04-09' with training cutoff Dec 2023, while Section 3.1 groups GPT-4 with GPT-4o/GPT-4mini as trained till October 2023; this discrepancy should be resolved because it affects the leakage argument.
Circularity Check
The benchmark tests models on trials completed before their training cutoffs, so the reported 'predictions' can be in-sample recall; the central GPT-4o/HINT comparison is not out-of-sample.
-
fitted input called prediction
[Section 3.1 (Clinical Trials Data Curation and Processing), Table 7; Section 2.1, Table 1]
"HINT [9]: training checkpoint as of April 2022; we obtained a total of 2,304 records. Among all models, the HINT model achieved the highest specificity at 0.473, which contributed to its relatively strong balanced accuracy of 0.525 and an MCC of 0.050."
The HINT test set is selected by the same April 2022 date as the model checkpoint's training period, and the paper reports no train/test split. A supervised model evaluated on trials from its own training period is not making out-of-sample predictions; the reported specificity and balanced accuracy are in-sample classification scores. The 'prediction' step therefore reduces to scoring a fitted model on data of the same vintage used to fit it.
-
fitted input called prediction
[Section 3.1 (Clinical Trials Data Curation and Processing), Section 2.1 (Prediction on the Entire Clinical Trial Dataset), Table 7]
"we restricted the primary completion date of the trials according to the cut-off dates corresponding to the training periods of different models: ... GPT-4 [20] and GPT-4 Mini [21]/GPT-4o [22]: trained till October 2023; we acquired 54 and 74 records, respectively (Fig. 2A). The GPT-4o model performed the best in terms of balanced accuracy and MCC, reaching a balanced accuracy of 0.573 and an MCC of 0.212(Table 1)."
For each LLM, the test set is defined as trials whose primary completion precedes the model's training cutoff. The paper restricts to trials with available results, so the success/failure label of each test trial predates the cutoff and lies inside the pretraining window. The prompt supplies the trial title, summary, conditions, interventions, and outcome measures, so a model that memorized the trial can emit the stored label. The headline GPT-4o metrics are therefore not out-of-sample forecasts; the benchmark cannot distinguish prediction from memorized recall.
full rationale
The paper is an empirical benchmark rather than a mathematical derivation, so the circularity is in the evaluation protocol. Section 3.1 explicitly restricts test trials to those with primary completion before each model's training cutoff (Table 7). For HINT, the checkpoint is dated April 2022 and the test set is all pre-April-2022 trials with no reported held-out split, making the HINT specificity advantage an in-sample result. For the LLMs, the same restriction places the test labels inside the pretraining period, so the abstract's and Section 2.1's central claims (GPT-4o best LLM; HINT best at negative outcomes) rest on metrics that can reflect memorization rather than prediction. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation is present. The 'terminated trial' exclusion is a selection-bias concern but not an equation-level circularity and is not counted here. Because the central comparison reduces, at least in part, to in-temporal-period evaluation, the score is 6 rather than a lower non-circularity score.
Assumptions & free parameters
free parameters (2)
- HINT success threshold =
0.5
- Trial duration bins =
0-1000 days (39.8%), 1001-3000 days (50.4%), 3000+ days (9.6%)
assumptions (3)
- ad hoc to paper Trials completed before a model's training cutoff are unseen by the model.
- domain assumption Manual labels assigned by two researchers are accurate and consistent.
- domain assumption HINT can be evaluated on trials from the same period as its training data without overlap.
Cite this review
Pith. "Pith review of Can artificial intelligence predict clinical trial outcomes?." pith.science (2026). https://pith.science/paper/VG2PAFQ5
@misc{pith2026241117595,
author = {Pith},
title = {Pith review of: Can artificial intelligence predict clinical trial outcomes?},
year = {2026},
howpublished = {\url{https://pith.science/paper/VG2PAFQ5}},
note = {Machine review of arXiv:2411.17595}
}
read the original abstract
This study evaluates the performance of large language models (LLMs) and the HINT model in predicting clinical trial outcomes, focusing on metrics including Balanced Accuracy, Matthews Correlation Coefficient (MCC), Recall, and Specificity. Results show that GPT-4o achieves superior overall performance among LLMs but, like its counterparts (GPT-3.5, GPT-4mini, Llama3), struggles with identifying negative outcomes. In contrast, HINT excels in negative sample recognition and demonstrates resilience to external factors (e.g., recruitment challenges) but underperforms in oncology trials, a major component of the dataset. LLMs exhibit strengths in early-phase trials and simpler endpoints like Overall Survival (OS), while HINT shows consistency across trial phases and excels in complex endpoints (e.g., Objective Response Rate). Trial duration analysis reveals improved model performance for medium- to long-term trials, with GPT-4o and HINT displaying stability and enhanced specificity, respectively. We underscore the complementary potential of LLMs (e.g., GPT-4o, Llama3) and HINT, advocating for hybrid approaches to leverage GPT-4o's predictive power and HINT's specificity in clinical trial outcome forecasting.
Reference graph
Works this paper leans on
-
[1]
Growth, Tren ds, and Forecast 2015 (2021)
Market, N.: Global industry analysis, size, share. Growth, Tren ds, and Forecast 2015 (2021)
work page 2021
-
[2]
A nnual review of genomics and human genetics 2(1), 177–211 (2001)
Pfeifer, A., Verma, I.M.: Gene therapy: promises and problems. A nnual review of genomics and human genetics 2(1), 177–211 (2001)
work page 2001
-
[3]
Carvalho, M., Sepodes, B., Martins, A.P.: Patient access to gene t herapy medicinal products: a comprehensive review. BMJ Innovations 7(1) (2021)
work page 2021
-
[4]
Value in h ealth 18(2), 161–172 (2015)
Ramsey, S.D., Willke, R.J., Glick, H., Reed, S.D., Augustovski, F., Jonss on, B., Briggs, A., Sullivan, S.D.: Cost-effectiveness analysis alongside clinic al tri- als ii—an ispor good research practices task force report. Value in h ealth 18(2), 161–172 (2015)
work page 2015
-
[5]
Nat Rev Drug Discov 16(6), 381–382 (2017)
Martin, L., Hutchens, M., Hawkins, C., Radnov, A.: How much do clinic al trials cost. Nat Rev Drug Discov 16(6), 381–382 (2017)
work page 2017
-
[6]
Biostatistics 20(2), 273–286 (2019) 19
Wong, C.H., Siah, K.W., Lo, A.W.: Estimation of clinical trial success ra tes and related parameters. Biostatistics 20(2), 273–286 (2019) 19
work page 2019
-
[7]
International journal of cancer 147(9), 2537–2549 (2020)
Schperberg, A.V., Boichard, A., Tsigelny, I.F., Richard, S.B., Kurzr ock, R.: Machine learning model to predict oncologic outcomes for drugs in ra ndomized clinical trials. International journal of cancer 147(9), 2537–2549 (2020)
work page 2020
-
[8]
NPJ precision onco logy 6(1), 37 (2022)
Qaiser, T., Lee, C.-Y., Vandenberghe, M., Yeh, J., Gavrielides, M.A., Hipp, J., Scott, M., Reischl, J.: Usability of deep learning and h&e images predict disease outcome-emerging tool to optimize clinical trials. NPJ precision onco logy 6(1), 37 (2022)
work page 2022
Show all 27 references
-
[9]
Patterns 3(4) (2022)
Fu, T., Huang, K., Xiao, C., Glass, L.M., Sun, J.: Hint: Hierarchical int eraction network for clinical-trial-outcome predictions. Patterns 3(4) (2022)
2022
-
[10]
In: Artificial Intelligence and Data Science for Healthca re: Bridging Data-Centric AI and People-Centric Healthcare (2018)
Gao, C., Fu, T., Sun, J.: Lint: Llm interaction network for clinical t rial outcome prediction. In: Artificial Intelligence and Data Science for Healthca re: Bridging Data-Centric AI and People-Centric Healthcare (2018)
2018
-
[11]
arXiv p reprint arXiv:2408.10995 (2024)
Reinisch, M., He, J., Liao, C., Siddiqui, S.A., Xiao, B.: Ctp-llm: Clinical trial phase transition prediction using large language models. arXiv p reprint arXiv:2408.10995 (2024)
2024 arXiv
-
[12]
: Assessing the risk of bias in randomized clinical trials with large language models
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., et al. : Assessing the risk of bias in randomized clinical trials with large language models. JAMA Network Open 7(5), 2412687–2412687 (2024)
2024
-
[13]
ArXiv (2023)
Jin, Q., Wang, Z., Floudas, C.S., Chen, F., Gong, C., Bracken-Clark e, D., Xue, E., Yang, Y., Sun, J., Lu, Z.: Matching patients to clinical trials with large la nguage models. ArXiv (2023)
2023
-
[14]
arXiv preprint arXiv:2402.16406 (2024)
Markey, N., El-Mansouri, I., Rensonnet, G., Langen, C., Meier, C .: From rags to riches: Using large language models to write documents for clinical tr ials. arXiv preprint arXiv:2402.16406 (2024)
2024 arXiv
-
[15]
Human genetics 138(2), 109–124 (2019)
Xu, J., Yang, P., Xue, S., Sharma, B., Sanchez-Martin, M., Wang, F., Beaty, K.A., Dehan, E., Parikh, B.: Translating cancer genomics into precision med icine with artificial intelligence: applications, challenges and future perspect ives. Human genetics 138(2), 109–124 (2019)
2019
-
[16]
https://clinicaltrials.gov/
Health, N.I.: ClinicalTrials.gov. https://clinicaltrials.gov/. Accessed: 2024-09-22 (n.d.)
2024
-
[17]
Therapeut ic Innovation & Regulatory Science 57(1), 49–56 (2023)
Getz, K., Smith, Z., Kravet, M.: Protocol design and performan ce benchmarks by phase and by oncology and rare disease subgroups. Therapeut ic Innovation & Regulatory Science 57(1), 49–56 (2023)
2023
-
[18]
https://chatgpt.com/g/g-F00faAwkE-open-a-i-gpt-3-5
OpenAI: GPT-3.5. https://chatgpt.com/g/g-F00faAwkE-open-a-i-gpt-3-5 . Accessed: 2024-10-26 20
2024
-
[19]
arX iv preprint arXiv:2407.21783 (2024)
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman , A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arX iv preprint arXiv:2407.21783 (2024)
2024 arXiv
-
[20]
arXiv preprint arXiv:2303.08774 (2023)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F .L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical r eport. arXiv preprint arXiv:2303.08774 (2023)
2023 arXiv
-
[21]
https://openai
OpenAI: GPT-4o Mini: Advancing Cost-Efficient Intelligence. https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ . Accessed: 2024- 10-26
2024
-
[22]
https://openai.com/index/hello-gpt-4o/
OpenAI: Hello GPT-4. https://openai.com/index/hello-gpt-4o/ . Accessed: 2024- 10-26
2024
-
[23]
ESMO Real World Data and Digital Oncology 4, 100042 (2024) https://doi.org/10.1016/ j.esmorw.2024.100042
Fabre, B.L., Magalhaes Filho, M.A.F., Aguiar, P.N., da Costa, F.M., Gut ierres, B., William, W.N., Del Giglio, A.: Evaluating gpt-4 as an academic support to ol for clinicians: a comparative analysis of case records from the litera ture. ESMO Real World Data and Digital Oncolo...
2024
-
[24]
Weininger, D.: Smiles, a chemical language and information system . 1. introduc- tion to methodology and encoding rules. Journal of chemical inform ation and computer sciences 28(1), 31–36 (1988)
1988
-
[25]
https://wordcount.com/keyword-extractor
Wordcount: Keyword Extractor. https://wordcount.com/keyword-extractor. Accessed: 2024-10-15 (n.d.)
2024
-
[26]
Chik, R.W., Sentosa, I., Shahruddin, S.: Strategic management c omparison anal- ysis of webometric top universities in world, asia and malaysia on the we b visibility
-
[27]
https:// www.cdc.gov/nchs/icd/icd-10-cm/index.html
Disease Control, C., Prevention: ICD-10-CM Official Coding Guide lines. https:// www.cdc.gov/nchs/icd/icd-10-cm/index.html. Accessed: 2024-09-22 (n.d.) 21
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.