Pith. sign in

REVIEW 5 major objections 5 minor 27 references

Can artificial intelligence predict clinical trial outcomes?

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read This paper claims that GPT-4o is the best large language model tested for predicting clinical trial outcomes, yet every LLM tested overlooks most failed trials, while the HINT model uniquely catches negative outcomes.

desk verdict The central comparison is invalidated by a reversed temporal split that lets LLMs memorize outcomes; the abstract and main table also disagree on the headline numbers. read the letter →

arxiv 2411.17595 v2 pith:VG2PAFQ5 submitted 2024-11-26 cs.LG stat.AP

classification cs.LGstat.AP
keywords clinicaltrialoutcomepredictionlargelanguagemodelsGPT-4oHINTmodelnegativesamplerecognitionbalancedaccuracyMatthewscorrelationcoefficienttermination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clinical trials fail for many reasons, and the cost of a failed late-stage trial runs into tens of millions of dollars. The paper asks whether large language models, which read trial protocols like a human would, can predict which trials will succeed, and how they compare with HINT, a purpose-built deep learning model for the same task. The central claim is that GPT-4o is the strongest LLM on balanced accuracy, but that all LLMs are heavily biased toward predicting success and almost never flag failures; HINT is the only model with substantial specificity, so it is the only one that can identify negative outcomes. The authors take these to be complementary profiles and conclude that hybrid LLM-plus-HINT systems would combine GPT-4o's predictive power with HINT's ability to spot failing trials. A reader should care because the paper is an explicit head-to-head comparison of the current LLM generation against a dedicated trial-outcome model, on the same public registry data.

What carries the argument

The machinery is two families of predictors. The LLM machinery is a prompt: each model receives the trial title, summary, conditions, interventions, primary outcome measures, and trial type, and must answer 'success' or 'failure' in one word. The HINT machinery is a hierarchical interaction network, a dynamic-attention graph neural network that embeds drug SMILES, disease ICD-10 codes, and eligibility criteria and outputs a success probability, with 0.5 as the threshold. The evaluation machinery is the metric set of balanced accuracy, Matthews correlation coefficient, recall, and specificity, chosen because the dataset is imbalanced at roughly 79 percent successful trials. A final piece of machinery is cutoff-based data selection: each model is tested only on trials whose primary completion date precedes that model's training cutoff, which the authors assume prevents outcome memorisation.

What would settle it

Re-run the same evaluations on trials whose outcomes were not public before each model's training cutoff, for example trials completed after the cutoff, and ask whether GPT-4o's balanced-accuracy lead over HINT persists. A cheap consistency check is to rerun the exact 74-trial GPT-4o evaluation: the abstract reports a balanced accuracy of 0.573 and an MCC of 0.212, while Table 1 reports 0.532 and 0.098, so only one set of numbers can reproduce.

Watch

Extended reading notes

Core claim

The study's central discovery is that no single model predicts clinical trial outcomes well, but the models fail in opposite directions. GPT-4o, the best LLM, is reported to reach a balanced accuracy of 0.573 and an MCC of 0.212 in the abstract, because it classifies most trials as successful and catches few negative outcomes. HINT, by contrast, has the highest specificity (0.473) and a balanced accuracy of 0.525, making it the only model that can reliably recognise failed trials. The paper further finds that LLMs are strongest on Phase I trials and Overall Survival endpoints, while HINT is more stable across phases and handles complex endpoints such as Objective Response Rate, but underperforms on oncology trials, which make up 30 to 37 percent of the data. The authors conclude that the two families are complementary and advocate hybrid systems rather than either model alone.

Load-bearing premise

The load-bearing premise is that limiting each model's test set to trials whose primary completion date falls before that model's training cutoff prevents data leakage; in reality, outcomes for those trials were public before the cutoff, so the models may have memorised them rather than predicted them.

Editorial extensions

If this is right

  • If the complementarity claim holds, a combined system that uses GPT-4o for overall scoring and HINT's negative prediction as a filter would flag failed trials that pure-LLM pipelines miss.
  • If the LLMs' near-zero specificity is a general property, any LLM-based trial screening tool needs an explicit negative-class mechanism or calibration before it can support go/no-go decisions.
  • If HINT's oncology shortfall is intrinsic, hybrid designs for cancer trials should weight LLM predictions more heavily for neoplasms, the opposite of the overall pattern.
  • If the phase and duration trends are real, deployment should be stratified: use LLMs for early-phase and simple-endpoint decisions and HINT for later-phase and complex-endpoint decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors stop at advocating hybrids; I would test a concrete two-stage design in which GPT-4o screens every trial and every trial it labels 'success' is re-checked by HINT, using HINT's specificity as a go/no-go gate for the most expensive decisions.
  • A prospective test that the paper does not run would collect trials whose results are posted after the paper's data-collection date; if GPT-4o's edge shrinks, the leaderboard here would reflect memorisation more than prediction.
  • The abstract-to-Table 1 discrepancy in GPT-4o's numbers (0.573 versus 0.532 balanced accuracy, 0.212 versus 0.098 MCC) is my main reason to treat the exact ranking as provisional until one set of numbers is rerun and confirmed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript compares several large language models (GPT-3.5, GPT-4, GPT-4o, GPT-4mini, Llama3) with the HINT deep learning model on the task of predicting clinical trial success versus failure, using trials drawn from ClinicalTrials.gov. The central claims are that GPT-4o achieves the best overall performance among LLMs, that all LLMs struggle to identify negative outcomes (low specificity), and that HINT has high specificity and therefore complements LLMs, motivating hybrid approaches. Performance is reported through balanced accuracy, MCC, recall, and specificity, with subgroup analyses by trial phase, disease category, endpoint, and trial duration.

Significance. If the evaluation were valid, the paper would provide a practically useful benchmark and an evidence-based argument for hybrid LLM-plus-HINT systems in clinical trial outcome forecasting. The question is important and the paper includes multiple metrics and several clinically relevant stratifications. However, the central comparison rests on a temporal split that is reversed: test trials are selected with primary completion dates before each model's training cutoff, so LLM predictions may reflect memorization of pretraining data rather than out-of-sample forecasting. The paper also does not release code or a complete dataset, and one headline metric is internally inconsistent. The claims are therefore not currently supported.

major comments (5)
  1. [Section 3.1, Fig. 2A] The evaluation design introduces a reversed temporal split. Section 3.1 states that 'we restricted the primary completion date of the trials according to the cut-off dates corresponding to the training periods of different models,' so the test set consists of trials completed before each model's training cutoff. For such trials, the outcome was publicly known before the cutoff, and the prompt provides the trial title and summary, making it possible for the LLM to retrieve the outcome from pretraining data. This directly threatens the headline comparison: the GPT-4o balanced accuracy of 0.573 and MCC of 0.212 in the Abstract and Section 2.1, the 'LLMs struggle with negative outcomes' pattern, and the HINT specificity advantage could all be artifacts of memorized labels rather than predictive skill. A proper evaluation would use trials whose outcomes were not available at the model's cutoff, e.g., primary completion or results disclosure after the cutoff.
  2. [Section 2.6 and Section 3.3] Terminated trials are excluded after observing poor LLM performance on them, which is outcome-based sample selection. Section 2.6 reports that LLMs achieved accuracy around 0.1 or lower on terminated trials and then says, 'Consequently, we excluded “terminated” trials from further analyses to avoid biasing the experimental results.' Section 3.3 similarly states, 'In subsequent analyses, we excluded data with a “terminated” status because we found that the LLMs lacked the ability to accurately recognize trial termination.' This post hoc removal of a difficult and clinically important subset inflates the reported metrics and changes the evaluation population. If termination prediction is a goal, these trials should be included; if not, the exclusion criteria should be specified before the analysis, not in response to the results.
  3. [Abstract, Section 2.1, Table 1] The headline results for GPT-4o are internally inconsistent. The Abstract states GPT-4o achieves a balanced accuracy of 0.573 and MCC of 0.212, and Section 2.1 repeats these values with recall 0.931 and specificity 0.214. However, Table 1 reports GPT-4o balanced accuracy 0.532, MCC 0.098, recall 0.913, and specificity 0.150. These discrepancies mean the reader cannot determine which numbers support the central claim, and the abstract's numbers do not match the paper's own results table.
  4. [Section 3.1, Table 1] The models are evaluated on non-identical test sets of very different sizes, which confounds the cross-model comparison. Section 3.1 reports 3,811 trials for GPT-3.5, 2,304 for HINT, 725 for Llama3, and only 54 and 74 for GPT-4 and GPT-4o/GPT-4mini, respectively, using different cutoff dates. Consequently, the disease distribution, outcome prevalence, and trial difficulty differ across model columns, as visible in Fig. 1A and Table 3. The claim that 'GPT-4o achieves superior overall performance among LLMs' is not supportable when each model is scored on a different test population.
  5. [Section 2.3, Table 3] Several subgroup claims are based on extremely small sample sizes. For GPT-4o, disease categories have between 2 and 18 trials, and endpoint categories between 5 and 28 trials, yet the text makes specific claims such as GPT-4o's specificity of 1.000 in respiratory diseases (3 trials) and 0.500 in digestive diseases (3 trials). These estimates have very large uncertainty, and the statement that GPT-4o 'displayed strong recognition of negative samples' in these categories is not statistically grounded. The paper should at least report confidence intervals or avoid strong claims for cells with fewer than, say, 20 trials.
minor comments (5)
  1. [Section 3.1] The paragraph ends with an incomplete sentence: 'For most of our analyses, we retained clinical trial data with the status' is followed by no completion; the intended status filter should be stated.
  2. [Title page and Abstract] There are typographical errors: 'Singpaore' on the title page and 'We underscores' in the Abstract should be 'We underscore'.
  3. [Fig. 2 caption] The Figure 2 caption labels subpanels C, D, E, F but the caption text is incomplete and lists only part of the described content; the description of subpanel D is missing, and subpanel C's description is unclear ('different labels in different study states').
  4. [Section 3.9 and Table 8] The duration bins are inconsistent: Section 3.9 defines short-term as 0-1000 days, medium as 1001-2000 days, and long-term as 2001+ days, but Table 8 uses categories '0-1000', '1001-3000', and '3000+'. The table also reports different total counts than the rest of the text.
  5. [Table 7 and Section 3.1] Table 7 lists GPT-4 as 'gpt-4-turbo-2024-04-09' with training cutoff Dec 2023, while Section 3.1 groups GPT-4 with GPT-4o/GPT-4mini as trained till October 2023; this discrepancy should be resolved because it affects the leakage argument.

Circularity Check

2 steps flagged · score 6.0 of 10

The benchmark tests models on trials completed before their training cutoffs, so the reported 'predictions' can be in-sample recall; the central GPT-4o/HINT comparison is not out-of-sample.

  1. fitted input called prediction [Section 3.1 (Clinical Trials Data Curation and Processing), Table 7; Section 2.1, Table 1]
    "HINT [9]: training checkpoint as of April 2022; we obtained a total of 2,304 records. Among all models, the HINT model achieved the highest specificity at 0.473, which contributed to its relatively strong balanced accuracy of 0.525 and an MCC of 0.050."

    The HINT test set is selected by the same April 2022 date as the model checkpoint's training period, and the paper reports no train/test split. A supervised model evaluated on trials from its own training period is not making out-of-sample predictions; the reported specificity and balanced accuracy are in-sample classification scores. The 'prediction' step therefore reduces to scoring a fitted model on data of the same vintage used to fit it.

  2. fitted input called prediction [Section 3.1 (Clinical Trials Data Curation and Processing), Section 2.1 (Prediction on the Entire Clinical Trial Dataset), Table 7]
    "we restricted the primary completion date of the trials according to the cut-off dates corresponding to the training periods of different models: ... GPT-4 [20] and GPT-4 Mini [21]/GPT-4o [22]: trained till October 2023; we acquired 54 and 74 records, respectively (Fig. 2A). The GPT-4o model performed the best in terms of balanced accuracy and MCC, reaching a balanced accuracy of 0.573 and an MCC of 0.212(Table 1)."

    For each LLM, the test set is defined as trials whose primary completion precedes the model's training cutoff. The paper restricts to trials with available results, so the success/failure label of each test trial predates the cutoff and lies inside the pretraining window. The prompt supplies the trial title, summary, conditions, interventions, and outcome measures, so a model that memorized the trial can emit the stored label. The headline GPT-4o metrics are therefore not out-of-sample forecasts; the benchmark cannot distinguish prediction from memorized recall.

full rationale

The paper is an empirical benchmark rather than a mathematical derivation, so the circularity is in the evaluation protocol. Section 3.1 explicitly restricts test trials to those with primary completion before each model's training cutoff (Table 7). For HINT, the checkpoint is dated April 2022 and the test set is all pre-April-2022 trials with no reported held-out split, making the HINT specificity advantage an in-sample result. For the LLMs, the same restriction places the test labels inside the pretraining period, so the abstract's and Section 2.1's central claims (GPT-4o best LLM; HINT best at negative outcomes) rest on metrics that can reflect memorization rather than prediction. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation is present. The 'terminated trial' exclusion is a selection-bias concern but not an equation-level circularity and is not counted here. Because the central comparison reduces, at least in part, to in-temporal-period evaluation, the score is 6 rather than a lower non-circularity score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the temporal split assumption, the label accuracy assumption, and the HINT applicability assumption. The HINT threshold and duration bins are hand-chosen values that shape reported metrics. No new entities are proposed.

free parameters (2)
  • HINT success threshold = 0.5
    Predictions above 0.5 are labeled success; this threshold materially shapes HINT's specificity and recall tradeoff and is not varied or justified.
  • Trial duration bins = 0-1000 days (39.8%), 1001-3000 days (50.4%), 3000+ days (9.6%)
    Arbitrary bin edges drive the duration analysis in Section 3.9; no sensitivity check is reported.
assumptions (3)
  • ad hoc to paper Trials completed before a model's training cutoff are unseen by the model.
    Section 3.1 selects trials with primary completion dates up to each model's cutoff, but this makes outcomes available in training data, so the assumption is false and load-bearing.
  • domain assumption Manual labels assigned by two researchers are accurate and consistent.
    Section 3.3 describes double-blind cross-validation but reports no inter-annotator agreement or validation against an objective standard.
  • domain assumption HINT can be evaluated on trials from the same period as its training data without overlap.
    No train/test split is described; HINT's checkpoint (Apr 2022) may have seen the same-period trials used for evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can artificial intelligence predict clinical trial outcomes?." pith.science (2026). https://pith.science/paper/VG2PAFQ5

@misc{pith2026241117595,
  author       = {Pith},
  title        = {Pith review of: Can artificial intelligence predict clinical trial outcomes?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VG2PAFQ5}},
  note         = {Machine review of arXiv:2411.17595}
}
read the original abstract

This study evaluates the performance of large language models (LLMs) and the HINT model in predicting clinical trial outcomes, focusing on metrics including Balanced Accuracy, Matthews Correlation Coefficient (MCC), Recall, and Specificity. Results show that GPT-4o achieves superior overall performance among LLMs but, like its counterparts (GPT-3.5, GPT-4mini, Llama3), struggles with identifying negative outcomes. In contrast, HINT excels in negative sample recognition and demonstrates resilience to external factors (e.g., recruitment challenges) but underperforms in oncology trials, a major component of the dataset. LLMs exhibit strengths in early-phase trials and simpler endpoints like Overall Survival (OS), while HINT shows consistency across trial phases and excels in complex endpoints (e.g., Objective Response Rate). Trial duration analysis reveals improved model performance for medium- to long-term trials, with GPT-4o and HINT displaying stability and enhanced specificity, respectively. We underscore the complementary potential of LLMs (e.g., GPT-4o, Llama3) and HINT, advocating for hybrid approaches to leverage GPT-4o's predictive power and HINT's specificity in clinical trial outcome forecasting.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages

  1. [1]

    Growth, Tren ds, and Forecast 2015 (2021)

    Market, N.: Global industry analysis, size, share. Growth, Tren ds, and Forecast 2015 (2021)

  2. [2]

    A nnual review of genomics and human genetics 2(1), 177–211 (2001)

    Pfeifer, A., Verma, I.M.: Gene therapy: promises and problems. A nnual review of genomics and human genetics 2(1), 177–211 (2001)

  3. [3]

    BMJ Innovations 7(1) (2021)

    Carvalho, M., Sepodes, B., Martins, A.P.: Patient access to gene t herapy medicinal products: a comprehensive review. BMJ Innovations 7(1) (2021)

  4. [4]

    Value in h ealth 18(2), 161–172 (2015)

    Ramsey, S.D., Willke, R.J., Glick, H., Reed, S.D., Augustovski, F., Jonss on, B., Briggs, A., Sullivan, S.D.: Cost-effectiveness analysis alongside clinic al tri- als ii—an ispor good research practices task force report. Value in h ealth 18(2), 161–172 (2015)

  5. [5]

    Nat Rev Drug Discov 16(6), 381–382 (2017)

    Martin, L., Hutchens, M., Hawkins, C., Radnov, A.: How much do clinic al trials cost. Nat Rev Drug Discov 16(6), 381–382 (2017)

  6. [6]

    Biostatistics 20(2), 273–286 (2019) 19

    Wong, C.H., Siah, K.W., Lo, A.W.: Estimation of clinical trial success ra tes and related parameters. Biostatistics 20(2), 273–286 (2019) 19

  7. [7]

    International journal of cancer 147(9), 2537–2549 (2020)

    Schperberg, A.V., Boichard, A., Tsigelny, I.F., Richard, S.B., Kurzr ock, R.: Machine learning model to predict oncologic outcomes for drugs in ra ndomized clinical trials. International journal of cancer 147(9), 2537–2549 (2020)

  8. [8]

    NPJ precision onco logy 6(1), 37 (2022)

    Qaiser, T., Lee, C.-Y., Vandenberghe, M., Yeh, J., Gavrielides, M.A., Hipp, J., Scott, M., Reischl, J.: Usability of deep learning and h&e images predict disease outcome-emerging tool to optimize clinical trials. NPJ precision onco logy 6(1), 37 (2022)

Show all 27 references
  1. [9]

    Patterns 3(4) (2022)

    Fu, T., Huang, K., Xiao, C., Glass, L.M., Sun, J.: Hint: Hierarchical int eraction network for clinical-trial-outcome predictions. Patterns 3(4) (2022)

  2. [10]

    In: Artificial Intelligence and Data Science for Healthca re: Bridging Data-Centric AI and People-Centric Healthcare (2018)

    Gao, C., Fu, T., Sun, J.: Lint: Llm interaction network for clinical t rial outcome prediction. In: Artificial Intelligence and Data Science for Healthca re: Bridging Data-Centric AI and People-Centric Healthcare (2018)

  3. [11]

    arXiv p reprint arXiv:2408.10995 (2024)

    Reinisch, M., He, J., Liao, C., Siddiqui, S.A., Xiao, B.: Ctp-llm: Clinical trial phase transition prediction using large language models. arXiv p reprint arXiv:2408.10995 (2024)

  4. [12]

    : Assessing the risk of bias in randomized clinical trials with large language models

    Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., et al. : Assessing the risk of bias in randomized clinical trials with large language models. JAMA Network Open 7(5), 2412687–2412687 (2024)

  5. [13]

    ArXiv (2023)

    Jin, Q., Wang, Z., Floudas, C.S., Chen, F., Gong, C., Bracken-Clark e, D., Xue, E., Yang, Y., Sun, J., Lu, Z.: Matching patients to clinical trials with large la nguage models. ArXiv (2023)

  6. [14]

    arXiv preprint arXiv:2402.16406 (2024)

    Markey, N., El-Mansouri, I., Rensonnet, G., Langen, C., Meier, C .: From rags to riches: Using large language models to write documents for clinical tr ials. arXiv preprint arXiv:2402.16406 (2024)

  7. [15]

    Human genetics 138(2), 109–124 (2019)

    Xu, J., Yang, P., Xue, S., Sharma, B., Sanchez-Martin, M., Wang, F., Beaty, K.A., Dehan, E., Parikh, B.: Translating cancer genomics into precision med icine with artificial intelligence: applications, challenges and future perspect ives. Human genetics 138(2), 109–124 (2019)

  8. [16]

    https://clinicaltrials.gov/

    Health, N.I.: ClinicalTrials.gov. https://clinicaltrials.gov/. Accessed: 2024-09-22 (n.d.)

  9. [17]

    Therapeut ic Innovation & Regulatory Science 57(1), 49–56 (2023)

    Getz, K., Smith, Z., Kravet, M.: Protocol design and performan ce benchmarks by phase and by oncology and rare disease subgroups. Therapeut ic Innovation & Regulatory Science 57(1), 49–56 (2023)

  10. [18]

    https://chatgpt.com/g/g-F00faAwkE-open-a-i-gpt-3-5

    OpenAI: GPT-3.5. https://chatgpt.com/g/g-F00faAwkE-open-a-i-gpt-3-5 . Accessed: 2024-10-26 20

  11. [19]

    arX iv preprint arXiv:2407.21783 (2024)

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman , A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arX iv preprint arXiv:2407.21783 (2024)

  12. [20]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F .L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical r eport. arXiv preprint arXiv:2303.08774 (2023)

  13. [21]

    https://openai

    OpenAI: GPT-4o Mini: Advancing Cost-Efficient Intelligence. https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ . Accessed: 2024- 10-26

  14. [22]

    https://openai.com/index/hello-gpt-4o/

    OpenAI: Hello GPT-4. https://openai.com/index/hello-gpt-4o/ . Accessed: 2024- 10-26

  15. [23]

    ESMO Real World Data and Digital Oncology 4, 100042 (2024) https://doi.org/10.1016/ j.esmorw.2024.100042

    Fabre, B.L., Magalhaes Filho, M.A.F., Aguiar, P.N., da Costa, F.M., Gut ierres, B., William, W.N., Del Giglio, A.: Evaluating gpt-4 as an academic support to ol for clinicians: a comparative analysis of case records from the litera ture. ESMO Real World Data and Digital Oncolo...

  16. [24]

    Weininger, D.: Smiles, a chemical language and information system . 1. introduc- tion to methodology and encoding rules. Journal of chemical inform ation and computer sciences 28(1), 31–36 (1988)

  17. [25]

    https://wordcount.com/keyword-extractor

    Wordcount: Keyword Extractor. https://wordcount.com/keyword-extractor. Accessed: 2024-10-15 (n.d.)

  18. [26]

    Chik, R.W., Sentosa, I., Shahruddin, S.: Strategic management c omparison anal- ysis of webometric top universities in world, asia and malaysia on the we b visibility

  19. [27]

    https:// www.cdc.gov/nchs/icd/icd-10-cm/index.html

    Disease Control, C., Prevention: ICD-10-CM Official Coding Guide lines. https:// www.cdc.gov/nchs/icd/icd-10-cm/index.html. Accessed: 2024-09-22 (n.d.) 21

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.