{"id":"48331b19-de4b-48b3-86f4-d7de66c1be42","arxiv_id":"2411.17595","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A comparison of LLMs and HINT for clinical trial outcome prediction that is undermined by data leakage and post-hoc exclusions.","lead":"Researchers compared five large language models and an older AI model called HINT on the task of predicting clinical trial success. The study's design is flawed, so its claims about which model is best are not supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation uses a reversed temporal split: trials completed before each model's cutoff are included, so LLM 'predictions' can be memorized outcomes from pretraining; the central GPT-4o/HINT comparison is untested.","rationale":"The single most load-bearing assumption is that a cutoff-before-completion split prevents LLMs from knowing trial outcomes. It does the opposite. A trial that completed before the cutoff will almost always have its result in the pretraining corpus (ClinicalTrials.gov results pages, press releases, FDA documents), so the model can answer from memory. This is not an objection to the field's consensus; it is an internal validity flaw. Even if the Table 1/Abstract discrepancy in GPT-4o's numbers were corrected, the evaluation would still not support the conclusion. I therefore agree with the reader's weakest_assumption. The proposed holdout test would settle the issue directly: if GPT-4o performs at chance on post-cutoff trials, the central claim collapses; if it remains above chance, the leakage concern is mitigated and the authors could then address the remaining inconsistencies. No ad hominem: the flaw is in the study design, not the authors' intent. The paper has no machine-checked proofs or public artifacts that would independently support the numbers, and the 'data available upon request' statement is not a substitute.","tokens_in":14224,"tokens_out":12282,"duration_ms":111665,"concrete_test":"Assemble a holdout set of trials from ClinicalTrials.gov with primary completion date after 2023-10-01 and results posted after 2023-10-01, so GPT-4o's stated knowledge cutoff cannot have seen the outcome. Run GPT-4o with the exact prompt from Section 3.4 (title, summary, conditions, interventions, primary outcome measures, trial type) and compute balanced accuracy and MCC. If these fall to chance (≈0.5/0) while the original set gave 0.573/0.212, the reported superiority is explained by memorization. As a complement, run HINT on trials completed after 2022-04-01; if its specificity advantage also disappears, the same conclusion applies to the HINT comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the reported accuracies measure out-of-sample prediction. Section 3.1 instead selects trials whose primary completion date is before each model's training cutoff (e.g., GPT-4o: completion before Oct 2023; HINT: checkpoint Apr 2022; Fig. 2A). For such trials the outcome was publicly available before the cutoff, so the model can have memorized the exact trial—title, summary, interventions—and its success/failure label from pretraining data. The prompt feeds the model the title and summary, making retrieval trivial. This reversed temporal split therefore threatens every headline metric: the abstract's and Section 2.1's GPT-4o balanced accuracy 0.573/MCC 0.212, the claimed HINT specificity advantage, and the 'LLMs struggle with negative outcomes' pattern could all be artifacts of training-data recall rather than predictive skill. Because a different cutoff-based subset is used for each model (3,811 vs 2,304 vs 725 vs 54–74 trials), the relative rankings are also confounded by non-identical test distributions. The leakage concern is sufficient by itself to invalidate the central comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares several large language models (GPT-3.5, GPT-4, GPT-4o, GPT-4mini, Llama3) with the HINT deep learning model on the task of predicting clinical trial success versus failure, using trials drawn from ClinicalTrials.gov. The central claims are that GPT-4o achieves the best overall performance among LLMs, that all LLMs struggle to identify negative outcomes (low specificity), and that HINT has high specificity and therefore complements LLMs, motivating hybrid approaches. Performance is reported through balanced accuracy, MCC, recall, and specificity, with subgroup analyses by trial phase, disease category, endpoint, and trial duration.","tokens_in":14456,"tokens_out":3482,"duration_ms":30169,"significance":"If the evaluation were valid, the paper would provide a practically useful benchmark and an evidence-based argument for hybrid LLM-plus-HINT systems in clinical trial outcome forecasting. The question is important and the paper includes multiple metrics and several clinically relevant stratifications. However, the central comparison rests on a temporal split that is reversed: test trials are selected with primary completion dates before each model's training cutoff, so LLM predictions may reflect memorization of pretraining data rather than out-of-sample forecasting. The paper also does not release code or a complete dataset, and one headline metric is internally inconsistent. The claims are therefore not currently supported.","major_comments":[{"comment":"The evaluation design introduces a reversed temporal split. Section 3.1 states that 'we restricted the primary completion date of the trials according to the cut-off dates corresponding to the training periods of different models,' so the test set consists of trials completed before each model's training cutoff. For such trials, the outcome was publicly known before the cutoff, and the prompt provides the trial title and summary, making it possible for the LLM to retrieve the outcome from pretraining data. This directly threatens the headline comparison: the GPT-4o balanced accuracy of 0.573 and MCC of 0.212 in the Abstract and Section 2.1, the 'LLMs struggle with negative outcomes' pattern, and the HINT specificity advantage could all be artifacts of memorized labels rather than predictive skill. A proper evaluation would use trials whose outcomes were not available at the model's cutoff, e.g., primary completion or results disclosure after the cutoff.","section":"Section 3.1, Fig. 2A"},{"comment":"Terminated trials are excluded after observing poor LLM performance on them, which is outcome-based sample selection. Section 2.6 reports that LLMs achieved accuracy around 0.1 or lower on terminated trials and then says, 'Consequently, we excluded “terminated” trials from further analyses to avoid biasing the experimental results.' Section 3.3 similarly states, 'In subsequent analyses, we excluded data with a “terminated” status because we found that the LLMs lacked the ability to accurately recognize trial termination.' This post hoc removal of a difficult and clinically important subset inflates the reported metrics and changes the evaluation population. If termination prediction is a goal, these trials should be included; if not, the exclusion criteria should be specified before the analysis, not in response to the results.","section":"Section 2.6 and Section 3.3"},{"comment":"The headline results for GPT-4o are internally inconsistent. The Abstract states GPT-4o achieves a balanced accuracy of 0.573 and MCC of 0.212, and Section 2.1 repeats these values with recall 0.931 and specificity 0.214. However, Table 1 reports GPT-4o balanced accuracy 0.532, MCC 0.098, recall 0.913, and specificity 0.150. These discrepancies mean the reader cannot determine which numbers support the central claim, and the abstract's numbers do not match the paper's own results table.","section":"Abstract, Section 2.1, Table 1"},{"comment":"The models are evaluated on non-identical test sets of very different sizes, which confounds the cross-model comparison. Section 3.1 reports 3,811 trials for GPT-3.5, 2,304 for HINT, 725 for Llama3, and only 54 and 74 for GPT-4 and GPT-4o/GPT-4mini, respectively, using different cutoff dates. Consequently, the disease distribution, outcome prevalence, and trial difficulty differ across model columns, as visible in Fig. 1A and Table 3. The claim that 'GPT-4o achieves superior overall performance among LLMs' is not supportable when each model is scored on a different test population.","section":"Section 3.1, Table 1"},{"comment":"Several subgroup claims are based on extremely small sample sizes. For GPT-4o, disease categories have between 2 and 18 trials, and endpoint categories between 5 and 28 trials, yet the text makes specific claims such as GPT-4o's specificity of 1.000 in respiratory diseases (3 trials) and 0.500 in digestive diseases (3 trials). These estimates have very large uncertainty, and the statement that GPT-4o 'displayed strong recognition of negative samples' in these categories is not statistically grounded. The paper should at least report confidence intervals or avoid strong claims for cells with fewer than, say, 20 trials.","section":"Section 2.3, Table 3"}],"minor_comments":[{"comment":"The paragraph ends with an incomplete sentence: 'For most of our analyses, we retained clinical trial data with the status' is followed by no completion; the intended status filter should be stated.","section":"Section 3.1"},{"comment":"There are typographical errors: 'Singpaore' on the title page and 'We underscores' in the Abstract should be 'We underscore'.","section":"Title page and Abstract"},{"comment":"The Figure 2 caption labels subpanels C, D, E, F but the caption text is incomplete and lists only part of the described content; the description of subpanel D is missing, and subpanel C's description is unclear ('different labels in different study states').","section":"Fig. 2 caption"},{"comment":"The duration bins are inconsistent: Section 3.9 defines short-term as 0-1000 days, medium as 1001-2000 days, and long-term as 2001+ days, but Table 8 uses categories '0-1000', '1001-3000', and '3000+'. The table also reports different total counts than the rest of the text.","section":"Section 3.9 and Table 8"},{"comment":"Table 7 lists GPT-4 as 'gpt-4-turbo-2024-04-09' with training cutoff Dec 2023, while Section 3.1 groups GPT-4 with GPT-4o/GPT-4mini as trained till October 2023; this discrepancy should be resolved because it affects the leakage argument.","section":"Table 7 and Section 3.1"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is invalidated by the reversed temporal split, and the post hoc exclusion of terminated trials compounds the problem. I would not recommend a minor revision. If the authors re-run the evaluation with a proper temporal split (test trials whose outcomes are released after the model cutoff), make the data and prompts fully public, and fix the metric inconsistencies, the comparison could be a useful contribution, but the current manuscript does not support its conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper compares five LLMs and HINT on clinical trial outcome prediction using a newly annotated dataset. The curation and labeling are real work, and the head-to-head question is reasonable. But the central comparison does not measure prediction. The authors restricted trials to those whose primary completion date fell before each model's training cutoff—e.g., GPT-4o: completion before Oct 2023. That means the outcome was public before the model was trained, so the model can have memorized the trial from pretraining data. With the prompt containing title and summary, retrieval is trivial. So every headline metric—GPT-4o's balanced accuracy, the claim that LLMs miss negatives, HINT's specificity advantage—could be recall, not forecasting.\n\nThe problems pile up. They excluded terminated trials post hoc after seeing poor LLM performance, which is test-set filtering. The abstract and Table 1 disagree on the main GPT-4o result (0.573/0.212 vs 0.532/0.098). HINT's specificity is 0.473, below chance, but the text sells it as a strength. The GPT-4 family is assessed on 54–74 trials, so those sub-analyses are noise. No confidence intervals, no public code or data.\n\nCredit where due: the labeling protocol is described concretely, and running modern LLMs and HINT on the same trials is a legitimate design. But as presented, the answer is not usable. The dataset could be re-split correctly—test trials completed after the training cutoff—and the comparison re-run. As is, the paper doesn't support its conclusion.\n\nWho is this for? A reader who wants a template for evaluating LLMs on trial outcomes might skim the methods, but the numbers should not be cited. I would not send this to peer review as is; it needs a redesign of the split and honest reporting. If the authors redo the evaluation properly, the benchmark could be worth revisiting.","headline":"The central comparison is invalidated by a reversed temporal split that lets LLMs memorize outcomes; the abstract and main table also disagree on the headline numbers.","tokens_in":14953,"tokens_out":2426,"would_cite":false,"duration_ms":21489,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that GPT-4o is the best large language model tested for predicting clinical trial outcomes, yet every LLM tested overlooks most failed trials, while the HINT model uniquely catches negative outcomes.","keywords":["clinical trial outcome prediction","large language models","GPT-4o","HINT model","negative sample recognition","balanced accuracy","Matthews correlation coefficient","trial termination prediction"],"falsifier":"Re-run the same evaluations on trials whose outcomes were not public before each model's training cutoff, for example trials completed after the cutoff, and ask whether GPT-4o's balanced-accuracy lead over HINT persists. A cheap consistency check is to rerun the exact 74-trial GPT-4o evaluation: the abstract reports a balanced accuracy of 0.573 and an MCC of 0.212, while Table 1 reports 0.532 and 0.098, so only one set of numbers can reproduce.","tokens_in":14029,"feed_emoji":"🧪","tokens_out":9573,"duration_ms":133174,"temperature":0.7,"pith_summary":"Clinical trials fail for many reasons, and the cost of a failed late-stage trial runs into tens of millions of dollars. The paper asks whether large language models, which read trial protocols like a human would, can predict which trials will succeed, and how they compare with HINT, a purpose-built deep learning model for the same task. The central claim is that GPT-4o is the strongest LLM on balanced accuracy, but that all LLMs are heavily biased toward predicting success and almost never flag failures; HINT is the only model with substantial specificity, so it is the only one that can identify negative outcomes. The authors take these to be complementary profiles and conclude that hybrid LLM-plus-HINT systems would combine GPT-4o's predictive power with HINT's ability to spot failing trials. A reader should care because the paper is an explicit head-to-head comparison of the current LLM generation against a dedicated trial-outcome model, on the same public registry data.","feed_headline":"GPT-4o tops LLMs at trial outcome prediction, but misses failures","feed_subtitle":"All tested LLMs rarely flag failed trials; pairing GPT-4o with the HINT model could catch them.","key_machinery":"The machinery is two families of predictors. The LLM machinery is a prompt: each model receives the trial title, summary, conditions, interventions, primary outcome measures, and trial type, and must answer 'success' or 'failure' in one word. The HINT machinery is a hierarchical interaction network, a dynamic-attention graph neural network that embeds drug SMILES, disease ICD-10 codes, and eligibility criteria and outputs a success probability, with 0.5 as the threshold. The evaluation machinery is the metric set of balanced accuracy, Matthews correlation coefficient, recall, and specificity, chosen because the dataset is imbalanced at roughly 79 percent successful trials. A final piece of machinery is cutoff-based data selection: each model is tested only on trials whose primary completion date precedes that model's training cutoff, which the authors assume prevents outcome memorisation.","core_discovery":"The study's central discovery is that no single model predicts clinical trial outcomes well, but the models fail in opposite directions. GPT-4o, the best LLM, is reported to reach a balanced accuracy of 0.573 and an MCC of 0.212 in the abstract, because it classifies most trials as successful and catches few negative outcomes. HINT, by contrast, has the highest specificity (0.473) and a balanced accuracy of 0.525, making it the only model that can reliably recognise failed trials. The paper further finds that LLMs are strongest on Phase I trials and Overall Survival endpoints, while HINT is more stable across phases and handles complex endpoints such as Objective Response Rate, but underperforms on oncology trials, which make up 30 to 37 percent of the data. The authors conclude that the two families are complementary and advocate hybrid systems rather than either model alone.","pith_inferences":["The authors stop at advocating hybrids; I would test a concrete two-stage design in which GPT-4o screens every trial and every trial it labels 'success' is re-checked by HINT, using HINT's specificity as a go/no-go gate for the most expensive decisions.","A prospective test that the paper does not run would collect trials whose results are posted after the paper's data-collection date; if GPT-4o's edge shrinks, the leaderboard here would reflect memorisation more than prediction.","The abstract-to-Table 1 discrepancy in GPT-4o's numbers (0.573 versus 0.532 balanced accuracy, 0.212 versus 0.098 MCC) is my main reason to treat the exact ranking as provisional until one set of numbers is rerun and confirmed."],"forward_implications":["If the complementarity claim holds, a combined system that uses GPT-4o for overall scoring and HINT's negative prediction as a filter would flag failed trials that pure-LLM pipelines miss.","If the LLMs' near-zero specificity is a general property, any LLM-based trial screening tool needs an explicit negative-class mechanism or calibration before it can support go/no-go decisions.","If HINT's oncology shortfall is intrinsic, hybrid designs for cancer trials should weight LLM predictions more heavily for neoplasms, the opposite of the overall pattern.","If the phase and duration trends are real, deployment should be stratified: use LLMs for early-phase and simple-endpoint decisions and HINT for later-phase and complex-endpoint decisions."],"supporting_citations":[{"why":"Supplies the HINT model and its training checkpoint, the main non-LLM comparator and the basis for HINT's input format and phase-specific preprocessing.","marker":"[9]"},{"why":"Provides the clinical trial complexity scoring used to explain HINT's poor oncology performance.","marker":"[14]"},{"why":"The ClinicalTrials.gov database is the sole data source for the curated trial records used in all model evaluations.","marker":"[16]"},{"why":"Defines GPT-3.5's version and training cutoff, which sets the evaluation set for GPT-3.5.","marker":"[18]"},{"why":"Defines Llama3-8B-Instruct and its training cutoff, setting the evaluation set for Llama3.","marker":"[19]"},{"why":"Supplies GPT-4's technical report and training details used to set its evaluation cutoff.","marker":"[20]"},{"why":"Defines GPT-4o Mini and its cutoff, setting GPT-4mini's evaluation set.","marker":"[21]"},{"why":"Defines GPT-4o and its cutoff, setting GPT-4o's evaluation set.","marker":"[22]"}],"fun_headline_variants":["GPT-4o and HINT: opposite trial flops, better together","LLMs miss failed trials; HINT catches them: hybrid wins","Best LLM GPT-4o blinds to failures; HINT fills gap","No single model predicts trial outcomes: hybrid necessary","GPT-4o's strength plus HINT's specificity predicts trials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that limiting each model's test set to trials whose primary completion date falls before that model's training cutoff prevents data leakage; in reality, outcomes for those trials were public before the cutoff, so the models may have memorised them rather than predicted them.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o and HINT: opposite trial flops, better together","LLMs miss failed trials; HINT catches them: hybrid wins","Best LLM GPT-4o blinds to failures; HINT fills gap","No single model predicts trial outcomes: hybrid necessary","GPT-4o's strength plus HINT's specificity predicts trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"total_tokens":2356,"prompt_tokens":935,"completion_tokens":1421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1344}},"tokens_in":551,"tokens_out":1421,"duration_ms":8990,"temperature":1.0,"reasoning_tokens":1344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:56:09.711366+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same evaluations on trials whose outcomes were not public before each model's training cutoff, for example trials completed after the cutoff, and ask whether GPT-4o's balanced-accuracy lead over HINT persists. A cheap consistency check is to rerun the exact 74-trial GPT-4o evaluation: the abstract reports a balanced accuracy of 0.573 and an MCC of 0.212, while Table 1 reports 0.532 and 0.098, so only one set of numbers can reproduce.","supporting_citations":[{"cited_title":"Patterns 3(4) (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the HINT model and its training checkpoint, the main non-LLM comparator and the basis for HINT's input format and phase-specific preprocessing."},{"cited_title":"From RAGs to riches: Utilizing large language models to write documents for clinical trials","cited_arxiv_id":"2402.16406","evidence_quote":"Provides the clinical trial complexity scoring used to explain HINT's poor oncology performance."},{"cited_title":"https://clinicaltrials.gov/","cited_arxiv_id":null,"evidence_quote":"The ClinicalTrials.gov database is the sole data source for the curated trial records used in all model evaluations."},{"cited_title":"https://chatgpt.com/g/g-F00faAwkE-open-a-i-gpt-3-5","cited_arxiv_id":null,"evidence_quote":"Defines GPT-3.5's version and training cutoff, which sets the evaluation set for GPT-3.5."},{"cited_title":"https://openai","cited_arxiv_id":null,"evidence_quote":"Defines GPT-4o Mini and its cutoff, setting GPT-4mini's evaluation set."},{"cited_title":"https://openai.com/index/hello-gpt-4o/","cited_arxiv_id":null,"evidence_quote":"Defines GPT-4o and its cutoff, setting GPT-4o's evaluation set."}],"review_version":1}