{"id":"ee3b8a91-768e-4b9a-b530-1c426fa7d7fd","arxiv_id":"2505.08508","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TrialMatchAI, a retrieval-augmented open-source LLM pipeline, reports strong trial retrieval and over 90% criterion-level eligibility accuracy, though key evaluations rely on selected cohorts and positive-only expert samples.","lead":"TrialMatchAI is an open-source system that recommends clinical trials for cancer patients by combining keyword search, vector search, and fine-tuned language models that judge each eligibility criterion. It matters because patient recruitment is a bottleneck in oncology, and a private, locally deployable matcher could reduce manual screening time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 92% real-world top-20 recall is computed on a hand-selected 52-patient sub-cohort with a 217-trial pool, so the headline real-world claim is not yet established for unselected referral populations.","rationale":"The central claim requires that the real-world validation be representative of the population where the tool would be deployed. The selection rule in Section 4.1 directly undermines that requirement: the 52 patients were chosen precisely because their WIDE fields align with the eligibility criteria of their ground-truth trials, and the search was confined to a 217-trial pool curated at NKI. The paper's own limitation statement about limited overlap in the 947-patient dataset makes this explicit. With only 52 patients, the difference between 84.6% top-5 recall and 92.3% top-20 recall is just four patients, so the headline number is fragile. The Ideal Candidates dataset is also partly circular because the patient profiles are generated from the trial criteria and then ranked against those same trials, but the TREC benchmarks are not circular and provide independent evidence of strong retrieval and ranking. The released code and Zenodo data are real strengths and make the proposed re-evaluation feasible. My concern does not change the reader's conditional verdict; it sharpens the condition: the paper should report performance on an unselected cohort with error bars and a full-corpus comparison before claiming real-world 92% recall.","tokens_in":40955,"tokens_out":7063,"duration_ms":70869,"concrete_test":"Run TrialMatchAI on all 947 WIDE patients (or a random unselected sample of comparable size) using first the same 217-trial NKI pool and then the full 61,731 cancer-related trial corpus, and report top-5/top-20 overall recall, MRR, and 95% confidence intervals. If recall on the unselected cohort is materially below 92.3% (for example, below 80%), the headline real-world claim does not generalize and should be rephrased as a proof-of-concept on an enriched sub-cohort.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 selects the WIDE sub-cohort by requiring that at least 75% of the available patient fields match eligibility criteria of the assigned ground-truth trials, and then restricts retrieval to a predefined pool of 217 molecularly driven NKI trials. The paper itself states that the full 947-patient WIDE dataset \"presented limited overlap between the fields recorded in this study and the comprehensive information in electronic health records, which were used for the original orientation of patients to their respective trials.\" Consequently, the 92.3% top-20 \"Overall Recall\" (48 of 52 patients) is a performance estimate for a deliberately evaluable subset, not for the cohort a deployed system would encounter. The selection rule guarantees that the available text is aligned with the ground-truth criteria, which inflates lexical and semantic retrieval, and the 217-trial search space cannot test the system's ability to discover the relevant trial from the full 61,731-trial cancer-related corpus. A related overstatement is the biomarker expert check: the 100 biomarker criteria were sampled only from model-predicted \"Met\" calls, so the reported 91% measures precision of positive calls, not accuracy across Met/Not-Met/Unclear/Irrelevant categories, although the TREC expert evaluation does include all categories.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TrialMatchAI presents an end-to-end, open-source, locally deployable clinical trial matching system built on fine-tuned Gemma-2-2B and Phi-4 models within a RAG pipeline. The system performs entity normalization, hybrid BM25/vector retrieval, LLM-based reranking, and chain-of-thought criterion-level eligibility classification. The authors evaluate on the TREC 2021/2022 clinical trials datasets, a synthetic 'Ideal Candidates' dataset, and a 52-patient real-world metastatic cancer cohort from the NKI WIDE study. Headline results are >90% recall within the top 3% of the TREC corpus, 92.3% top-20 overall recall on the real-world cohort, and >90% accuracy in criterion-level eligibility classification per expert review.","tokens_in":1719,"tokens_out":1837,"duration_ms":65839,"significance":"If the claims are sustained, the paper makes a useful contribution: it demonstrates that small open-source models can achieve retrieval and ranking performance competitive with a proprietary LLM-based system (TrialGPT) on the standard TREC clinical trials benchmark, and it ships code, fine-tuned adapters, and data on Zenodo/GitHub, which is a reproducible and privacy-preserving alternative. The TREC evaluation provides an external, independently labeled benchmark, which is a notable strength over purely synthetic or self-constructed evaluations. The main value of the paper is in the architecture and open-source release; the real-world and criterion-level accuracy claims need to be calibrated to the actual evaluation designs used.","major_comments":[{"comment":"The 92.3% top-20 recall is estimated on a sub-cohort of 52 patients selected because at least 75% of available WIDE fields matched the eligibility criteria of their assigned ground-truth trials, and retrieval was restricted to a predefined pool of 217 molecularly driven trials. The paper itself states that the full WIDE dataset 'presented limited overlap between the fields recorded in this study and the comprehensive information in electronic health records, which were used for the original orientation of patients to their respective trials.' Therefore the headline real-world recall is a performance estimate for a deliberately evaluable subset, not for an unselected referral population, and the restricted search space cannot measure the system's ability to discover relevant trials from the full 61,731-trial cancer-related corpus. The authors should either evaluate on an unselected cohort with the full corpus or explicitly reframe the 92% figure as a feasibility result for a curated pool, and report a confidence interval for this proportion (48/52).","section":"Section 4.1, paragraph 'To ensure meaningful evaluation'; Table 2"},{"comment":"The biomarker expert evaluation sampled 100 criteria that were all model-predicted 'Met' calls, and the reported 91% is the fraction of those positive calls judged correct by experts. This measures precision of positive calls, not overall classification accuracy across Met/Not-Met/Unclear/Irrelevant for inclusion criteria (and the analogous categories for exclusion criteria). The abstract and Discussion claim 'over 90% accuracy in criterion-level eligibility classification,' which overstates what this experiment measures. The authors should report a full contingency table over all prediction categories, or at least rephrase the claim as precision on positive calls, and propagate this correction to the abstract and Discussion.","section":"Section 2.4, paragraph 'Building on our evaluation...'"},{"comment":"The 'Ideal Candidates' benchmark generates synthetic patient profiles from the eligibility criteria of the ground-truth trial via GPT-4o-mini, so the ground truth is constructed from the same source text that the retrieval and ranking stages are matching against. The 95% top-2 result is therefore expected to be inflated by paraphrase and lexical similarity to the query source and should be presented as a sanity check rather than as independent evidence of matching ability. The paper should explicitly acknowledge this circularity, and the result should not be used in the abstract or introduction as a standalone validation of state-of-the-art performance.","section":"Section 2.2, 'Validating AI-based Matching: The Ideal Candidates Dataset'"},{"comment":"The abstract states that TrialMatchAI is 'outperforming existing AI-based tools that rely on significantly larger, proprietary GPT models,' but the reported mean nDCG@10 (0.7232) is slightly lower than TrialGPT's highest aggregated average (0.7275), with only p@10 higher (0.6865 vs 0.6688). The text in Section 2.2 correctly says 'highly competitive'; the abstract overstates the comparison. In addition, these small differences are reported without any significance tests or confidence intervals, so claims of superiority are not supported. The authors should align the abstract with the measured metrics and add uncertainty quantification for the TREC comparison.","section":"Section 2.2, comparison with TrialGPT; Abstract"}],"minor_comments":[{"comment":"The equations for the inclusion score, exclusion score, and composite score S are missing from the rendered text; the formulas appear as blank placeholders after 'computed as follows:'. Please provide the actual equations.","section":"Section 4.5"},{"comment":"The definition of P@k contains a spurious factor: '𝑃@𝑘 = 1/2 × ...' should be the standard mean of rel(i) over the top k positions. Please verify and correct the formula.","section":"Supplementary Materials, Section E"},{"comment":"The text says 'Approximately 60% (61,731) of these trials are interventional and cancer-related' and Supplementary F says 'Over 99% of these trials are interventional.' Please reconcile these numbers, since a trial that is both interventional and cancer-related would be 100% interventional by definition.","section":"Section 4.1 and Supplementary Section F"},{"comment":"The biomarker patient profiles used in the expert evaluation were 'completely rephrased' by Phi-4 from MTB reports, making them semi-synthetic; this limitation should be stated in the main text alongside the 91% result, not only in the methods description.","section":"Section 2.4"},{"comment":"Please provide per-category counts and confidence intervals (or at least denominators) for the expert evaluation percentages so the reader can assess the reliability of the 88–98% figures, especially given the small numbers in some categories.","section":"Figure 3B and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with credible TREC benchmark results and a commendable open-source release. The main issues are overstatements of the real-world 92% recall and the biomarker criterion-level accuracy, both of which stem from evaluation design choices that the authors can address with reanalysis and rewording. The missing equations and minor statistical details are readily fixable. I would encourage the editor to request a revision rather than reject, because the underlying TREC-based retrieval claims are externally grounded and the system is a useful public resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TrialMatchAI is a serious, reproducible piece of systems work. The genuinely new parts are the full open-source pipeline (BGE-M3 + BM25 retrieval, fine-tuned Gemma-2-2B reranking, fine-tuned Phi-4 chain-of-thought classifier), the curated 61k-trial cancer corpus, and the WIDE real-world oncology evaluation. The TREC 2021/2022 numbers are credible: medians are reported, the retrieval recall curves look sane, and the external benchmark gives independent grounding for the central retrieval claims. The expert evaluation of 950 TREC patient-criterion pairs covers all four categories (Met/Not Met/Unclear/Irrelevant) and is a genuine strength. Code and data on Zenodo/GitHub are a real plus.\n\nThe soft spots are real but localized. The headline 92% top-20 recall on the WIDE data is computed on a sub-cohort of 52 patients selected because at least 75% of their available WIDE fields overlapped with the eligibility criteria of the ground-truth trials. That selection rule inflates lexical and semantic retrievability, and the search space is a predefined pool of 217 molecularly driven NKI trials, not the full corpus. So that particular number does not yet generalize to an unselected referral population. The biomarker expert check is also a precision-of-positive-calls measure: it samples only model-predicted \"Met\" criteria, so it cannot support the abstract's \"over 90% accuracy\" claim across categories. The Ideal Candidates benchmark is partly circular, as the reader notes, but its role as a proof-of-concept is minor. The comparison to TrialGPT is a bit hand-wavy: average nDCG@10 and p@10 are close, but there are no significance tests or confidence intervals, and the aggregation differs.\n\nNone of this undercuts the TREC results, which are the core technical claim. The paper is honest about the WIDE dataset's limited overlap and about confabulation rates. It is a solid engineering contribution with careful, mostly appropriate evaluation, let down by an overreaching abstract and discussion.\n\nWho gets value: researchers building patient-trial matching systems, especially those who want a locally deployable, privacy-preserving baseline that beats a naive RAG stack. The paper deserves a serious referee: conditional acceptance, with requests for an unselected-cohort evaluation, a full confusion-matrix biomarker expert study, and toned-down claims.","headline":"TrialMatchAI is a genuinely useful open-source system with credible TREC numbers, but the 92% real-world headline rests on a hand-picked 52-patient sub-cohort and a 217-trial pool; the system deserves peer review, not desk rejection.","tokens_in":41785,"tokens_out":1039,"would_cite":true,"duration_ms":12750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TrialMatchAI claims that a fully open-source, locally deployable pipeline of fine-tuned lightweight LLMs can match patients to clinical trials as accurately as proprietary LLM-based systems, retrieving a relevant trial for 92% of oncology…","keywords":["clinical trial matching","patient recruitment","retrieval-augmented generation","large language models","criterion-level eligibility classification","precision oncology","biomarker-driven trials","chain-of-thought reasoning"],"falsifier":"Run TrialMatchAI on an unselected consecutive oncology cohort against the full 61,731-trial cancer database and measure top-20 recall; a result well below 92% would show the headline real-world figure depends on the cohort-selection filter rather than general matching ability.","tokens_in":40709,"feed_emoji":"🧬","tokens_out":8048,"duration_ms":72232,"temperature":0.7,"pith_summary":"TrialMatchAI sets out to show that clinical trial matching can be automated without relying on proprietary AI APIs. The system combines hybrid lexical and semantic retrieval, LLM-based reranking, and criterion-level eligibility classification to recommend trials for cancer patients. If the results hold, hospitals could run a privacy-preserving matching service locally while clinicians see why each trial was recommended. The paper reports 92% top-20 recall on a real-world oncology cohort, over 90% recall within the top 3% of trials on TREC benchmarks, and expert-validated criterion-level accuracy above 90%.","feed_headline":"Open-source trial matcher hits 92% top-20 recall for cancer patients","feed_subtitle":"Small, locally run LLMs find a relevant trial for 92% of oncology patients within the top 20.","key_machinery":"The load-bearing machinery is criterion-level eligibility assessment performed by a fine-tuned Phi-4 model prompted with medical Chain-of-Thought reasoning, a technique that forces the model to write intermediate reasoning steps before classifying each criterion. It labels inclusion criteria as Met, Not Met, Unclear, or Irrelevant and exclusion criteria as Violated, Not Violated, Unclear, or Irrelevant, and must justify each label from the provided patient data and retrieved trial context. Before that classifier runs, a retrieval-augmented pipeline narrows the search: biomedical named-entity recognition and normalization standardize patient and trial text, Elasticsearch executes a hybrid BM25 plus k-nearest-neighbor vector search, and a fine-tuned Gemma-2-2B model reranks candidate trials by aggregating criterion-level relevance. A final scoring step then ranks trials so that satisfied inclusion criteria count positively and violated exclusion criteria count negatively, with Unclear and Irrelevant labels treated as neutral.","core_discovery":"The central claim is that a fully open-source, locally deployable system built from fine-tuned lightweight open-source LLMs can match patients to clinical trials as effectively as proprietary LLM-based tools. In real-world validation, 92% of oncology patients had at least one relevant trial in the top 20 recommendations, and expert assessment put criterion-level eligibility classification accuracy above 90%, with the best results on biomarker-driven matches. The paper further claims that this removes the cost, privacy, and reproducibility barriers of API-driven models. The proposed mechanism is an end-to-end pipeline that normalizes biomedical entities, retrieves candidates with BM25 plus dense-vector search, reranks with a fine-tuned Gemma-2-2B model, and classifies each eligibility criterion with a fine-tuned Phi-4 model using medical Chain-of-Thought reasoning.","pith_inferences":["Beyond the paper, the 92% real-world recall is an upper-bound estimate: the 52-patient sub-cohort was selected for at least 75% overlap between available WIDE fields and assigned-trial criteria, and retrieval ran against a fixed pool of 217 molecularly driven trials, not the full database.","Beyond the paper, the same criterion-level classification output could be aggregated to audit trial designs, since persistent Unclear labels across patients would reveal eligibility criteria that routinely cannot be assessed from typical electronic health record data.","As a testable extension, we suggest a prospective deployment on consecutive molecular tumor board cases without cohort prefiltering, with clinician acceptance of recommendations as the endpoint."],"forward_implications":["Hospitals can run trial matching entirely on local infrastructure, so patient data never has to leave the institution for a commercial API.","Clinicians can audit each recommendation because the system attaches a criterion-by-criterion justification to every eligibility decision.","The open modular design means better open-source LLMs can be swapped in without redesigning the retrieval and scoring layers.","Biomarker-driven trials, where matching depends on genomic findings in tumor-board reports, are the setting where the system reports the highest criterion-level accuracy.","TREC retrieval results suggest that over 90% of relevant trials can be captured while discarding more than 95% of the search space, which would reduce the manual screening burden for recruiting teams."],"supporting_citations":[{"why":"Provides the proprietary-GPT TrialGPT system and benchmark numbers that TrialMatchAI compares against.","marker":"[10]"},{"why":"Supplies the TREC 2022 clinical trials benchmark with relevance judgments used for retrieval and ranking evaluation.","marker":"[30]"},{"why":"Supplies the TREC 2021 clinical trials benchmark with relevance judgments used for retrieval and ranking evaluation.","marker":"[31]"},{"why":"Describes the WIDE study that provided the 52-patient real-world oncology cohort and ground-truth trial assignments.","marker":"[32]"},{"why":"Introduces chain-of-thought prompting, the reasoning technique the eligibility classifier is fine-tuned to follow.","marker":"[33]"},{"why":"Defines retrieval-augmented generation, the framework that anchors eligibility reasoning in retrieved trial text.","marker":"[34]"},{"why":"Is the open-source Gemma-2 model fine-tuned to rerank candidate trials at the criterion level.","marker":"[38]"},{"why":"Is the open-source Phi-4 model fine-tuned for criterion-level eligibility classification and justification generation.","marker":"[39]"},{"why":"Provides BGE-M3 dense embeddings used for semantic retrieval of trials and patient queries.","marker":"[45]"}],"fun_headline_variants":["Open-source AI matches cancer patients to trials in top 20 for 92%","92% of cancer patients get a trial match in top 20 via open-source AI","Lightweight LLMs match 92% of cancer patients to trials in top 20","Privacy-first AI trial matching: 92% cancer patients hit top 20","AI trial matcher: 92% oncology patients find a match in top 20"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-world evaluation rests on the assumption that a 52-patient sub-cohort selected for having at least 75% of their available WIDE fields match the eligibility criteria of their assigned trials is representative of real-world referral populations.","fun_headline_variants_meta":{"raw":{"variants":["Open-source AI matches cancer patients to trials in top 20 for 92%","92% of cancer patients get a trial match in top 20 via open-source AI","Lightweight LLMs match 92% of cancer patients to trials in top 20","Privacy-first AI trial matching: 92% cancer patients hit top 20","AI trial matcher: 92% oncology patients find a match in top 20"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3677,"prompt_tokens":975,"completion_tokens":2702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2594}},"tokens_in":591,"tokens_out":2702,"duration_ms":17229,"temperature":1.0,"reasoning_tokens":2594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:53:03.670562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TrialMatchAI on an unselected consecutive oncology cohort against the full 61,731-trial cancer database and measure top-20 recall; a result well below 92% would show the headline real-world figure depends on the cohort-selection filter rather than general matching ability.","supporting_citations":[],"review_version":1}