{"id":"792a3d23-5824-4266-9c6e-0081799f5a74","arxiv_id":"2501.11114","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Open-source LLMs achieve moderate F1 on straightforward clinical trial criteria but underperform challenge-winning systems on criteria requiring fine-grained reasoning across three n2c2 datasets.","lead":"This paper tested seven open-source language models on three medical datasets to see if they could automatically flag which patients are eligible for clinical trials. The models handled simple, clearly defined criteria quite well, but performed poorly on nuanced medical judgments and did not beat the best original challenge systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's 'generative LLMs generally underperform' conclusion overreaches from seven small open-source models with one generic prompt; a stronger open model or tuned prompt could reverse the nuanced-criteria finding.","rationale":"The paper is honest and internally consistent: the same prompt family is applied across seven models, and the per-criterion gaps against official challenge winners are large and clearly presented. The weakest point is not the computation but the inference target. Section 6.1's sentence about 'generative LLMs' is the central claim, and it is broader than the evidence, which covers only small open-weight models with generic prompting. The reader's weakest assumption identified exactly this generalization, and I agree. A conditional verdict remains appropriate because the flaw is a scope overclaim rather than a broken method: Tables 3–6 demonstrate real limitations for the tested models, and the paper's own limitation section acknowledges the absence of commercial and larger models. To make the conditional acceptance actionable, the authors should either restrict the conclusion to the tested model class or include a larger open model in the evaluation. The proposed concrete test is feasible under the data-use agreement because it uses open weights on local compute and requires no additional data sharing.","tokens_in":12996,"tokens_out":4946,"duration_ms":46112,"concrete_test":"Run the n2c2-2018 test evaluation with the same 5-shot iterated prompt template on a representative 70B-class open-weight instruction-tuned model (e.g., Llama-3-70B-Instruct or Mixtral-8x7B-Instruct), using the identical patient text input protocol, and compare per-criterion F1 to vicuna-13b and the official campaign best in Table 3. If the new model's mean F1 on the nuanced criteria (ABDOMINAL, MI-6MOS, MAKES-DECISIONS, DRUG-ABUSE, ALCOHOL-ABUSE, ADV-CAD) rises to within 0.05 of the campaign best, the paper's 'LLMs perform poorly on nuanced criteria' claim fails to generalize to larger open models; if it stays near vicuna-13b levels, the scope-limited conclusion is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 6.1 — 'our results run counter to a common view that generative LLMs generally outperform existing methods, or are even likely to solve the problem of patient selection once and for all' — is supported only by results from seven open-weight models, the largest being vicuna-13b, evaluated with a single generic prompting family (Section 4.1). The paper itself concedes in Section 6.2 that commercial LLMs were excluded and prompt optimization/CoT variants were not explored. The empirical pattern that matters most, LLMs collapsing on criteria requiring inference (e.g., MI-6MOS 0.414, ALCOHOL-ABUSE 0.375, DRUG-ABUSE 0.222 vs. campaign bests of 0.876, 0.897, and 0.920 in Table 3), is real for these specific small models. But the claim about 'generative LLMs' generally requires that this pattern is invariant across model scale, instruction tuning, and prompting. The included models already show enormous variance — open-orca-mistral-7b gets 0.000 on several criteria while vicuna-13b gets 0.7–0.8 — so a 70B-class model or a better-tuned prompt could plausibly close the gap, especially on criteria involving dose thresholds or temporal inference. Without a broader model sweep or an explicit restriction of the conclusion to 'small open-source LLMs with generic prompting,' the headline conclusion is a scope overreach.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates seven open-weight LLMs on three n2c2 clinical cohort-selection benchmarks (2006 smoking status, 2008 obesity comorbidities, 2018 trial eligibility criteria) using basic, few-shot, and iterated few-shot prompting, with 5-fold cross-validation on the training data and comparison to official challenge winners. The main finding is that the best tested models (vicuna-13b and mistral-7b-instruct) achieve competitive or strong F1 on simple criteria (e.g., smoking class, aspirin use) but perform poorly on nuanced criteria such as MI-6MOS (0.414), ALCOHOL-ABUSE (0.375), and DRUG-ABUSE (0.222), and the paper concludes in Section 6.1 that generative LLMs do not generally outperform existing methods for patient selection.","tokens_in":13261,"tokens_out":6151,"duration_ms":49926,"significance":"If the claim is scoped to the tested models and prompting regime, the paper provides a useful, reproducible benchmark: it uses public datasets, reports per-criterion F1, compares to official campaign winners, and uses open-weight models that can be run locally. The per-task breakdown in Tables 2-6 is informative, and the negative result on fine-grained criteria is a valuable cautionary data point. However, the broad conclusion about 'generative LLMs' is not supported by the evidence, as only seven small open-weight models and one generic prompting family were tested, which the authors acknowledge in Section 6.2. With a scoped conclusion and a clarified evaluation protocol, the paper would be a solid empirical contribution.","major_comments":[{"comment":"The sentence 'our results run counter to a common view that generative LLMs generally outperform existing methods, or are even likely to solve the problem of patient selection once and for all' overreaches beyond the evidence. The evaluation covers seven open-weight models, the largest being vicuna-13b, with a single generic prompt family and no commercial or larger open models, as Section 6.2 concedes. The empirical pattern is real for these specific models, but the general claim requires invariance across model scale, instruction tuning, and prompting. Please restrict the conclusion to 'the small open-source LLMs and generic prompts evaluated here' or substantially broaden the evaluation.","section":"Section 6.1"},{"comment":"The method text says 'the average and standard deviation of their F1 score across the five folds', while the table captions say 'median and standard deviation over 5 runs' and report a single value with a parenthesized value. With the skewed per-fold results visible for models like open-orca-mistral-7b (0.000 on several criteria), the median and mean can diverge substantially. Please state explicitly which statistic is reported, report both where the distribution is skewed, and describe how the standard deviation is computed for a median.","section":"Section 4.1 and Tables 2/3"},{"comment":"Within-model and between-model comparisons are made using means/medians and standard deviations without any significance test, bootstrap confidence intervals, or correction for multiple comparisons. The claim in Section 6.1 that vicuna-13b has 'more stability (lower standard deviation)' is not formally tested. Given that five runs per fold are reported, please add bootstrap confidence intervals or a paired significance test across folds, and state whether sampling temperature and random seeds were fixed or varied.","section":"Tables 2-6 and Section 6.1"},{"comment":"The section is titled 'Fine-tuning' but describes iterative selection of few-shot examples, not gradient-based fine-tuning. No training hyperparameters, epochs, losses, or updated weights are mentioned. As written, the stage-2 results in Tables 4-6 could be mistaken for fine-tuned outputs. Please either retitle the section (e.g., 'Few-shot example selection for n2c2-2006/2008') or provide the actual fine-tuning setup.","section":"Section 4.2"},{"comment":"The relationship between the five-fold cross-validation, the train/validation split, and the reported results on the 'val set' (Table 2) and 'test set' (Table 3) is ambiguous. It is unclear whether the five folds are formed from the original training set, how many folds are used to select few-shot examples, and whether the reported scores are averaged over folds for a single validation/test split. Please specify the exact data flow: how the few-shot examples are selected from each fold, how the validation set is used for model selection, and how the official test set is evaluated.","section":"Section 4.1 and 5-fold protocol"}],"minor_comments":[{"comment":"The word 'interannotor' should be 'inter-annotator'; similarly, 'critera' appears in Section 2 (first paragraph) and should be 'criteria'.","section":"Section 3.1"},{"comment":"The row 'best from campaign' for ASP-MI (0.770) is lower than several LLM scores (e.g., vicuna-7b 0.878, mistral-7b-instruct 0.875); this is worth a sentence in the discussion, as it qualifies the overall underperformance narrative.","section":"Table 3"},{"comment":"The text says results are reported for 'the n2c2-2006 and n2c2-2008 challenge test dataset for the vicuna-13b model', but Table 4 also reports mistral-7b-instruct; align the prose with the tables.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The authors' decision to restrict to open-source models because of the n2c2 data-use agreement is legitimate, but it should not be used as cover for a broad claim about 'generative LLMs'. If the journal's scope prioritizes general claims about LLM capabilities, the overreach in Section 6.1 is a blocking issue; if it accepts scoped empirical benchmarks with negative results, the paper is appropriate after revision. I would support acceptance of a revised version that narrows the conclusion and resolves the protocol ambiguities in Sections 4.1 and 4.2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take on arXiv:2501.11114: it's a useful, careful benchmark of seven small open-source LLMs on three n2c2 cohort selection tasks, and the reported pattern – models do okay on simple criteria, badly on nuanced ones – is real for the models and prompts tested. The new thing here is breadth: previous LLM studies on n2c2-2018 used GPT-3.5/GPT-4; this is the first systematic look at open-weight models across n2c2-2006, 2008, and 2018 with a consistently applied prompt family and comparison to official challenge winners. The iterated few-shot selection strategy is a reasonable contribution, and the authors are admirably clear about what they didn't do: no CoT tuning, no prompt optimization, no commercial APIs. They also report median and std across five folds, which is more than many LLM papers do.\n\nThat said, the headline conclusion in Section 6.1 – results 'run counter to a common view that generative LLMs generally outperform existing methods' – overreaches. The evidence is from seven models, largest being vicuna-13b, with one generic prompting family. The stress-test note is right: a 70B class model or a better-tuned prompt could plausibly close the gap on criteria involving dose thresholds or temporal inference. The paper's own Limitations section concedes this, but the abstract and conclusion still state it broadly. Scope it to 'small open-source LLMs with generic prompting' and it's a solid finding.\n\nTwo concrete flaws to flag. First, Section 4.2 is titled 'Fine-tuning' but never tunes weights; it's iterative few-shot example selection. That's a mislabel that will confuse readers. Second, the supplementary few-shot prompt for ASP-MI contains a reasoning error: 'Ecotrin 325 mg PO QD' is described as '325 mg is a low dose (less than 325 mg)' and the example is labeled Yes. That is a factual mistake in the prompt itself, which likely tampers with dose-related criteria. Fixing that alone might shift some results.\n\nThe evaluation protocol also has gaps: no significance testing, no control for decoding stochasticity (the std reported is across folds, not across repeated runs with different seeds), and the few-shot example selection algorithm is not specified tightly enough for exact replication. No code or prompts are released. These are fixable, not fatal.\n\nVerdict: this deserves a serious referee. It's a reproducible-ish benchmark with honest limitations that, once scoped correctly, provides a useful cautionary data point for anyone building practical cohort selection systems. I'd want revisions before publication: correct the mislabel, fix the prompt bug, report significance or at least repeated-run variance, and release the prompts/code. But the core empirical work is sound enough to engage.","headline":"A useful small-model benchmark on n2c2 cohort selection, overbroad in its 'generative LLMs underperform' conclusion; the empirical pattern is real but confined to the tested models and prompt family.","tokens_in":13784,"tokens_out":3095,"would_cite":true,"duration_ms":28060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs fall short on nuanced patient-selection criteria","keywords":["text extraction","classification","clinical trial cohort selection","large language models","n2c2 challenges","few-shot prompting","chain-of-thought reasoning","open-source LLMs"],"falsifier":"Run the same three n2c2 tasks through a current larger open-weight model, or a commercial API on de-identified records that comply with the data agreement, using the same five-shot chain-of-thought prompts. If that model meets or beats the original challenge winners on the nuanced 2018 criteria—abdominal surgery, recent myocardial infarction, drug abuse, and decision-making capacity—while keeping the simple-criterion scores high, the paper's claim that generative LLMs do not generally outperform existing methods would be refuted for that model class.","tokens_in":12784,"feed_emoji":"🩺","tokens_out":11748,"duration_ms":103517,"temperature":0.7,"pith_summary":"This paper tries to establish how far open-source large language models are from being usable for clinical trial cohort selection. The authors benchmark seven models on three standard clinical-text tasks—smoking status, obesity co-morbidities, and thirteen trial eligibility criteria—using the same basic, few-shot, and iterated few-shot prompts. They find that the models match or beat earlier non-LLM systems on criteria that are explicitly stated, such as smoking class or aspirin use, but score well below the original challenge winners on criteria that require fine-grained medical reasoning, such as prior abdominal surgery, drug abuse, or the patient's decision-making capacity. The paper reads this as evidence against the common assumption that generative LLMs generally outperform existing methods or will solve patient selection outright. The central claim, in the authors' own words, is that generative LLMs do not generally outperform existing methods and are not likely to solve the problem of patient selection once and for all.","feed_headline":"Open-source LLMs fall short on nuanced patient-selection criteria","feed_subtitle":"Benchmarked on three clinical-text challenges, they beat previous systems only on simple eligibility rules.","key_machinery":"The argument is carried by a two-stage benchmark comparison. Stage one scores seven open-source LLMs on the 2018 cohort-selection task under three prompt recipes—a basic prompt, few-shot learning, and an iterated few-shot version whose examples are selected from earlier mistakes—each run with five-fold cross-validation and measured by per-criterion F1. Stage two takes the best-performing model from stage one and applies the same iterated few-shot procedure to the 2006 smoking and 2008 obesity tasks. The decisive comparison is between these generic-prompt LLM scores and the published best results from the original shared-task campaigns, especially the per-criterion breakdown on the 2018 task, which exposes exactly where the models fail.","core_discovery":"Using the three n2c2 challenge datasets as a controlled benchmark, the paper's central empirical claim is that open-source LLMs with generic prompting perform poorly as selection criteria become nuanced. On the 2018 cohort-selection test set, the best of the seven tested models post F1 scores ranging from below 0.1 to about 0.6 on nuanced criteria such as abdominal surgery, myocardial infarction within six months, drug abuse, and decision-making capacity, while the original challenge winners scored above 0.87 on those same criteria; on plainly worded criteria such as aspirin use the LLMs match or exceed the campaign baseline. The same split appears on the 2006 and 2008 tasks: smoking status and several co-morbidity labels are handled reasonably, while under-specified or judgment-heavy categories lose ground. The authors therefore conclude that their results run counter to the view that generative LLMs generally outperform existing methods or are likely to solve patient selection once and for all.","pith_inferences":["The paper's scope is deliberately limited to models that can run on an academic computing cluster, so its negative conclusion is a statement about that model generation and prompt recipe, not a law; the same protocol applied to newer or larger models could shift the balance on nuanced criteria.","A practical deployment that follows from the results is a two-tier screen: let an LLM flag patients who clearly pass or fail simple criteria, and send only ambiguous and judgment-heavy cases to human reviewers, cutting manual workload without trusting LLMs on the hard criteria.","The per-criterion F1 gaps on the 2018 task are a ready-made diagnostic for future model releases: any model that closes those gaps while keeping simple-criterion performance would be evidence of genuine reasoning gains rather than better pattern matching."],"forward_implications":["Simple eligibility criteria such as smoking status or aspirin use can be screened automatically with these models, reducing manual chart review for those parts of a protocol.","Criteria requiring inference or judgment—abdominal surgery history, drug abuse, decision-making capacity—should be routed to human review or rule-based verification rather than accepted from an LLM.","Choosing and iteratively curating few-shot examples is a measurable performance lever; the iterated few-shot prompt consistently beat the basic and plain few-shot prompts.","Medical-domain pretraining did not automatically help: the general-text models outperformed the medical-specific model on the 2018 tasks.","Claims that LLMs beat existing patient-selection systems need the same per-criterion benchmark comparison used here, because overall scores can hide failures on hard criteria."],"supporting_citations":[{"why":"Supplies the n2c2-2006 smoking-status records, labels, and original challenge performance used for the smoking task.","marker":"[11]"},{"why":"Supplies the n2c2-2008 obesity co-morbidity records and original challenge baselines for the fifteen textual and intuitive tasks.","marker":"[12]"},{"why":"Supplies the n2c2-2018 cohort-selection dataset, the thirteen eligibility criteria, and the official per-criterion best scores that set the comparison bar.","marker":"[13]"},{"why":"Reports high zero-shot F1 on n2c2-2018 with a proprietary LLM, the prior result whose 'LLMs solve patient selection' implication the paper's findings run counter to.","marker":"[24]"},{"why":"Reports proprietary-LLM cohort-matching results with a dynamic prompting strategy, the other prior study the paper compares against.","marker":"[25]"},{"why":"Introduces chain-of-thought prompting, which the paper's few-shot templates use.","marker":"[29]"}],"fun_headline_variants":["LLMs ace simple rules, flunk nuanced trials","Clinical trial LLMs: good on easy, bad on hard","LLMs fail nuanced patient selection in n2c2 tests","Why LLMs struggle with tricky trial eligibility","Simple criteria easy, nuanced ones trip up LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume that seven open-source models prompted with one generic recipe stand in for 'LLMs' broadly; a newer, larger, or differently prompted model could still outperform the traditional systems on the nuanced criteria.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ace simple rules, flunk nuanced trials","Clinical trial LLMs: good on easy, bad on hard","LLMs fail nuanced patient selection in n2c2 tests","Why LLMs struggle with tricky trial eligibility","Simple criteria easy, nuanced ones trip up LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1158,"prompt_tokens":880,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":496,"tokens_out":278,"duration_ms":3124,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:36:50.806939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three n2c2 tasks through a current larger open-weight model, or a commercial API on de-identified records that comply with the data agreement, using the same five-shot chain-of-thought prompts. If that model meets or beats the original challenge winners on the nuanced 2018 criteria—abdominal surgery, recent myocardial infarction, drug abuse, and decision-making capacity—while keeping the simple-criterion scores high, the paper's claim that generative LLMs do not generally outperform existing methods would be refuted for that model class.","supporting_citations":[{"cited_title":"Identifying patient smoking status from medical discharge records,","cited_arxiv_id":null,"evidence_quote":"Supplies the n2c2-2006 smoking-status records, labels, and original challenge performance used for the smoking task."},{"cited_title":"Recognizing obesity and comorbidities in sparse data,","cited_arxiv_id":null,"evidence_quote":"Supplies the n2c2-2008 obesity co-morbidity records and original challenge baselines for the fifteen textual and intuitive tasks."},{"cited_title":"Cohort selection for clinical trials: n2c2 2018 shared task track 1,","cited_arxiv_id":null,"evidence_quote":"Supplies the n2c2-2018 cohort-selection dataset, the thirteen eligibility criteria, and the official per-criterion best scores that set the comparison bar."},{"cited_title":"Zero-shot clinical trial patient matching with llms,","cited_arxiv_id":null,"evidence_quote":"Reports high zero-shot F1 on n2c2-2018 with a proprietary LLM, the prior result whose 'LLMs solve patient selection' implication the paper's findings run counter to."},{"cited_title":"Utilizing large language models for enhanced clinical trial matching: A study on automation in patient screening,","cited_arxiv_id":null,"evidence_quote":"Reports proprietary-LLM cohort-matching results with a dynamic prompting strategy, the other prior study the paper compares against."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting, which the paper's few-shot templates use."}],"review_version":1}