Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECK

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CHECK is a dual-pipeline framework that claims to cut medical LLM hallucination rates from 31% to 0.3% on clinical-trial questions and to push GPT-4o's USMLE accuracy to 92.1% via targeted compute escalation.

desk verdict The headline numbers don't survive contact with the methods: the 31% to 0.3% reduction is a context effect, not a CHECK effect, and the classifier's training labels are confounded with context. read the letter →

arxiv 2506.11129 v1 pith:QXFZ6SCT submitted 2025-06-10 cs.CL cs.AI

Carlos Garcia-Fernandez , Luis Felipe , Monique Shotande , Muntasir Zitu , Aakash Tripathi , Ghulam Rasool , Issam El Naqa , Vivek Rudrapatna
show 1 more author
Gilmer Valdes
This is my paper
classification cs.CLcs.AI
keywords hallucinationdetectionmedicallargelanguagemodelsinformation-theoreticfeaturestoken-levelprobabilityclinicaltrialsMedQAUSMLEcomputeescalationHealthBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CHECK is a two-part framework for keeping medical language models honest. It first checks model answers against a curated clinical-trials database using an LLM judge; in parallel, a database-free classifier inspects token-level probability distributions from an ensemble of models and flags outputs whose uncertainty and cross-model disagreement resemble hallucinations. On 1,500 questions drawn from 100 clinical trials, giving Llama3.3-70B-Instruct structured summaries instead of raw titles cut its hallucination rate from 31% to 0.3%, and the classifier alone reached AUCs of 0.95–0.96 on clinical trials, UMLS disorder descriptions, MedQA (USMLE), and HealthBench. The paper also uses the classifier's hallucination probability as a reliability signal: escalating to extra chain-of-thought compute only for the 40% of USMLE questions ranked most likely to be hallucinated pushed GPT-4o to 92.1% accuracy, above the previous state of the art. The aim is a transparent, self-improving safety layer that works even when the underlying model is proprietary and its weights are hidden.

What carries the argument

The central object is the dual-pipeline arbitration: a database-guided fact-checker that labels answers as Fact, Hallucination, Judgment Error, or Coverage Gap by asking an LLM judge to test support and contradiction against a structured clinical-trial database, and a database-free stacking classifier that consumes token-level probability features—entropy, generated-token rank, top-50 token probabilities, and pairwise Kullback–Leibler divergences across a five-model ensemble (Llama3.1-8B/70B-Instruct, Llama3.3-70B-Instruct, Nemotron-70B, DeepSeek-V1)—aggregated to the fifth statistical moment. The classifier carries the argument that hallucination is detectable without a database; the database pipeline carries the argument against contamination; the arbitration rules connect the two and route disagreements to human review.

What would settle it

Take a new set of, say, 1,000 clinical answers, have them labeled independently by multiple human clinicians rather than an LLM judge, run CHECK's database-free classifier on them, and measure its agreement with the human labels; if the AUC falls well below 0.95 or the classifier is no better than a baseline that knows only whether the input was a title or a summary, the central claim is refuted.

Watch

Extended reading notes

Core claim

The paper claims that hallucinations have statistical signatures that survive across models and benchmarks: factual statements produce stable, sharply peaked next-token probability distributions, while confabulations produce high-entropy, high-variance distributions that diverge across independently trained models and across rephrased questions. CHECK operationalizes this by running forced inference on candidate answers through an ensemble of five LLMs, extracting entropy, rank, and pairwise KL-divergence features up to the fifth statistical moment, and training a stacking classifier (Random Forest, Logistic Regression, XGBoost) to separate facts from hallucinations. On its clinical-trial test set the classifier reaches AUC 0.95; it transfers to UMLS disorder paragraphs (0.96), MedQA answer choices (0.95), and HealthBench dialogues (97.7–98.4% accuracy, depending on feature source). The authors further claim that when the database and classifier disagree, escalating to human review turns CHECK into a continuously learning system, and that the classifier's probability output can rank answers by risk well enough to make targeted compute escalation on MedQA beat a prior state-of-the-art system at lower cost.

Load-bearing premise

The whole evaluation depends on Llama3.3-70B-Instruct serving as the LLM judge for both the training labels and the clinical-trial test labels, and the training examples are chosen so that summaries supply most 'fact' labels and titles supply most 'hallucination' labels; if that judge is biased or the classifier is learning the input-context difference rather than factual content, the reported AUCs will not transfer to real clinical hallucinations.

Editorial extensions

If this is right

  • If CHECK's numbers hold, open-source models such as Llama3.3-70B-Instruct can be made reliable enough for clinical question answering, since the hallucination rate drops below commonly cited medication-error thresholds.
  • Hallucination probability from the classifier can act as a confidence score, letting hospitals escalate only the riskiest outputs to human experts or extra compute rather than reviewing everything.
  • Because the classifier works on token distributions of any target model, it can police proprietary models like GPT-4o without access to weights or training data.
  • The framework's continuous feedback loop implies that coverage gaps shrink over time as expert-reviewed cases are fed back into the database and classifier.
  • The 92.1% MedQA result implies that targeted test-time compute, guided by a hallucination-risk ranker, can outperform uniform compute escalation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same variance-based signature may generalize beyond medicine to other high-stakes text domains (legal, financial) where ground truth is verifiable, though the classifier would need retraining on domain-specific data.
  • A testable extension is to check whether the classifier's ranking on MedQA is calibrated well enough to set a fixed escalation threshold in production, or whether thresholds must be tuned per benchmark.
  • Because the training labels come from an LLM judge, the reported AUCs bound the classifier's agreement with that judge, not necessarily with expert truth; independent expert-labeled test sets would be the natural next validation.
  • The atomic-claim results (AUC 0.90 on clinical trials, 0.76 on UMLS) suggest the method's power degrades at finer granularity, so deploying CHECK on sentence-level claims will likely need domain-specialized ensemble members.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents CHECK, a dual-pipeline framework for detecting hallucinations in medical LLM outputs. The first pipeline cross-references model answers against a curated clinical trial database using an LLM judge that performs factual and counterfactual analyses. The second is a "database-free" stacking classifier trained on token-level probability features (entropy, top-token ranks, pairwise KL divergences) extracted from an ensemble of open-source models. The authors report that supplying the model with structured summaries reduces Llama3.3-70B-Instruct hallucination rates from 31% (title context) to 0.3% (summary context), that the classifier achieves AUCs of 0.95–0.96 on clinical trials, UMLS disorders, MedQA, and HealthBench, and that using the classifier's hallucination probabilities to trigger extra chain-of-thought iterations raises GPT-4o's MedQA accuracy to 92.1%, which they call state of the art. The framework is productionized at the BlueScrubs platform, and the clinical database pipeline is open-source.

Significance. If the central claims were supported, CHECK would be a valuable contribution: a model-agnostic, database-free detector of clinical hallucinations and a compute-escalation signal that improves accuracy while limiting test-time cost. The paper ships a public database-curation pipeline and code for classifier training, and the basic idea—that cross-model divergence and token-level uncertainty correlate with factual reliability—is empirically plausible and worth testing. The MedQA analysis in Section 3.6.1, which shows a monotonic relationship between predicted hallucination probability and accuracy, is a useful exploratory finding if validated on a held-out split. However, the current evaluation design does not establish the headline claims: the 31% to 0.3% reduction is a between-context comparison, the classifier's training labels are confounded with input context, and the generalization benchmarks rely on synthetically generated hallucinations. The significance is therefore conditional on a substantially revised validation protocol.

major comments (4)
  1. [Section 3.1, Table 1, Abstract] The headline claim that "CHECK reduced LLama3.3-70B-Instruct hallucination rates from 31% to 0.3%" is a between-context comparison, not a measured effect of CHECK's detection or arbitration pipeline. Table 1 compares three input contexts (title, JSON, summary), and Section 3.1 itself states that "the summary context achieved the highest factual accuracy... a 100-fold reduction compared to the title context." The Discussion then attributes this reduction to CHECK ("CHECK reduces hallucination rates by over 100-fold"). Because the database pipeline and classifier are not part of the context-manipulation experiment, this misattributes a prompt/context effect to the framework, and the abstract's central claim is therefore unsupported as written.
  2. [Sections 3.2, 5.3, 5.4] The classifier's training and test labels are produced by Llama3.3-70B-Instruct acting as an LLM judge, and Section 3.2 explicitly stratifies the training data so that summaries provide most 'fact' labels and titles provide most 'hallucination' labels. This creates a direct confound: a classifier can achieve an AUC near 0.95 by learning the input context source, question template, or judge's stylistic biases rather than factual content. The human validation is far too thin to break this confound: Section 3.3 reviews only 20 selected extreme cases (10 highest and 10 lowest hallucination probabilities), and Section 5.2 reports no agreement statistics for its unspecified "representative subset." Consequently, the AUC 0.95 on the clinical trial test set does not demonstrate that the classifier detects hallucinations independently of context or judge artifacts.
  3. [Sections 3.4, 3.7, S1.3.2] The UMLS and HealthBench generalization evaluations use synthetically generated hallucinations: GPT-4o counterfactual rewrites of factual medical text (Section S1.3.2) and GPT-4o rewrites of its own factual HealthBench responses (Section 3.7). These synthetic examples are constructed with explicit prompts to preserve style and structure, so the high reported accuracies (97.7–98.4%) may reflect detectable artifacts of the rewriting procedure rather than a general ability to detect naturally occurring clinical hallucinations. The HealthBench protocol is especially circular: GPT-4o both generates the answers and provides the single-token 'yes'/'no' self-classifications used as features, so the classifier is effectively trained to predict GPT-4o's own judgments on GPT-4o-generated text.
  4. [Section 3.6.1, Table 2] The compute-escalation result (92.1% on MedQA) depends on two free parameters: the percentile threshold (top 40%) and the number of chain-of-thought iterations (12). The paper does not report any validation procedure that separates the selection of these hyperparameters from the test set, nor does it provide a sensitivity analysis. Without an independent validation split or a pre-specified selection rule, the 5-percentage-point improvement over the GPT-4o CoT baseline may be overoptimistic. Additionally, the comparison to Med-Gemini's 91.1% does not control for compute budgets or model size, so the "state-of-the-art" claim is not established.
minor comments (6)
  1. [Section 2.3] The bulleted list of arbitration rules has inconsistent capitalization and one bullet begins with a lowercase 'if' without a concluding period; this should be cleaned up.
  2. [Throughout] There are numerous typos and inconsistent spellings, including 'LLama3.3-70B-Instruct' (inconsistent capitalization), 'priopretary' (Section 3), 'independet' (Section 5.4), and 'ChatGTP' (Section 3.7.2).
  3. [References] Reference [17] is incomplete: it reads 'Y. Shen et al. [title/details of the reference]. 2022.' and needs a full citation.
  4. [Section 3.3] The text refers to 'Section 2.5, Stage 3' for the arbitration procedure, but the integration and arbitration stage is actually described in Section 2.3; the cross-reference is incorrect.
  5. [Sections 5.1.1 and 5.2] The human-review procedures are described but their results are not reported: Section 5.1.1 defers to the Supplementary Information without providing the scores, and Section 5.2 mentions a 'representative subset' with no counts, agreement statistics, or a description of how the subset was selected.
  6. [Abstract and Section 2.1] The phrase 'grounded in information theory' overstates the contribution: the paper proposes information-theoretic features, but it does not present a theoretical derivation or bound linking these features to hallucination probability.

Circularity Check

2 steps flagged · score 4.0 of 10

Headline '31% to 0.3%' is a context comparison, and the database-free classifier is trained on LLM-judge labels confounded with context source; MedQA supplies partial independent support.

  1. fitted input called prediction [Section 3.2 and Table 1; see also Abstract]
    "To create clear and distinct training signals for the classifier, we strategically selected data from model responses produced using high-quality context: summaries (which predominantly yielded factual content) and titles (which were identified as significant sources of hallucination; see Table 1)."

    Table 1 shows title-context responses are 31% hallucinations and summary-context responses 0.3% hallucinations. Thus, by the paper's own selection rule, the training labels are nearly collinear with the input context source. The classifier is then presented as 'database-free hallucination detection' with AUC 0.95, but on this training set a model only needs to separate title-context from summary-context responses to reproduce the label distribution; it need not measure factual content. This is a fitted input (context source or judge style) relabeled as a hallucination prediction. The later MedQA evaluation, with externally labeled correct/incorrect options, provides independent evidence that partially breaks this confound.

  2. self definitional [Sections 5.3 and 5.4; see also Sections 2.1 and 3.1]
    "we collected labeled data by systematically evaluating each LLM-generated response against the curated clinical trials database. When the LLM judge performed its factual/counterfactual analysis, every evaluated answer was assigned one of four outcomes (factual, hallucination, coverage gap or error). ... across an ensemble of five distinct language models (LLama3.1-8B-Instruct, LLama3.1-70B-Instruct, LLama3.3-70B-Instruct, Nemotron-70B, DeepSeek-V1)."

    The judge that creates the fact/hallucination labels is Llama3.3-70B-Instruct, as stated in Section 3.1: 'we employed LLama3.3-70B-Instruct as an independent judge.' That exact model is a member of the ensemble from which the classifier's token-level features are extracted. Hence, on the clinical-trial and UMLS data, the 'database-free' classifier can learn to imitate the judge's verdict from the judge model's own next-token probabilities, making the detected 'hallucination' a self-referential prediction of the LLM judge rather than an independently established clinical fact. The MedQA experiment, where labels are correct/incorrect exam answers, supplies an external anchor, so this circularity is partial.

full rationale

Two load-bearing reductions or near-reductions are present. First, the flagship hallucination reduction is not a measured CHECK intervention: Table 1 compares title, JSON, and summary input contexts, and the drop from 31% to 0.3% is the difference between conditions, not an effect of CHECK's classifier or arbitration. Second, the database-free classifier is fit to labels produced by Llama3.3-70B-Instruct as an LLM judge, with training examples deliberately chosen so that summaries supply most factual labels and titles most hallucination labels (Section 3.2). With Table 1's context-label correlation, a classifier can reach AUC 0.95 by detecting context source rather than factual error. Because Llama3.3-70B-Instruct is also one of the ensemble models supplying token-probability features (Section 5.4), the 'database-free' signal can encode the judge's own tendencies. These issues are not fatal to the whole paper: MedQA supplies externally defined correct/incorrect labels, and the classifier maintains AUC 0.95 there, while the 92.1% compute-escalation result is benchmarked externally. Section 5.2's human review of 'a representative subset' reports no agreement statistics, and Section 3.3 reviews only 20 extreme cases, which limits external grounding without being circular by itself. Overall, the central claim has independent content through MedQA, so the circularity is partial rather than total.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results rest on four unproven domain assumptions: the stability hypothesis that motivates the classifier, the validity of the LLM-judge labels, the fidelity of LLM-generated clinical summaries, and the representativeness of GPT-4o-generated synthetic counterfactuals. No standard-math axioms are needed. The free parameters are the hand-chosen compute-escalation threshold and iteration count, plus classifier hyperparameters.

free parameters (3)
  • compute escalation threshold (top 40%) = 40%
    Section 3.6.1: the authors apply 12x CoT only to the 40% of MedQA questions with the highest hallucination probability; this hand-chosen cutoff directly determines the reported 92.06% result.
  • CoT iterations (12x) = 12
    Section 3.6.1: the number of chain-of-thought passes is selected without a reported scaling curve; the state-of-the-art claim depends on this specific value.
  • classifier hyperparameters = RF 1000 trees, XGB 5000 estimators, etc.
    Section 5.4.1: hyperparameters (e.g., max_depth=None, min_samples_leaf=1, random_state=42, colsample_bytree=0.8) are fixed heuristically with no tuning or sensitivity analysis; they affect the reported AUCs.
assumptions (4)
  • domain assumption Factual statements yield stable, low-entropy token distributions across models and rephrasings; hallucinations yield high-entropy, high-variance distributions
    Section 2.2 states this 'principle that truth is stable, but hallucinations arise from high-entropy and high-variance distributions'. It is the basis for the classifier features and is not proven; it may conflate answer style or context with factuality.
  • domain assumption The LLM judge's supported/contradicted labels are valid ground truth for hallucination
    Section 2.1 defines the labels and Section 5.2 describes a human review of a 'representative subset' with only 3 reviewers and no reported agreement numbers. Training and test labels inherit any judge bias.
  • domain assumption LLM-generated structured summaries faithfully represent the source clinical trials
    Section 5.1 describes summaries generated by Llama3.1-70B from ClinicalTrials.gov JSON. A human evaluation is mentioned in Section 5.1.1, but no scores are reported, so summary fidelity is assumed.
  • domain assumption Synthetic counterfactuals created by GPT-4o are representative of real LLM hallucinations
    Sections 3.7 and 5.5 use GPT-4o rewrites of factual text into 'false but plausible' versions for UMLS and HealthBench. There is no evidence that these mimic naturally occurring hallucinations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECK." pith.science (2026). https://pith.science/paper/QXFZ6SCT

@misc{pith2026250611129,
  author       = {Pith},
  title        = {Pith review of: Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECK},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXFZ6SCT}},
  note         = {Machine review of arXiv:2506.11129}
}
read the original abstract

Large language models (LLMs) show promise in healthcare, but hallucinations remain a major barrier to clinical use. We present CHECK, a continuous-learning framework that integrates structured clinical databases with a classifier grounded in information theory to detect both factual and reasoning-based hallucinations. Evaluated on 1500 questions from 100 pivotal clinical trials, CHECK reduced LLama3.3-70B-Instruct hallucination rates from 31% to 0.3% - making an open source model state of the art. Its classifier generalized across medical benchmarks, achieving AUCs of 0.95-0.96, including on the MedQA (USMLE) benchmark and HealthBench realistic multi-turn medical questioning. By leveraging hallucination probabilities to guide GPT-4o's refinement and judiciously escalate compute, CHECK boosted its USMLE passing rate by 5 percentage points, achieving a state-of-the-art 92.1%. By suppressing hallucinations below accepted clinical error thresholds, CHECK offers a scalable foundation for safe LLM deployment in medicine and other high-stakes domains.

Figures

Figures reproduced from arXiv: 2506.11129 by the authors.

Figure 1
Figure 1. (a) CHECK - Continuous-learning fact-checker framework operationalized at The BlueScrubs platform. (b) Database-driven effective approach against data contamination and adversarial attacks. (c) Database-Free classification model based on probability distributions over next-token predictions across an ensemble of generative models. • Not Contradicted: The LLM judge cannot deduce the negation of the statement from any… view at source ↗
Figure 2
Figure 2. Factual analysis performed by an LLM judge on model-generated answe [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Distribution of LLM Answer Labels Across 15 Question Categories, by C [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Adaptive intervention framework based on hallucination probabil [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Model Performance across Hallucination Probability Percentil [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Overall Model Performance across Confidence Percentiles. Ac [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Feature extraction pipeline for factuality/hallucination det [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptation

    cs.CL 2025-12 conditional novelty 4.0 of 10

    A deterministic, proposition-level fact-checker that compares clinical summaries against electronic health records via (entity, attribute, value, time) claims and hard-coded logical checks reports 0.8904 precision and...

Reference graph

Works this paper leans on

31 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep learning-enabled medi cal computer vision

    Andre Esteva, Katherine Chou, Serena Yeung, Nikhil Naik, Ali Madani, Al i Mottaghi, Yun Liu, Eric Topol, Jeff Dean, and Richard Socher. Deep learning-enabled medi cal computer vision. NPJ digital medicine , 4(1):5, 2021

    work page 2021

  2. [2]

    Role of chat gpt in public health

    Som S Biswas. Role of chat gpt in public health. Annals of biomedical engineering , 51(5):868–869, 2023

    work page 2023

  3. [3]

    Cognitive mirage : A review of hallucinations in large language models

    Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage : A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794 , 2023

    arXiv 2023

  4. [4]

    Survey of hallucination in natural language gen eration

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Et suko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language gen eration. ACM computing surveys , 55(12):1–38, 2023

    work page 2023

  5. [5]

    Detec ting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detec ting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024

    work page 2024

  6. [6]

    Early disc ontinuation and non- adherence to adjuvant hormonal therapy are associated with increased mor tality in women with breast cancer

    Dawn L Hershman, Theresa Shao, Lawrence H Kushi, Donna Buono, Wei Y ann Tsai, Louis Fehren- bacher, Marilyn Kwan, Scarlett Lin Gomez, and Alfred I Neugut. Early disc ontinuation and non- adherence to adjuvant hormonal therapy are associated with increased mor tality in women with breast cancer. Breast cancer research and treatment , 126:529–537, 2011

    work page 2011

  7. [7]

    Hallucination rates and reference accuracy of chatgpt and bard for systematic reviews: compar ative analysis

    Mika¨ el Chelli, Jules Descamps, Vincent Lavou´ e, Christophe Trojani, Michel Azar, Marcel Deckert, Jean-Luc Raynier, Gilles Clowez, Pascal Boileau, and Caroline Ruetsc h-Chelli. Hallucination rates and reference accuracy of chatgpt and bard for systematic reviews: compar ative analysis. Journal of medical Internet research , 26:e53164, 2024

    work page 2024

  8. [8]

    E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. O n the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, F AccT, 2021

    work page 2021

Show all 31 references
  1. [9]

    McCloskey and N

    M. McCloskey and N. J. Cohen. Catastrophic interference in connect ionist networks: The sequential learning problem. In The Psychology of Learning and Motivation , volume 24, pages 109–165. 1989

    1989

  2. [10]

    Overcom- ing catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sci...

    2017

  3. [11]

    Medical large language models are vulnerable to data-poisoning attacks

    Daniel Alexander Alber, Zihao Yang, Anton Alyakin, Eunice Yang, Sumedha Rai, Aly A Valliani, Jeff Zhang, Gabriel R Rosenbaum, Ashley K Amend-Thomas, David B Kurland, et al. Medical large language models are vulnerable to data-poisoning attacks. Nature Medicine, pages 1–9, 2025

    2025

  4. [12]

    Guu et al

    K. Guu et al. Realm: Retrieval-augmented language model pre-trainin g. In Proceedings of the 37th International Conference on Machine Learning , ICML, 2020

    2020

  5. [13]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vlad imir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨ aschel, etal. Retrieval-augmented generation for knowledge-intensive nlp tasks. volume 33, pages 9459–9474, 2020

    2020

  6. [14]

    Holtzman, J

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neu ral text degeneration. In International Conference on Learning Representations , ICLR, 2020

    2020

  7. [15]

    Zellers, A

    R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, an d Y. Choi. Defending against neural fake news. In Advances in Neural Information Processing Systems , NeurIPS, 2019

    2019

  8. [16]

    Langu age models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, E than Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Langu age models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022. 15

    2022 arXiv

  9. [17]

    Shen et al

    Y. Shen et al. [title/details of the reference]. 2022

    2022

  10. [18]

    Seven failure points when engineering a retrieval augmented gener ation system, 2024

    Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brann elly, and Mohamed Abdelrazek. Seven failure points when engineering a retrieval augmented gener ation system, 2024

    2024

  11. [19]

    UMLS Knowledge Sources, 2024

    National Library of Medicine. UMLS Knowledge Sources, 2024. Available from: http://www.nlm. nih.gov/research/umls/licensedcontent/umlsknowledgesources.html

    2024

  12. [20]

    What disease does this patient have? a large-scale open domain question answ ering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Pet er Szolovits. What disease does this patient have? a large-scale open domain question answ ering dataset from medical exams. Applied Sciences, 11(14), 2021

    2021

  13. [21]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaqui n Qui˜ nonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beu tel, Johannes Hei- decke, and Karan Singhal. Healthbench: An evaluation for ai systems and human health, 20...

    2025

  14. [22]

    T he llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abh ishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. T he llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  15. [23]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Mu ennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skow ron, Lintang Su...

    2024

  16. [24]

    Deepseek-r1: Incentivizing re asoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing re asoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  17. [25]

    Capabilities of gemin i models in medicine, 2024

    Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Elle ry Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemin i models in medicine, 2024

    2024

  18. [26]

    Scaling LLM test-time com- pute optimally can be more effective than scaling parameters for reasonin g

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling parameters for reasonin g. In The Thirteenth International Conference on Learning Representations , 2025

    2025

  19. [27]

    simple-evals: Healthbench evaluation framework, 2025

    OpenAI. simple-evals: Healthbench evaluation framework, 2025. GitHub r epository

    2025

  20. [28]

    MacDowell, A

    P. MacDowell, A. Cabri, and M. Davis. Medication administration err ors. Psnet internet, Agency for Healthcare Research and Quality, US Department of Health and Human Servic es, Rockville, MD, 2021

    2021

  21. [29]

    Random forests

    Leo Breiman. Random forests. Machine learning , 45:5–32, 2001

    2001

  22. [30]

    Applied logistic regression

    David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. Applied logistic regression . John Wiley & Sons, 2013

    2013

  23. [31]

    Xgboost: A scalable tree boosting sy stem

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting sy stem. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and da ta mining , pages 785–794, 2016. 16

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.