Pith. sign in

REVIEW 3 major objections 4 minor 57 references

A Knowledge-Injection Framework for Zero-Shot Adaptation of LLMs to Delirium Prediction

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a clinically meaningful, one-time knowledge report to the prompt improves zero-shot ICU delirium prediction, with an 8.57-point AUROC gain for the 8B model.

desk verdict A clean, well-controlled prompt-engineering result for zero-shot delirium prediction, but the paper overinterprets the active ingredient: the winning report embeds an explicit scoring rubric, and the controls don't rule that out. read the letter →

arxiv 2607.20453 v1 pith:MLFKY2TH submitted 2026-05-14 cs.CL cs.AIcs.LG

Jessica Sena , Shesadree Priyadarshani , Miguel Contreras , Bharat Gandhi , Scott Siegel , Subhash Nerella , Parisa Rashidi This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords deliriumpredictionzero-shotlearningknowledgeinjectionlargelanguagemodelsclinicaldecisionsupportintensivecareunitpromptengineeringMIMIC-IV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small, locally deployable LLM can be made substantially better at zero-shot ICU delirium prediction by injecting a one-time, task-level clinical knowledge report into the prompt at inference time — no fine-tuning, no retrieval. On 3,160 MIMIC-IV ICU stays, the framework lifts LLaMA 3.1 8B AUROC by 8.57 percentage points over patient-data-only prompting, and gives a smaller but significant gain to LLaMA 3.3 70B. The gain depends on the report's semantic content: random length-matched reports do not help and can hurt. The paper also claims that how knowledge is structured matters — a concise qualitative rubric beats a detailed threshold-based one — and that SHAP attribution shows the model actually uses the injected knowledge. A sympathetic reader would care because it suggests a pragmatic, privacy-preserving way to close part of the gap between open-weight models and frontier models in structured clinical prediction without retraining.

What carries the argument

The knowledge report: a task-level text generated once by a frontier model, describing delirium risk factors across four domains (baseline vulnerability, acute physiologic instability, neurocognitive state, medication-related risk proxies) with a stepwise probability adjustment scheme. It is fused with a deterministic EHR summary into a fixed prompt template that asks the model for a 3-step chain-of-thought and a 0-100 probability. The comparison between a concise qualitative report (v1) and a threshold-heavy report (v2), plus length-matched random reports, isolates semantic content as the driver.

What would settle it

Evaluate LLaMA 3.1 8B with the v1 report text edited to remove the probability-mapping rubric and adjustment ranges, keeping only the clinical risk-factor statements. If AUROC falls back near the 53.20% baseline, the reported gain is caused by the embedded scoring rule, not by the clinical knowledge content.

Watch

Extended reading notes

Core claim

The central claim is that inference-time knowledge injection — appending a short, model-authored clinical knowledge report to a deterministic natural-language summary of structured EHR features — improves zero-shot delirium prediction in open-weight LLMs, with the largest gain for the smaller model. The winning report (v1) is a qualitative risk-assessment scaffold with four evidence domains and a probability-mapping rubric; a length-matched random report does not replicate the gain. The paper interprets this as evidence that clinically meaningful structure, not added prompt text, is the active ingredient, and that knowledge injection acts as a capacity-compensation mechanism for smaller mode

Load-bearing premise

The load-bearing premise is that the v1 knowledge report contains valid, generalizable clinical knowledge, but it was generated by a frontier model without expert review and encodes an explicit probability-scoring rubric; if the gain comes from that rubric rather than from accurate clinical facts, the central interpretation narrows to a specific AI-authored scoring guide.

Editorial extensions

If this is right

  • Smaller open-weight models can recover a large fraction of the zero-shot gap to frontier models on structured clinical prediction with no retraining.
  • Clinicians deploying 8B-scale local models could use a one-time knowledge prompt rather than labeled-data fine-tuning.
  • Report format is at least as important as report content; concise qualitative guidance outperforms detailed numerical thresholds.
  • Random context degrades the 8B model, so prompt hygiene matters in clinical LLM pipelines.
  • Knowledge injection does not add inference-time overhead; for 70B it reduced generation time in this setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the v1 gain actually comes from its embedded probability-mapping rubric (start at 15%, add 5-40% per domain) rather than from accurate clinical facts, then a much shorter calibration-style prompt might reproduce part of the gain — a cheap testable extension.
  • The framework's success on delirium suggests the same recipe may transfer to other structured prediction tasks with published clinical risk factors, e.g., sepsis or pressure injury, where a clinician-authored report could be tested head-to-head against the model-generated one.
  • The report was generated without expert validation; a human-edited version may either improve performance further or reveal that some 'clinical knowledge' in v1 is spurious — an important unstated edge.
  • The 8B baseline at 53.20% AUROC is close to random on a balanced cohort, so the absolute gain leaves the model still below clinically usable performance; the practical claim is about gap-narrowing, not deployment readiness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes a zero-shot knowledge-injection framework for ICU delirium prediction. A structured EHR summary is augmented at inference time with a task-level 'external clinical knowledge report' generated once by GPT-5.2; LLaMA 3.1 8B and LLaMA 3.3 70B are prompted to produce a 0-100 probability. On a balanced MIMIC-IV cohort of 3,160 ICU stays, the v1 real report improves AUROC by 8.57 points (8B) and 1.99 points (70B) over the data-only baseline, while length-matched random reports do not improve performance. The v2 report yields smaller, largely non-significant gains. A GPT-5.2 data-only reference achieves AUROC 68.86%, and the authors report that knowledge injection narrows the gap. SHAP-based attribution is used to claim the injected report is actively used.

Significance. Strengths: the evaluation is a controlled prompting comparison with no fitted parameters, the random-report control is well-designed for prompt length, both report variants are disclosed, and bootstrapped significance is reported. If the effect is driven by clinically valid content, the framework is a practical, privacy-preserving way to adapt small local LLMs to structured clinical prediction without fine-tuning or retrieval. The central interpretation, however, is currently too broad: the winning v1 report contains an explicit probability-mapping rubric and is generated by the same GPT-5.2 model used as the frontier reference, so the experiments establish a more specific claim—that a particular AI-authored rubric helps—not the general claim that clinically meaningful external knowledge is the active ingredient. With appropriate ablations and qualified claims, the contribution would be useful to the clinical NLP community.

major comments (3)
  1. [§II-E, Appendix A, Appendix C; Table II] The v1 report's Step 6 and probability-mapping scale ('Start at 15%', '+5–10%/+10–20%/+20–40%') are an explicit scoring algorithm, and the prompt (Appendix C, knowledge_instructions) tells the model to base its prediction ONLY on factors described in the report. The random lorem-ipsum control matches token length but does not control for the presence of such a rubric, so the +8.57 AUROC gain may be attributable to the rubric heuristic rather than to clinically meaningful knowledge. Without a human-validated clinical report lacking the numerical mapping, or a rubric-only placebo control, the central claim in the abstract is not established.
  2. [Abstract; Table II; §III-C] The headline gain is v1-specific. The v2 real report gives +1.53 for 8B (p=0.163, prop=0.35) and +0.55 for 70B (p=0.561), and the 70B v1 gain is only p=0.023 with 70% of bootstraps significant. Since two real variants were tested and the larger effect is featured, the manuscript should either correct for the variant selection or explicitly qualify the claim as 'v1 report variant' throughout, including the title/abstract.
  3. [§II-E, §III-D, §IV-D] The external knowledge report is generated by GPT-5.2, the same model used as the frontier reference. The reported 'gap reduction' therefore partly measures distillation from the reference model, and even the framework's privacy narrative requires generating the report via a proprietary API (one time). The authors should discuss this dependence explicitly and, if possible, compare a report authored or validated by clinicians to disentangle 'external clinical knowledge' from 'frontier-model output.' Section IV-H mentions prompt sensitivity but not this self-referential source.
minor comments (4)
  1. [§II.G vs Table IV] The methods state a Mann-Whitney U test for inference-time comparisons, but Table IV and §III.E report Wilcoxon signed-rank tests. Align the description with the paired design actually used.
  2. [§III.F / Fig. 4] The SHAP methodology is not described: which explainer was used, how token-level SHAP values were computed and aggregated over 80 samples, and whether the patient-summary section and knowledge report were tokenized jointly. Without this, the attribution results are difficult to interpret.
  3. [Table III] GPT-5.2 is reported from a single full-cohort evaluation while LLaMA results are means over 20 bootstraps; state whether a CI for GPT-5.2 was computed, since the gap comparison relies on these point estimates.
  4. [Fig. 3 caption] The caption 'AUROC curve' is imprecise; the figure plots ROC curves. The legend should match the terminology used in the text.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the evaluation is a controlled, parameter-free prompting comparison, with only a minor non-load-bearing self-citation.

full rationale

The paper's central comparison is a controlled prompting experiment: the same deterministic EHR summary is presented with no report, with a GPT-5.2-generated clinical report, or with a token-length-matched random report, and AUROC is measured on a fixed balanced cohort. No parameter is fitted to the evaluation labels, and the knowledge report is a fixed, task-level text reused across all patient inferences. The +8.57 AUROC gain for LLaMA 8B is therefore an empirical outcome of the intervention, not an identity forced by construction. The only overlapping-author citation is [45] (Contreras et al.), which supplies the 81-feature set and the structured-text EHR representation; both the report and no-report arms share this input, so the measured delta does not reduce to that citation. The GPT-5.2 authorship of the report, combined with GPT-5.2 also serving as the frontier reference, is a validity/distillation concern rather than a circular reduction: the report is not a fitted output and the LLaMA models do not inherit GPT-5.2's AUROC by definition. Similarly, the v1 report's embedded 'Start at 15%' probability-mapping rubric (Appendix A) means the random-report control does not isolate clinical facts from the scoring heuristic; this is an acknowledged confound (Section IV-H: 'systematic evaluation of prompt sensitivity remains an open area'), not a self-definitional or fitted-input circularity. Overall, the derivation chain is self-contained as an empirical evaluation, and no step reduces to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper contains no fitted model parameters. Its central claim rests on the GPT-generated knowledge report being valid clinical knowledge, the min-max EHR summary being sufficient, and raw probability scores being meaningful. The v1 report's explicit probability mapping is a hand-chosen prompt rubric, not an independently measured clinical constant.

free parameters (1)
  • Report v1 probability-mapping constants = baseline 15%; adjustment ranges +5-40% / -5-30%
    These hand-authored (GPT-5.2-generated) constants are embedded in the winning knowledge report and directly shape the model's output probability. They were not fitted to MIMIC data but are ad hoc prompt content that may drive the AUROC gain.
assumptions (4)
  • domain assumption The GPT-5.2-generated knowledge report is clinically accurate enough to improve predictions.
    No human expert validation is reported; if the report contained hallucinations, the observed gains could reflect a scoring rubric rather than valid clinical knowledge. See Section II-E.
  • domain assumption The min/max structured EHR summary preserves sufficient predictive information.
    Time-varying features are collapsed to min-max ranges, discarding trends; the paper relies on [45] for this representation. See Section II-D.
  • domain assumption Raw LLM probability scores are usable as risk predictions without calibration.
    AUROC is rank-based so calibration is not required for the primary metric, but the reported accuracy/precision/F1 at threshold 50 assume the 0-100 score is interpretable as a probability. See Section II-F.
  • domain assumption Token-level SHAP values reflect how the model actually uses prompt sections.
    No SHAP citation or implementation details are given; attributions for generative LLM outputs are not standard. See Section III-F.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Knowledge-Injection Framework for Zero-Shot Adaptation of LLMs to Delirium Prediction." pith.science (2026). https://pith.science/paper/MLFKY2TH

@misc{pith2026260720453,
  author       = {Pith},
  title        = {Pith review of: A Knowledge-Injection Framework for Zero-Shot Adaptation of LLMs to Delirium Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MLFKY2TH}},
  note         = {Machine review of arXiv:2607.20453}
}
read the original abstract

Large language models show promise for clinical prediction, but zero-shot performance on specialized tasks is limited by incomplete domain knowledge, especially for smaller locally deployable models. We present a lightweight knowledge-injection framework for zero-shot ICU delirium prediction that augments a deterministic natural-language summary of structured electronic health record data with an external clinical knowledge report at inference time, without fine-tuning or retrieval. We evaluate LLaMA 3.1 8B and LLaMA 3.3 70B on 3,160 ICU admissions from the MIMIC IV dataset. Adding a clinically meaningful external knowledge report improves AUROC by 8.57 percentage points for the 8B model and 1.99 percentage points for the 70B model compared to no external knowledge. Relative to a GPT-5.2 frontier-model reference without external knowledge report (AUROC 68.86%), knowledge injection reduces the performance gap from 15.66 to 7.09 AUROC points for LLaMA 8B and from 5.30 to 3.31 AUROC points for LLaMA 70B. Random control reports do not improve performance and often degrade it, indicating that gains depend on clinically meaningful content rather than added prompt length alone. SHAP-based attribution further confirms that the injected knowledge is actively used during prediction. These findings suggest that inference-time knowledge injection can narrow the gap between locally deployable open-weight models and frontier closed models while preserving a practical, privacy-preserving workflow for resource-constrained clinical settings.

Figures

Figures reproduced from arXiv: 2607.20453 by the authors.

Figure 2
Figure 2. After applying these exclusion criteria, the full cohort com￾prised 22,213 ICU admissions. The dataset exhibited substan￾tial class imbalance, with delirium-positive cases representing [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Clinical workflow for delirium risk assessment using an LLM. Patient-specific information from the EHR is summarized into a structured report and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Inclusion and exclusion criteria for MIMIC IV dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: AUROC curve under different report conditions. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: Mean absolute SHAP value per word for the knowledge report and patient data sections across v1 and v2 report versions ( [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 5 linked inside Pith

  1. [1]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  2. [2]

    Large language models in medicine,

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,”Nature medicine, vol. 29, no. 8, pp. 1930–1940, 2023

    1930

  3. [3]

    Large language models in medicine: the potentials and pitfalls: a narrative review,

    J. A. Omiye, H. Gui, S. J. Rezaei, J. Zou, and R. Daneshjou, “Large language models in medicine: the potentials and pitfalls: a narrative review,”Annals of internal medicine, vol. 177, no. 2, pp. 210–220, 2024

    2024

  4. [4]

    Roles and potential of large language models in healthcare: a comprehensive review,

    C. Lin and C.-F. Kuo, “Roles and potential of large language models in healthcare: a comprehensive review,”Biomedical Journal, p. 100868, 2025

    2025

  5. [5]

    Large language models in healthcare and medical applications: a review,

    S. Maity and M. J. Saikia, “Large language models in healthcare and medical applications: a review,”Bioengineering, vol. 12, no. 6, p. 631, 2025

    2025

  6. [6]

    Testing and evaluation of health care applications of large language models: a systematic review,

    S. Bedi, Y . Liu, L. Orr-Ewing, D. Dash, S. Koyejo, A. Callahan, J. A. Fries, M. Wornow, A. Swaminathan, L. S. Lehmannet al., “Testing and evaluation of health care applications of large language models: a systematic review,”Jama, vol. 333, no. 4, pp. 319–328, 2025

    2025

  7. [7]

    Revolutionizing health care: the transformative impact of large language models in medicine,

    K. Zhang, X. Meng, X. Yan, J. Ji, J. Liu, H. Xu, H. Zhang, D. Liu, J. Wang, X. Wanget al., “Revolutionizing health care: the transformative impact of large language models in medicine,”Journal of Medical Internet Research, vol. 27, p. e59069, 2025

    2025

  8. [8]

    Health system-scale language models are all-purpose prediction engines,

    L. Y . Jiang, X. C. Liu, N. P. Nejatian, M. Nasir-Moin, D. Wang, A. Abidin, K. Eaton, H. A. Riina, I. Laufer, P. Punjabiet al., “Health system-scale language models are all-purpose prediction engines,”Na- ture, vol. 619, no. 7969, pp. 357–362, 2023

    2023

Show all 57 references
  1. [9]

    Characterizing research domain criteria symptoms among psychiatric inpatients using large language models,

    T. H. McCoy and R. H. Perlis, “Characterizing research domain criteria symptoms among psychiatric inpatients using large language models,” Journal of Mood & Anxiety Disorders, vol. 8, p. 100079, 2024

    2024

  2. [10]

    Hr-bgcn: Predicting readmission for heart failure from electronic health records,

    H. Ma, D. Li, J. Zhao, W. Li, J. Fu, and C. Li, “Hr-bgcn: Predicting readmission for heart failure from electronic health records,”Artificial Intelligence in Medicine, vol. 150, p. 102829, 2024

    2024

  3. [11]

    A large language model approach to identifying preoperative frailty among older adults from clinical notes,

    Y . Q. Zhou, O. Litake, M. N. Meineke, J. L. Tully, N. Xu, W. Abdou, and R. A. Gabriel, “A large language model approach to identifying preoperative frailty among older adults from clinical notes,”Journal of the American Geriatrics Society, vol. 73, no. 8, pp. 2422–2430, 2025

    2025

  4. [12]

    Development and prospective implementation of a large language model based system for early sepsis prediction,

    S. P. Shashikumar, S. Mohammadi, R. Krishnamoorthy, A. Patel, G. Wardi, J. C. Ahn, K. Singh, E. Aronoff-Spencer, and S. Nemati, “Development and prospective implementation of a large language model based system for early sepsis prediction,”npj Digital Medicine, vol. 8, no. 1, ...

    2025

  5. [13]

    Trajectory-ordered objectives for self-supervised representation learn- ing of temporal healthcare data using transformers: Model development and evaluation study,

    A. Amirahmadi, F. Etminani, J. Björk, O. Melander, and M. Ohlsson, “Trajectory-ordered objectives for self-supervised representation learn- ing of temporal healthcare data using transformers: Model development and evaluation study,”JMIR medical informatics, vol. 13, no. 1, p. ...

    2025

  6. [14]

    First steps toward building natural history of diseases computationally: Lessons learned from the noonan syndrome use case,

    T. Groza, W. Rayabsri, D. Gration, H. Hariram, S. S. Jamuar, and G. Baynam, “First steps toward building natural history of diseases computationally: Lessons learned from the noonan syndrome use case,” The American Journal of Human Genetics, vol. 112, no. 5, pp. 1158– 1172, 2025

    2025

  7. [15]

    Language-model- based patient embedding using electronic health records facilitates phenotyping, disease forecasting, and progression analysis,

    S. Xian, M. E. Grabowska, I. J. Kullo, Y . Luo, J. W. Smoller, W.- Q. Wei, G. Jarvik, S. Mooney, and D. Crosslin, “Language-model- based patient embedding using electronic health records facilitates phenotyping, disease forecasting, and progression analysis,”Research Square, p...

    2024

  8. [16]

    Zero-shot inter- pretable phenotyping of postpartum hemorrhage using large language models,

    E. Alsentzer, M. J. Rasmussen, R. Fontoura, A. L. Cull, B. Beaulieu- Jones, K. J. Gray, D. W. Bates, and V . P. Kovacheva, “Zero-shot inter- pretable phenotyping of postpartum hemorrhage using large language models,”NPJ digital medicine, vol. 6, no. 1, p. 212, 2023

    2023

  9. [17]

    Matching patients to clinical trials with large language models,

    Q. Jin, Z. Wang, C. S. Floudas, F. Chen, C. Gong, D. Bracken-Clarke, E. Xue, Y . Yang, J. Sun, and Z. Lu, “Matching patients to clinical trials with large language models,”Nature communications, vol. 15, no. 1, p. 9074, 2024

    2024

  10. [18]

    How chatgpt works: a mini review,

    G. Briganti, “How chatgpt works: a mini review,”European Archives of Oto-Rhino-Laryngology, vol. 281, no. 3, pp. 1565–1569, 2024

    2024

  11. [19]

    Chatent: augmented large language model for expert knowledge retrieval in otolaryngology– head and neck surgery,

    C. Long, D. Subburam, K. Lowe, A. Dos Santos, J. Zhang, S. Hwang, N. Saduka, Y . Horev, T. Su, D. W. Côtéet al., “Chatent: augmented large language model for expert knowledge retrieval in otolaryngology– head and neck surgery,”Otolaryngology–Head and Neck Surgery, vol. 171, no...

    2024

  12. [20]

    Deepseek in healthcare: revealing opportu- nities and steering challenges of a new open-source artificial intelligence frontier,

    A. Temsah, K. Alhasan, I. Altamimi, A. Jamal, A. Al-Eyadhy, K. H. Malki, and M.-H. Temsah, “Deepseek in healthcare: revealing opportu- nities and steering challenges of a new open-source artificial intelligence frontier,”Cureus, vol. 17, no. 2, p. e79221, 2025

    2025

  13. [21]

    From text to data: Open-source large language models in extracting cancer related medical attributes from german pathology reports,

    S. Bartels and J. Carus, “From text to data: Open-source large language models in extracting cancer related medical attributes from german pathology reports,”International Journal of Medical Informatics, vol. 203, p. 106022, 2025

    2025

  14. [22]

    Multimodal fine- tuning of clinical language models for predicting covid-19 outcomes,

    A. Henriksson, Y . Pawar, P. Hedberg, and P. Nauclér, “Multimodal fine- tuning of clinical language models for predicting covid-19 outcomes,” Artificial Intelligence in Medicine, vol. 146, p. 102695, 2023

    2023

  15. [23]

    Fine- tuning large language models for specialized use cases,

    D. Anisuzzaman, J. G. Malins, P. A. Friedman, and Z. I. Attia, “Fine- tuning large language models for specialized use cases,”Mayo Clinic Proceedings: Digital Health, vol. 3, no. 1, p. 100184, 2025

    2025

  16. [24]

    A closer look at the limitations of instruction tuning,

    S. Ghosh, C. K. R. Evuru, S. Kumar, D. Aneja, Z. Jin, R. Duraiswami, D. Manochaet al., “A closer look at the limitations of instruction tuning,”arXiv preprint arXiv:2402.05119, 2024

    2024 arXiv

  17. [25]

    Novel transfer learning approach for medical imaging with limited labeled data,

    L. Alzubaidi, M. Al-Amidie, A. Al-Asadi, A. J. Humaidi, O. Al- Shamma, M. A. Fadhel, J. Zhang, J. Santamaría, and Y . Duan, “Novel transfer learning approach for medical imaging with limited labeled data,”Cancers, vol. 13, no. 7, p. 1590, 2021

    2021

  18. [26]

    A systematic review of few-shot learning in medical imaging,

    E. Pachetti and S. Colantonio, “A systematic review of few-shot learning in medical imaging,”Artificial intelligence in medicine, vol. 156, p. 102949, 2024

    2024

  19. [27]

    A unified framework on generaliz- ability of clinical prediction models,

    B. Wan, B. Caffo, and S. S. Vedula, “A unified framework on generaliz- ability of clinical prediction models,”Frontiers in Artificial Intelligence, vol. 5, p. 872720, 2022

    2022

  20. [28]

    Multi-site cross-organ calibrated deep learning (muscld): Automated diagnosis of non-melanoma skin cancer,

    Y . Zhou, C. Koyuncu, C. Lu, R. Grobholz, I. Katz, A. Madabhushi, and A. Janowczyk, “Multi-site cross-organ calibrated deep learning (muscld): Automated diagnosis of non-melanoma skin cancer,”Medical image analysis, vol. 84, p. 102702, 2023

    2023

  21. [29]

    Transfer learning enables predictions in network biology,

    C. V . Theodoris, L. Xiao, A. Chopra, M. D. Chaffin, Z. R. Al Sayed, M. C. Hill, H. Mantineo, E. M. Brydon, Z. Zeng, X. S. Liuet al., “Transfer learning enables predictions in network biology,”Nature, vol. 618, no. 7965, pp. 616–624, 2023

    2023

  22. [30]

    Large language models leverage external knowledge to extend clinical insight beyond language boundaries,

    J. Wu, X. Wu, Z. Qiu, M. Li, S. Lin, Y . Zhang, Y . Zheng, C. Yuan, and J. Yang, “Large language models leverage external knowledge to extend clinical insight beyond language boundaries,”Journal of the American Medical Informatics Association, vol. 31, no. 9, pp. 2054–2064, 2024

  23. [31]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschelet al., “Retrieval- augmented generation for knowledge-intensive nlp tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  24. [32]

    Retrieval augmentation reduces hallucination in conversation,

    K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” inFindings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 3784–3803

    2021

  25. [33]

    Retrieval-augmented generation for generative artificial intelligence in health care,

    R. Yang, Y . Ning, E. Keppo, M. Liu, C. Hong, D. S. Bitterman, J. C. L. Ong, D. S. W. Ting, and N. Liu, “Retrieval-augmented generation for generative artificial intelligence in health care,”Npj health systems, vol. 2, no. 1, p. 2, 2025

    2025

  26. [34]

    Almanac—retrieval-augmented language models for clinical medicine,

    C. Zakka, R. Shad, A. Chaurasia, A. R. Dalal, J. L. Kim, M. Moor, R. Fong, C. Phillips, K. Alexander, E. Ashleyet al., “Almanac—retrieval-augmented language models for clinical medicine,” Nejm ai, vol. 1, no. 2, p. AIoa2300068, 2024

    2024

  27. [35]

    Development of a liver disease–specific large language model chat interface using retrieval-augmented generation,

    J. Ge, S. Sun, J. Owens, V . Galvez, O. Gologorskaya, J. C. Lai, M. J. Pletcher, and K. Lai, “Development of a liver disease–specific large language model chat interface using retrieval-augmented generation,” Hepatology, vol. 80, no. 5, pp. 1158–1168, 2024

    2024

  28. [36]

    Medrag: Enhancing retrieval-augmented generation with knowledge graph-elicited reasoning for healthcare copilot,

    X. Zhao, S. Liu, S.-Y . Yang, and C. Miao, “Medrag: Enhancing retrieval-augmented generation with knowledge graph-elicited reasoning for healthcare copilot,” inProceedings of the ACM on Web Conference 2025, 2025, pp. 4442–4457

    2025

  29. [37]

    Medical graph rag: Evidence-based medical large language model via graph retrieval-augmented generation,

    J. Wu, J. Zhu, Y . Qi, J. Chen, M. Xu, F. Menolascina, Y . Jin, and V . Grau, “Medical graph rag: Evidence-based medical large language model via graph retrieval-augmented generation,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V...

    2025

  30. [38]

    Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines,

    S. Liu, A. B. McCoy, and A. Wright, “Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines,” Journal of the American Medical Informatics Association, vol. 32, no. ...

    2025

  31. [39]

    Knowledge-aware zero-shot learning: Survey and perspective,

    J. Chen, Y . Geng, Z. Chen, I. Horrocks, J. Z. Pan, and H. Chen, “Knowledge-aware zero-shot learning: Survey and perspective,”arXiv preprint arXiv:2103.00070, 2021

    2021 arXiv

  32. [40]

    Llms are not zero-shot reasoners for biomedical information extraction,

    A. Nagar, V . Schlegel, T.-T. Nguyen, H. Li, Y . Wu, K. Binici, and S. Winkler, “Llms are not zero-shot reasoners for biomedical information extraction,” inThe Sixth Workshop on Insights from Negative Results in NLP, 2025, pp. 106–120

    2025

  33. [41]

    Characterizing llm abstention behavior in science qa with context perturbations,

    B. Wen, B. Howe, and L. L. Wang, “Characterizing llm abstention behavior in science qa with context perturbations,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 3437–3450

    2024

  34. [42]

    Delirium,

    J. E. Wilson, M. F. Mart, C. Cunningham, Y . Shehabi, T. D. Girard, A. M. MacLullich, A. J. Slooter, and E. W. Ely, “Delirium,”Nature reviews Disease primers, vol. 6, no. 1, p. 90, 2020

    2020

  35. [43]

    MIMIC-IV, a freely accessible electronic health record dataset,

    A. E. W. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, L.-w. H. Lehman, L. A. Celi, and R. G. Mark, “MIMIC-IV, a freely accessible electronic health record dataset,”Scientific Data, vol. 10, no. 1, p. 1, Jan. 2023. ...

    2023 doi

  36. [44]

    Gusmao-Flores, J

    D. Gusmao-Flores, J. I. F. Salluh, R. Á. Chalhub, and L. C. Quarantini, “The confusion assessment method for the intensive care unit (cam-icu) and intensive care delirium screening checklist (icdsc) for the diagnosis of delirium: a systematic review and meta-analysis of clinic...

    2012

  37. [45]

    A large language model for delirium prediction in the intensive care unit using structured electronic health records,

    M. Contreras, S. Kapoor, J. Zhang, A. Davidson, Y . Ren, Z. Guan, T. Ozrazgat-Baslanti, J. Sena, S. Nerella, A. Bihoracet al., “A large language model for delirium prediction in the intensive care unit using structured electronic health records,”Scientific Reports, vol. 15, no...

    2025

  38. [46]

    Gpt-5.2,

    OpenAI, “Gpt-5.2,” https://openai.com/index/introducing-gpt-5-2/, 2026, accessed 2026-04-27

    2026

  39. [47]

    Will large language models transform clinical prediction?

    Y . Yildiz, G. Nenadic, M. Jani, and D. A. Jenkins, “Will large language models transform clinical prediction?”Diagnostic and Prognostic Re- search, vol. 9, no. 1, pp. 1–7, 2025

    2025

  40. [48]

    Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach,

    E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson, “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach,”Biometrics, vol. 44, no. 3, pp. 837– 845, 1988

    1988

  41. [49]

    Medcalc-bench: Evaluating large language models for medical calcu- lations,

    N. Khandekar, Q. Jin, G. Xiong, S. Dunn, S. S. Applebaum, Z. Anwar, M. Sarfo-Gyamfi, C. W. Safranek, A. A. Anwar, A. Zhanget al., “Medcalc-bench: Evaluating large language models for medical calcu- lations,”Advances in Neural Information Processing Systems, vol. 37, pp. 84 730...

    2024

  42. [50]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,”arXiv preprint arXiv:2410.05229, 2024

    2024 arXiv

  43. [51]

    Limitations of large language models in clinical problem-solving aris- ing from inflexible reasoning,

    J. Kim, A. Podlasek, K. Shidara, F. Liu, A. Alaa, and D. Bernardo, “Limitations of large language models in clinical problem-solving aris- ing from inflexible reasoning,”Scientific reports, vol. 15, no. 1, p. 39426, 2025

    2025

  44. [52]

    Less is more: information needs, information wants, and what makes causal models useful,

    S. Kleinberg and J. K. Marsh, “Less is more: information needs, information wants, and what makes causal models useful,”Cognitive Research: Principles and Implications, vol. 8, no. 1, p. 57, 2023

    2023

  45. [53]

    Does prompt formatting have any impact on llm performance?

    J. He, M. Rungta, D. Koleczek, A. Sekhon, F. X. Wang, and S. Hasan, “Does prompt formatting have any impact on llm performance?”arXiv preprint arXiv:2411.10541, 2024

    2024 arXiv

  46. [54]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,”arXiv preprint arXiv:2310.11324, 2023

    2023 arXiv

  47. [55]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 31 210–31 227

    2023

  48. [56]

    Deiden- tifying medical documents with local, privacy-preserving large language models: the llm-anonymizer,

    I. C. Wiest, M.-E. Leßmann, F. Wolf, D. Ferber, M. V . Treeck, J. Zhu, M. P. Ebert, C. B. Westphalen, M. Wermke, and J. N. Kather, “Deiden- tifying medical documents with local, privacy-preserving large language models: the llm-anonymizer,”NEJM AI, vol. 2, no. 4, p. AIdbp2400537, 2025

    2025

  49. [57]

    Focused chain-of-thought: Efficient llm reasoning via structured input information,

    L. Struppek, D. Hintersdorf, H. Struppek, D. Neider, and K. Kersting, “Focused chain-of-thought: Efficient llm reasoning via structured input information,”arXiv preprint arXiv:2511.22176, 2025. APPENDIXA REPORT1 A. Report v1: Factor-Based Risk Framework To compute the probabil...

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.