Pith. sign in

REVIEW 3 major objections 5 minor 56 references

ADRD-Bench is introduced as the first benchmark dedicated to Alzheimer's disease and related dementias, claiming that high accuracy on clinical and caregiving questions still hides systematic overgeneralization and goal misalignment in prac

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:06 UTC pith:LR4ATMWV

load-bearing objection Worth a serious look as the first ADRD-focused LLM QA resource, but the abstract/body inconsistencies and unvalidated 'AD' keyword filter mean it needs revision before the numbers can be trusted. the 3 major comments →

arxiv 2602.11460 v2 pith:LR4ATMWV submitted 2026-02-12 cs.CL

ADRD-Bench: A Preliminary LLM Benchmark for Alzheimer's Disease and Related Dementias

Guangxin Zhao , Jiahao Zheng , Malaz Boustani , Jarek Nabrzyski , Yiyu Shi , Meng Jiang , Zhi Zheng This is my paper
classification cs.CL
keywords Large language modelsAlzheimer's diseasedementiamedical benchmarkcaregivingquestion answeringevaluationADRD
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces ADRD-Bench, which the authors present as the first evaluation benchmark tailored to Alzheimer's disease and related dementias (ADRD). It comprises 1,352 ADRD-related questions consolidated from seven established medical benchmarks (ADRD Unified QA) and 149 new caregiving questions drawn from the Aging Brain Care program (ADRD Caregiving QA). The authors evaluate 33 large language models and report that top models exceed 0.9 accuracy, yet qualitative case studies reveal systematic overgeneralization and a tendency to prioritize conversation continuity over patient disengagement. The paper argues that existing medical benchmarks contain under 1% ADRD content and almost no caregiving context, leaving a critical gap for safe deployment. If correct, ADRD-Bench offers a reusable resource and evidence that accuracy alone is insufficient for ADRD care.

Core claim

The central claim is that ADRD-Bench is the first ADRD-specific LLM benchmark, providing a unified assessment of clinical knowledge and practical caregiving reasoning. On the 1,352-question Unified QA set, accuracies across 33 models ranged from 0.47 to 0.93, with Llama-3.1-70B-Instruct scoring 0.9334; on the 149-question Caregiving QA set, accuracies ranged from 0.63 to 0.97, with GPT-5.2 scoring 0.9664. The paper's case analysis shows that even the best models overgeneralize stage-specific facts (e.g., treating all Alzheimer's patients as unable to perform daily activities) and misinterpret disengagement cues, recommending re-engagement when the patient signals a need for space. The author

What carries the argument

The load-bearing mechanism is the benchmark's split structure: ADRD Unified QA, built by keyword filtering 'Alzheimer's', 'Dementia', 'ADRD', and the bare token 'AD' across seven medical benchmarks, and ADRD Caregiving QA, 149 clinician-reviewed True/False and multiple-choice items derived from the Aging Brain Care (ABC) program. The two sets are designed to separate clinical knowledge from daily caregiving reasoning, and the exact-match accuracy metric is used to rank models; the Caregiving QA's focus on practical scenarios is what exposes failures that exam-style questions miss.

Load-bearing premise

The claim that the Unified QA set measures ADRD knowledge depends on keyword matching—especially the bare token 'AD'—cleanly separating ADRD from non-ADRD questions; the paper reports only counts, not precision or recall.

What would settle it

Manually audit a random sample (e.g., 100 items) of the 1,352 Unified QA questions and compute the fraction that are not genuinely ADRD-related. If false positives exceed a few percent, the reported per-model accuracies and rankings would need re-estimation, and the benchmark's ADRD-specificity claim weakens.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A reusable ADRD evaluation resource now exists, letting future LLM releases be compared on dementia-specific clinical and caregiving tasks.
  • The finding that top models overgeneralize on caregiving questions implies that accuracy-based scores are insufficient for safety-critical ADRD deployment.
  • The significant correlation between Unified QA and Caregiving QA accuracy among medical models, but not general models, suggests domain-specific fine-tuning yields more consistent knowledge transfer.
  • Model size correlates with accuracy on both sets, but size alone does not predict caregiving competence, since several smaller medical models outperformed larger general ones.
  • High benchmark accuracy does not establish clinical reliability, per the paper's own case studies and stated limitations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The keyword filter's reliance on the bare token 'AD' likely admits non-ADRD questions (e.g., 'as directed' or 'admission'), so a precision audit of the 1,352-question Unified QA set could materially shift the reported model rankings.
  • A natural extension would convert the Caregiving QA into multi-turn or open-ended tasks; the paper itself notes this as a limitation, and such tasks may reveal empathy and conversational failures that discrete-choice accuracy cannot capture.
  • Since the Caregiving QA is rooted in a single US-based care model (ABC), adapting it to other care protocols or languages would test whether the observed failures are model-level or context-specific.
  • The abstract and full text give different model counts (36 vs 33); readers should treat the evaluation as preliminary until the model list is reconciled.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ADRD-Bench, a two-part benchmark for evaluating LLMs on Alzheimer's disease and related dementias. The ADRD Unified QA component aggregates 1,352 questions extracted from seven public medical benchmarks via keyword filtering, and the ADRD Caregiving QA component adds 149 new True/False and multiple-choice questions derived from the Aging Brain Care (ABC) program. The authors evaluate 33 LLMs using exact-match accuracy, report model rankings and correlation analyses, and provide qualitative case studies of reasoning failures. The stated contribution is the first ADRD-specific LLM benchmark, enabling assessment of both clinical knowledge and practical caregiving reasoning.

Significance. If the benchmark is valid, it fills a real gap: existing medical benchmarks have very low ADRD coverage, and caregiving scenarios are almost absent. The paper's strengths include a reproducible evaluation protocol (prompts in the appendix, exact-match scoring, 33 models), public release of the dataset, and concrete case analyses that go beyond accuracy. The caregiving set is a useful first step toward testing LLMs in daily-care situations. However, the validity of the central claim depends on the purity of the keyword-extracted Unified QA set, on resolving internal inconsistencies in the reported numbers, and on the independence of the caregiving ground truth. These issues require correction before the benchmark can be relied upon as ADRD-specific.

major comments (3)
  1. [§3.1, Table 1] The ADRD Unified QA set is constructed by searching for the keywords 'Alzheimer's', 'Dementia', 'ADRD', and 'AD' in 12 public benchmarks, and Table 1 reports only aggregate counts. No precision, recall, or manual audit of the extracted questions is provided. The token 'AD' is highly ambiguous in clinical text (e.g., 'as directed', right ear, autosomal dominant, admission, and as a prefix in ADHD/ADEM), so substring matching is likely to introduce non-ADRD items. Since the 1,352 Unified QA questions constitute about 90% of the benchmark, contamination directly affects the headline accuracies (e.g., Llama-3.1-70B's 0.9334) and the claim that this is an 'ADRD-specific' benchmark. The Limitations section does not mention this risk. Please report a stratified manual audit with inter-annotator agreement, provide examples of excluded items, and either tighten the filter or quantify the contamin
  2. [Abstract vs. §3.3/§4] The abstract and the body disagree on load-bearing numbers. The abstract states 1,438 questions and 36 evaluated models; the body and Table 2 state 1,352 questions and 33 models. The abstract reports closed-source accuracy of 0.83–0.93 (mean 0.90, std 0.03), while the body reports 0.83–0.91 (mean 0.89, std 0.03). These inconsistencies affect the reported performance envelope and must be reconciled. I also note the summed column sizes in Table 2 match the body (1352 and 149), which suggests the abstract is erroneous, but the published version should be internally consistent throughout.
  3. [§3.2, §5] The 149 Caregiving QA items were derived from ABC program materials and reviewed by a senior clinician who is the lead designer of the ABC program and a coauthor. The gold answers therefore encode a single program's perspective and a single reviewer's judgment, which is a validity concern for the only newly constructed portion of the benchmark. The limitations mention that caregiving decisions are context-dependent and that the benchmark is rooted in the ABC model, but they do not address the absence of external validation or inter-annotator agreement. Please provide an independent clinical review, an annotation protocol, agreement statistics, or explicitly frame the set as measuring adherence to ABC guidance rather than general ADRD caregiving competence.
minor comments (5)
  1. [§3.1, Table 1] The text says 'searched 12 public benchmark datasets' but the consolidated set is drawn from seven. Clarify the relationship between the 12 datasets in Table 1 and the seven used in ADRD Unified QA. Also specify whether the keyword search was case-insensitive and whether 'AD' was matched as a token or substring.
  2. [Table 2, §4.1–4.2] Several subcategory accuracies are computed on only 12 or 29 questions (PubMedQA, MedBullets, MedHallu, MEDEC). Exact match on 12 items has a standard error around 14 percentage points, so comparisons such as 0.8333 vs 0.7500 are single-question flips. Please either pool these small subsets, report confidence intervals, or avoid interpreting the subcategory scores individually.
  3. [§4.4, Figures 5–8] The text reports '85% LLMs answered' and '79% LLMs chose' without stating the denominator. Since 33 models were evaluated, please give exact counts (e.g., 28/33) to make the qualitative claims precise.
  4. [§3.2, Appendix A.1] The multiple-choice prompt for ADRD Caregiving QA lists options A–E, but the set contains 29 multiple-choice questions that may have fewer than five options. Clarify whether the option count varies and how the prompt is adapted accordingly.
  5. [Figures 1–2] Closed-source models are shown as green points but have no model-size value on the x-axis. State how these points are placed (e.g., by parameter-equivalent estimate or ordinal position) and ensure the correlation analyses exclude them or handle them consistently.

Circularity Check

0 steps flagged

No load-bearing circularity: ADRD-Bench is a curation and empirical evaluation resource; model scores are measured against fixed labels, not derived from the construction inputs.

full rationale

ADRD-Bench is a benchmark construction and empirical evaluation paper, not a derivation with fitted parameters or equations. The ADRD Unified QA half inherits ground-truth labels from seven external benchmarks (PubMedQA, HEAD-QA, MedBullets, MedMCQA, MedQA, MEDEC, MedHallu), and the reported accuracies are direct measurements against those fixed labels, so they are not predictions forced by construction. The keyword-extraction step (Section 3.1, Table 1) is an operationalization whose precision is a legitimate correctness risk—'AD' is ambiguous in medical text and no precision/recall audit is reported—but this is a measurement-validity concern, not a circular reduction. The novel ADRD Caregiving QA (Section 3.2) is constructed from ABC program materials and reviewed by the ABC program's lead designer, who is also a coauthor; this is a self-referential provenance/validation loop and a limitation on the independence of the gold answers, but the benchmark does not use those labels to derive the construction itself, and the model scores remain empirical. Self-citations to ABC program publications support the source's credentials, but they are backed by a published RCT and CMS policy adoption and are not used to forbid alternatives or to derive the headline accuracies. No load-bearing argument reduces to its own input; the central claim is a curated resource plus external evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters in the mathematical sense; the benchmark's authority rests on extraction assumptions, source-program validity, and metric adequacy.

axioms (4)
  • domain assumption Keyword search on {'Alzheimer's', 'Dementia', 'ADRD', 'AD'} identifies all and only ADRD-related questions in source benchmarks.
    Section 3.1; no precision/recall reported; bare 'AD' is ambiguous in medical text.
  • domain assumption ABC program materials encode correct ADRD caregiving ground truth.
    Section 3.2; evidence-based program, but labels were reviewed by the program's lead designer, a coauthor, without independent adjudication.
  • domain assumption Exact-match accuracy over discrete-choice items is an adequate primary measure of ADRD LLM capability.
    Section 3.3 and Limitations; paper itself notes it does not measure empathy, multi-turn dialogue, or open-ended generation.
  • domain assumption Source benchmarks' ground-truth labels are correct and their ADRD items are representative.
    Section 3.1; the paper inherits labels from MedQA, PubMedQA, MedMCQA, HEAD-QA, MedBullets, MEDEC, MedHallu without re-validation.

pith-pipeline@v1.3.0-alltime-deepseek · 17643 in / 9572 out tokens · 101740 ms · 2026-08-03T00:06:52.563206+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have shown great potential for healthcare applications. However, existing evaluation benchmarks provide minimal coverage of Alzheimer's Disease and Related Dementias (ADRD). To address this gap, we introduce ADRD-Bench, a preliminary ADRD-specific LLM benchmark. ADRD-Bench has two components: 1) ADRD Unified QA, a synthesis of 1,438 questions consolidated from seven established medical benchmarks, providing a unified assessment of clinical knowledge; and 2) ADRD Caregiving QA, a novel set of 149 questions derived from a nationally adopted, large clinical trials supported brain health management program, mitigating the lack of practical caregiving context in existing benchmarks. We evaluated 36 state-of-the-art LLMs on the proposed ADRD-Bench. Results showed that the accuracy of open-weight general models, open-weight medical models, and frontier closed-source general models ranged from 0.63 to 0.93 (mean: 0.77; std: 0.09), 0.47 to 0.93 (mean: 0.81; std: 0.14), and 0.83 to 0.93 (mean: 0.90; std: 0.03), respectively. While top-tier models achieved high accuracies (>0.9), case studies revealed inconsistent reasoning quality and stability, highlighting a critical need for domain-specific improvement to enhance LLMs' knowledge and reasoning grounded in daily caregiving data. The entire dataset is available at https://github.com/IIRL-ND/ADRD-Bench.

Figures

Figures reproduced from arXiv: 2602.11460 by Guangxin Zhao, Jarek Nabrzyski, Jiahao Zheng, Malaz Boustani, Meng Jiang, Yiyu Shi, Zhi Zheng.

Figure 1
Figure 1. Figure 1: Model accuracy vs. model size on the ADRD Unified [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Model accuracy vs. model size on the ADRD Care [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: True/False question example of ADRD Caregiving [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Reasoning by llama3:8b-instruct-fp16 and claude [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Multiple-choice example of ADRD Caregiving QA [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Reasoning by qwen3-max-2025-09-23 and claude [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 10 linked inside Pith

  1. [1]

    Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the MEDIQA 2019 shared task on textual inference, question entail- ment and question answering. InProceedings of the 18th BioNLP workshop and shared task. 370–379

    2019

  2. [2]

    Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen-Yildiz, Fei Xia, and Thomas Lin. 2025. MedEC: A benchmark for medical error detection and correction in clinical notes. InFindings of the Association for Computational Linguistics: ACL 2025. 22539–22550

    2025

  3. [3]

    Felix Agbavor and Hualou Liang. 2022. Predicting dementia from spontaneous speech using large language models.PLOS Digital Health1, 12 (2022), e0000168

    2022

  4. [4]

    Catherine A Alder, Michael A LaMantia, Mary Guerriero Austrom, and Malaz A Boustani. 2024. The Indiana Aging Brain Care Project. InGeriatrics Models of Care: Bringing Best Practice to an Aging America. Springer, 317–324

    2024

  5. [5]

    2024.Dementia Care Delivery and Accreditation

    Alzheimer’s Association. 2024.Dementia Care Delivery and Accreditation. Re- trieved January 22, 2026 from https://www.alz.org/professionals/health-systems- medical-professionals/health-systems/dementia-care-delivery-accreditation

    2024

  6. [6]

    Alzheimer’s Association. 2024. Mapping a Better Future for Dementia Care Navi- gation. Retrieved January 22, 2026 from https://www.alz.org/getmedia/a7390c29- e4c0-4d3c-b000-377b6c112eea/alzheimers-facts-and-figures-special- report.pdf

    2024

  7. [7]

    Alzheimer’s Association. 2024. 2024 Alzheimer’s disease facts and figures. Alzheimer’s & Dementia20, 5 (2024), 3708–3821. doi:10.1002/alz.13809

    doi:10.1002/alz.13809 2024

  8. [8]

    Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, An- drea Vallone, Alex Beutel, et al

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, An- drea Vallone, Alex Beutel, et al. 2025. HealthBench: Evaluating large language models towards improved human health.arXiv preprintarXiv:2505.08775 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Alder, Craig A

    Malaz Boustani, Catherine A. Alder, Craig A. Solid, and David Reuben. 2019. An alternative payment model to support widespread use of collaborative dementia care models.Health Affairs38, 1 (2019), 54–59

    2019

  10. [10]

    Malaz A Boustani, Steven R Counsell, Anthony Perkins, Abdelfattah Alhader, Kathryn I Frank, Diana P Summanwar, and Karen L Fortuna. 2025. Effectiveness of a Telephonic Aging Brain Care Model for Medicaid Home and Community Services for Dementia Patients and Their Caregivers.Journal of the American Geriatrics Society(2025)

    2025

  11. [11]

    Callahan, Malaz A

    Christopher M. Callahan, Malaz A. Boustani, Frederick W. Unverzagt, Mary G. Austrom, Teresa M. Damush, Anthony J. Perkins, Bridget A. Fultz, Siu L. Hui, Steven R. Counsell, and Hugh C. Hendrie. 2006. Effectiveness of collaborative care for older adults with Alzheimer disease in primary care: A randomized controlled trial.JAMA295, 18 (2006), 2148–2157

    2006

  12. [12]

    Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2025. Benchmark- ing large language models on answering and explaining challenging medical questions. InProceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 3563–3599

    2025

  13. [13]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2023. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research24, 240 (2023), 1–113

    2023

  14. [14]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    French, Michael A

    Dustin D. French, Michael A. LaMantia, Lee R. Livin, Dorian Herceg, Catherine A. Alder, and Malaz A. Boustani. 2014. Healthy Aging Brain Center improved care coordination and produced net savings.Health Affairs33, 4 (2014), 613–618

    2014

  16. [16]

    Reuben, Rebecca Stoeckle, David Bass, Malaz Boustani, Carolyn Clevenger, Ian Kremer, David R

    Kristin Lees Haggerty, David B. Reuben, Rebecca Stoeckle, David Bass, Malaz Boustani, Carolyn Clevenger, Ian Kremer, David R. Lee, Madelyn Johnson, Mor- gan J. Minyo, et al . 2025. GUIDE and Beyond: Strategies for Comprehensive Dementia Care Integration.Journal of the American Geriatrics Society73, 11 (2025), 3548–3554

    2025

  17. [17]

    Wordh Ul Hasan, Kimia Tuz Zaman, Xin Wang, Juan Li, Bo Xie, and Cui Tao. 2024. Empowering alzheimer’s caregivers with conversational AI: A novel approach for enhanced communication and personalized support.npj Biomedical Innovations 1, 1 (2024), 3

    2024

  18. [18]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing.arXiv preprintarXiv:2009.03300 (2020)

    Pith/arXiv arXiv 2020

  19. [19]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences11, 14 (2021), 6421

    2021

  20. [20]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). 2567–2577

    2019

  21. [21]

    Mert Karabacak and Konstantinos Margetis. 2023. Embracing large language models for medical applications: Opportunities and challenges.Cureus15, 5 (2023)

    2023

  22. [22]

    Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, et al. 2024. Medcalc-bench: Evaluating large language models for medical calcula- tions.Advances in Neural Information Processing Systems37 (2024), 84730–84745

    2024

  23. [23]

    Jonathan Kim, Anna Podlasek, Kie Shidara, Feng Liu, Ahmed Alaa, and Danilo Bernardo. 2025. Limitations of large language models in clinical problem-solving arising from inflexible reasoning.Scientific Reports15, 1 (2025), 39426

    2025

  24. [24]

    Siun Kim and Hyung-Jin Yoon. 2025. Questioning our questions: How well do medical QA benchmarks evaluate clinical capabilities of language models?. In Proceedings of the 24th Workshop on Biomedical Language Processing. 274–296

    2025

  25. [25]

    Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Chanwoo Park, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, et al . 2025. Medical hallucinations in foundation models and their impact on healthcare. arXiv preprintarXiv:2503.05777 (2025)

    arXiv 2025

  26. [26]

    Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al

    Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models.PLOS Digital Health 2, 2 (2023), e0000198

    2023

  27. [27]

    LaMantia, Catherine A

    Michael A. LaMantia, Catherine A. Alder, Christopher M. Callahan, Sujuan Gao, Dustin D. French, Mary G. Austrom, Karim Boustany, Lee Livin, Bharath Bynagari, and Malaz A. Boustani. 2015. The aging brain care medical home: Preliminary data.Journal of the American Geriatrics Society63, 6 (2015), 1209–1213

    2015

  28. [28]

    Yesheng Liu, Hao Li, Haiyu Xu, Baoqi Pei, Jiahao Wang, Mingxuan Zhao, Jing- Shu Zheng, Zheqi He, J. G. Yao, Bowen Qin, Xi Yang, and Jiajun Zhang. 2025. Beyond multiple choice: A hybrid framework for unifying robust evaluation and verifiable reasoning training.arXiv preprintarXiv:2511.17405 (2025)

    arXiv 2025

  29. [29]

    Subhankar Maity and Manob Jyoti Saikia. 2025. Large language models in healthcare and medical applications: A review.Bioengineering12, 6 (2025), 631

    2025

  30. [30]

    2024.Llama-3.1-70B-Instruct

    Meta AI. 2024.Llama-3.1-70B-Instruct. Retrieved January 10, 2026 from https: //huggingface.co/meta-llama/Llama-3.1-70B-Instruct

    2024

  31. [31]

    Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of GPT-4 on medical challenge problems.arXiv preprintarXiv:2303.13375 (2023)

    Pith/arXiv arXiv 2023

  32. [32]

    2025.Introducing GPT-5.2

    OpenAI. 2025.Introducing GPT-5.2. Retrieved January 10, 2026 from https: //openai.com/index/introducing-gpt-5-2/

    2025

  33. [33]

    2025.Meditron3-70B

    OpenMeditron. 2025.Meditron3-70B. Retrieved January 10, 2026 from https: //huggingface.co/OpenMeditron/Meditron3-70B

    2025

  34. [34]

    2024.Open Medical LLM Leaderboard

    Ankit Pal, Pasquale Minervini, Andreas Geert Motzfeldt, Aryo Pradipta Gema, and Beatrice Alex. 2024.Open Medical LLM Leaderboard. Retrieved January 22, 2026 from https://huggingface.co/spaces/openlifescienceai/open_medical_llm_ leaderboard

    2024

  35. [35]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. InConference on Health, Inference, and Learning. 248–260

    2022

  36. [36]

    Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, and Ying Ding. 2025. MedHallu: A comprehensive benchmark for detecting medical hallucinations in large language models.arXiv preprintarXiv:2502.14302 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Made Agus Setiawan, Yuhan Wang, Haomin Hu, Andi Saptono, and Yong Kyung Choi

    Bambang Parmanto, Bayu Aryoyudanta, Timothius Wilbert Soekinto, I. Made Agus Setiawan, Yuhan Wang, Haomin Hu, Andi Saptono, and Yong Kyung Choi. 2024. A reliable and accessible caregiving language model (CaLM) to support tools for caregivers: Development and evaluation study.JMIR Formative Research8 (2024), e54633

    2024

  38. [38]

    Lotem Peled-Cohen and Roi Reichart. 2025. A systematic review of NLP for dementia: Tasks, datasets, and opportunities.Transactions of the Association for Computational Linguistics13 (2025), 1204–1244

    2025

  39. [39]

    Phi Team. 2024. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprintarXiv:2404.14219 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al

  41. [41]

    Dirk Steijger, Hannah Christie, Sil Aarts, Wijnand IJselsteijn, Hilde Verbeek, and Marjolein de Vugt. 2025. Use of artificial intelligence to support quality of life of people with dementia: A scoping review.Ageing Research Reviews(2025), 102741

    2025

  42. [42]

    Nestor, Ali Soroush, Pierre A

    Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G. Nestor, Ali Soroush, Pierre A. Elias, Ziyang Xu, Ying Ding, Greg Durrett, and Justin F. Rousseau. 2023. Evalu- ating large language models on medical evidence summarization.NPJ Digital Medicine6, 1 (2023), 158

    2023

  43. [43]

    Treder, Sojin Lee, and Kamen A

    Matthias S. Treder, Sojin Lee, and Kamen A. Tsvetanov. 2024. Introduction to large language models (LLMs) for dementia care and research.Frontiers in Dementia3 (2024), 1385303. 9

    2024

  44. [44]

    Emre Umucu, Guillermina Solis, Leon Garza, Emilia Rivas, Beatrice Lee, Anantaa Kotal, and Aritran Piplai. 2025. Empathy by design: Aligning large language models for healthcare dialogue.arXiv preprintarXiv:2512.06097 (2025)

    arXiv 2025

  45. [45]

    David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A healthcare dataset for complex reasoning.arXiv preprintarXiv:1906.04701 (2019)

    Pith/arXiv arXiv 2019

  46. [46]

    2024.Ethics and Governance of Artificial Intelligence for Health: Large Multi-Modal Models

    World Health Organization. 2024.Ethics and Governance of Artificial Intelligence for Health: Large Multi-Modal Models. WHO Guidance. World Health Organization, Geneva, Switzerland

    2024

  47. [47]

    2025.Dementia

    World Health Organization. 2025.Dementia. Retrieved January 22, 2026 from https://www.who.int/news-room/fact-sheets/detail/dementia

    2025

  48. [48]

    2024.Grok 4.1 Fast (Non-Reasoning)

    xAI. 2024.Grok 4.1 Fast (Non-Reasoning). Retrieved January 10, 2026 from https://docs.x.ai/docs/models/grok-4-1-fast-non-reasoning

    2024

  49. [49]

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al

  50. [50]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprintarXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    Chen, Peilin Zhou, Junling Liu, et al

    Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S. Chen, Peilin Zhou, Junling Liu, et al . 2023. A survey of large language models in medicine: Progress, application, and challenge.arXiv preprint arXiv:2311.05112 (2023). 10 A Appendix A.1 ADRD-Bench Prompts Prompt for PubMedQA You are a highly knowledgeable AI assista...

    Pith/arXiv arXiv 2023

  52. [54]

    Early stage: Patients may have mild memory problems but can generally still perform most ADLs independently (bathing, dressing, eating, toileting, etc.)

  53. [55]

    Middle stage: Patients begin to have more difficulty with ADLs and may need assistance with some tasks

  54. [56]

    The Alzheimer’s patient

    Late/Advanced stage: Patients typically lose the ability to perform basic ADLs and require full-time care and assistance. The question states "The Alzheimer’s patient" in a general sense, implying the condition overall. In advanced stages of Alzheimer’s disease, patients do become unable to perform basic activities of daily living such as bathing, dressin...

    2025

  55. [2023]

    Large language models encode clinical knowledge.Nature620, 7972 (2023), 172–180

    2023

  56. [2025]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprintarXiv:2506.07044 (2025)

    Pith/arXiv arXiv 2025