REVIEW 3 major objections 3 minor 13 references
Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness
T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper shows that non-expert humans and large language models both miss many clinically diagnosed dementia cases, while the cues they rely on differ: humans use a narrow, sometimes misleading set, and LLMs use a richer set closer to…
desk verdict A genuinely new perception-angle paper whose central human-vs-LLM comparison is compromised by GPT-4o serving as both feature extractor and one of the perception voters. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a four-step explainable pipeline. First, 38 binary features are defined in consultation with a neurologist and a neuropsychologist, grouped into five categories: Objective Interpretation, Subjective Interpretation, Linguistic, Human Experience, and Interview Context. Second, a large language model (GPT-4o) annotates every transcript for all 38 features, after an Alternative Annotator Test showed a 90% chance its annotations were as good as or better than human annotations. Third, stepwise logistic regression models are fit separately to human perception, LLM perception, and clinical diagnosis using the same features. Fourth, the coefficients are compared to see which cues each judgment type actually relies on; the same feature representation across all three makes the comparison interpretable.
What would settle it
Have a fresh group of non-expert humans annotate the same 38 features on a random sample of transcripts, then fit the same logistic regression using those human feature annotations to predict human perception. If the model fit rises well above the reported McFadden's $R^2=0.058$, the conclusion that human perception is inherently inconsistent would be undermined, because the noisy part would be the original LLM feature annotations rather than human judgment.
Extended reading notes
Core claim
The paper's central claim is that non-expert human perception of dementia in language is inconsistent and driven by a narrow, partly misleading set of cues, whereas LLM perception is driven by a broader set that overlaps more with clinical diagnosis; nevertheless both are biased toward false negatives. Concretely, human annotators agreed only weakly (Fleiss' $\kappa=0.28$), a logistic regression on the 38 expert-guided features explained almost none of their judgments (McFadden's $R^2=0.058$), and the few cues that mattered included reading short sentences as a sign of health even though clinicians associate short sentences with dementia. LLM judgments were better explained by the same features (McFadden's $R^2=0.527$) and drew on subjective and emotional cues such as Theory of Mind, lightheartedness, and sadness. With 283 clinically diagnosed dementia cases, humans correctly identified 57% and LLMs 60%, and among their errors both groups predominantly missed dementia cases (65% and 70% false negatives, respectively). The authors also report that annotators' self-described reasoning did not match the features that actually predicted their judgments.
Load-bearing premise
The load-bearing premise is that the 38 binary features extracted by a single LLM (GPT-4o) faithfully capture the linguistic cues that actually drive human and LLM judgments; the validation of this feature set was carried out on only 10 transcripts.
Editorial extensions
If this is right
- If non-experts read short sentences as a sign of health, awareness materials can be aimed directly at that specific misreading.
- Any LLM-based early-warning tool should be designed around the demonstrated false-negative tendency, since 70% of LLM errors were missed dementia cases.
- If people's self-reported cues do not match their modeled decision patterns, asking users to reflect on their own reasoning is not a reliable way to audit or improve their judgments.
- If LLMs already use a feature set closer to clinical diagnosis, their explanations could serve as coaching signals to help non-experts attend to clinically relevant cues.
Reading between the lines
- The single-transcript design likely understates how well humans detect decline in real life, where repeated exposure gives a baseline; the narrow-cue result may not transfer to longitudinal monitoring.
- A practical screening system would need to track changes over time rather than classify individual transcripts, because the LLM false-negative pattern shows a transcript without overt linguistic problems is almost always labeled healthy.
- If the 38-feature set omits cues humans actually use, the low human model fit could reflect incomplete measurement rather than human inconsistency; re-annotating a larger sample with human feature labels would settle this.
- The paper does not test whether LLM explanations improve human accuracy, but its results make that a natural next experiment for public-awareness tools.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates how non-expert humans and LLMs perceive dementia from language, using 514 Cookie Theft picture descriptions from the Pitt corpus. The authors collect intuitive healthy/dementia judgments from 27 non-expert annotators and from three LLMs (GPT-4o, LLaMA 3, Gemini-1.5-Pro), define 38 expert-guided binary features, annotate those features with GPT-4o, and fit stepwise logistic regressions to explain human perception, LLM majority perception, and clinical diagnosis. The central claims are that human perception is inconsistent and relies on a narrow and sometimes misleading cue set, that LLMs draw on a richer feature set more aligned with clinical patterns, and that both groups show a tendency toward false negatives. The paper also analyzes misperceptions and self-reported human rationales.
Significance. If the central claims were fully supported, the paper would make a valuable contribution to dementia awareness and to the design of LLM-based monitoring tools. The research question is timely and understudied, the use of a clinically grounded corpus is appropriate, and the authors are transparent about several limitations. They also apply a statistical test for LLM-as-annotator quality and present inherently interpretable models. However, the main human-vs-LLM asymmetry currently rests on a confounded measurement setup and on an interpretation of a very low human model fit that may not be unique. The paper's headline conclusion is therefore not yet established, although it is plausibly fixable with additional analyses.
major comments (3)
- [§4.2, §5.2, Table 3] The main evidence for the claim that LLMs rely on a richer feature set is confounded. GPT-4o is both the annotator that produced all 38 feature values for all 514 transcripts (Section 5.2) and one of the three LLMs whose majority vote defines the LLM perception label (Section 4.2). The very high McFadden R² of 0.527 in Table 3 may therefore reflect GPT-4o's internal consistency rather than a genuinely broader or more clinically aligned cue set. I request that the authors re-estimate the LLM perception model using only the LLaMA-3 and Gemini-1.5-Pro majority, and ideally with features annotated by a model that is not part of the perception panel, and report whether the R² and the coefficient pattern survive.
- [§7.1, Table 3] The low human-perception fit (McFadden R²=0.058) is interpreted as evidence of human inconsistency, but it is equally consistent with the alternative that the 38 binary features annotated by GPT-4o do not capture the cues humans actually use. The Alternative Annotator Test in Section 5.2 validates agreement between GPT-4o and human annotators on the feature values for 10 transcripts (380 values); it does not establish that this feature vocabulary is complete or causally aligned with human perception. Without a larger human-annotated feature validation, or an analysis using an expanded feature set derived from the human rationales in Section 7.3, the central asymmetry between narrow human cues and rich LLM cues may be an artifact of measurement rather than a substantive finding.
- [§6, Table 3] The set of significant features is obtained through stepwise logistic regression on the full dataset, and the p-values in Table 3 are not adjusted for the 38 candidate predictors. Under this procedure, the number of significant features per judgment (4 for humans, 13 for LLMs, 8 for clinical diagnosis) is not a clean measure of cue 'richness,' since stepwise selection is known to be unstable and to produce inflated significance. I ask for a robustness check, such as bootstrap stability of selected features or a regularized alternative like LASSO, before the richness comparison is used as a central result. In addition, the claim that LLMs align 'more closely' with clinical patterns is not quantified; a formal comparison, such as correlation or overlap of coefficient vectors including signs, would make this claim testable.
minor comments (3)
- [§7.3] The statement that 65% of self-reported cues align with the predefined feature set is based on a manual review of responses from 18 of 27 annotators, but the coding procedure and coder agreement are not described. Please provide the coding protocol and report inter-coder reliability.
- [§6] Please state explicitly in the modeling section that all logistic regressions use the majority-vote perception labels, and clarify whether the clinical diagnosis model is fit on the same feature matrix as the perception models.
- [Appendix C.1] Five features are removed because they are positive in fewer than 5% of samples. Since some of these features (e.g., empathy, irritability) are clinically interesting, please report whether the main conclusions change if these features are merged with related categories or analyzed descriptively.
Circularity Check
Partial circularity: GPT-4o both annotates the 38 features and contributes to the LLM perception label, so the LLM-side 'richer feature set' claim is partly a self-consistency artifact.
-
other
[Sections 4.2, 5.2, 7.1]
"GPT-4o, LLaMA 3, and Gemini-1.5-Pro were provided with the same transcripts and also asked to provide their best judgment ... we used their majority vote in our analysis. ... we used GPT-4o to label all 38 binary features across our 514 descriptions ... a very strong fit for LLM dementia perception (0.527), suggesting that our model and features captures a reliable underlying signal."
The LLM perception label is a majority vote over three LLMs, one of which is GPT-4o, while the 38 features are generated by GPT-4o on the same 514 transcripts. The logistic-regression fit of LLM perception on these features therefore partly measures GPT-4o's internal consistency between its own feature annotations and its own perception vote, rather than an independent property of 'LLM perception.' The inference that LLMs use a 'richer, more nuanced feature set' is thus partially a self-consistency artifact. The human-perception model (R2=0.058) and clinical-diagnosis model (R2=0.209) use labels independent of GPT-4o, so the central comparison retains some independent content, but the LLM-side richness claim is not independently established by this design.
full rationale
The only substantive circularity concern is GPT-4o's dual role: it is the sole annotator of all 38 features and also one of the three models whose majority vote defines LLM perception. This can inflate the LLM model's McFadden R2 and the number of significant features, making the 'LLMs are richer' finding partly an artifact of same-source measurement. The Alternative Annotator Test (Calderon et al., 2025) is a self-citation, but it is a general statistical method applied to this data rather than a dementia-specific result, and it is supplemented by human agreement (kappa=0.557) on 10 transcripts; it is therefore not a load-bearing circular self-citation. The human-perception and clinical-diagnosis models are grounded in independent labels, so the central human-vs-LLM asymmetry is not wholly reduced to a fitting or self-citation chain. However, because the feature representation used for all models is GPT-4o's output, and GPT-4o also contributes to the LLM perception label, the LLM-side conclusions are partially confounded. Score 4 reflects this partial, localized circularity rather than a fully circular derivation.
Assumptions & free parameters
free parameters (3)
- Low-frequency feature removal threshold =
5% (fewer than 25 positive samples)
- Alternative Annotator Test threshold epsilon =
0.1
- Stepwise regression significance level =
p < 0.05
assumptions (5)
- domain assumption The 38 expert-guided binary features capture the linguistically relevant cues in Cookie Theft picture descriptions that drive perceptions of dementia.
- domain assumption GPT-4o's binary feature annotations are as reliable as human annotations for all 514 transcripts.
- domain assumption Logistic regression coefficients approximate the cues that drive perceptions.
- domain assumption Binarizing MCI and 'Other' diagnoses into the Dementia class is appropriate for perception modeling.
- domain assumption The Pitt corpus Cookie Theft transcripts are representative of dementia-related language.
Cite this review
Pith. "Pith review of Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness." pith.science (2026). https://pith.science/paper/XGJIDKOT
@misc{pith2026250513418,
author = {Pith},
title = {Pith review of: Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness},
year = {2026},
howpublished = {\url{https://pith.science/paper/XGJIDKOT}},
note = {Machine review of arXiv:2505.13418}
}
read the original abstract
Cognitive decline often surfaces in language years before diagnosis. It is frequently non-experts, such as those closest to the patient, who first sense a change and raise concern. As LLMs become integrated into daily communication and used over prolonged periods, it may even be an LLM that notices something is off. But what exactly do they notice--and should be noticing--when making that judgment? This paper investigates how dementia is perceived through language by non-experts. We presented transcribed picture descriptions to non-expert humans and LLMs, asking them to intuitively judge whether each text was produced by someone healthy or with dementia. We introduce an explainable method that uses LLMs to extract high-level, expert-guided features representing these picture descriptions, and use logistic regression to model human and LLM perceptions and compare with clinical diagnoses. Our analysis reveals that human perception of dementia is inconsistent and relies on a narrow, and sometimes misleading, set of cues. LLMs, by contrast, draw on a richer, more nuanced feature set that aligns more closely with clinical patterns. Still, both groups show a tendency toward false negatives, frequently overlooking dementia cases. Through our interpretable framework and the insights it provides, we hope to help non-experts better recognize the linguistic signs that matter.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[8]
The optimization of a natural language pro- cessing approach for the automatic detection of alzheimer’s disease using gpt embeddings.Brain Sciences, 14(3):211. Douglas W Scharre. 2019. Preclinical, prodromal, and dementia stages of alzheimer’s disease.Pract Neurol, 15:36–47. Ernest S Shtatland, Ken Kleinman, and Emily M Cain
work page 2019
-
[10]
Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.arXiv preprint arXiv:2403.05530. Argonde C van Harten, Michelle M Mielke, Dana M Swenson-Dravis, Clinton E Hagen, Kelly K Edwards, Rosebud O Roberts, Yonas E Geda, David S Knop- man, and Ronald C Petersen. 2018. Subjective cog- nitive decline and risk of mci: The mayo cl...
arXiv 2018
-
[11]
Vimbi Viswan, Noushath Shaffi, Mufti Mahmud, Karthikeyan Subramanian, and Faizal Hajamo- hideen
Interpreting artificial intelligence models: a systematic review on the application of lime and shap in alzheimer’s disease detection.Brain Informatics, 11(1):10. Vimbi Viswan, Noushath Shaffi, Mufti Mahmud, Karthikeyan Subramanian, and Faizal Hajamo- hideen. 2024. Explainable artificial intelligence in alzheimer’s disease classification: A systematic re-...
-
[12]
Lexical-semantic properties of verbs and nouns used in conversation by people with Alzheimer’s disease.PLOS ONE, 18(8):e0288556. Kathryn M. Yorkston and David R. Beukelman. 1980. An Analysis of Connected Speech Samples of Apha- sic and Normal Speakers.Journal of Speech and Hearing Disorders, 45(1):27–36. Jonathan Adrian Zegarra-Valdivia, Myrthe Gwen Ri- j...
work page 1980
-
[1983]
Anat Goren, Carol Swindell, and Arifulla Khan
BDAE: The Boston Diagnostic Aphasia Exam- ination.PA: Lea & Febiger. Anat Goren, Carol Swindell, and Arifulla Khan. 1992. Expressive language characteristics of schizophrenic subjects with different medication histories.Journal of Neurolinguistics, 7(1):67–90. Dianne Gove, Murna Downs, MJFJ Vernooij-Dassen, and Neil Small. 2016. Stigma and gps’ perception...
work page 1992
-
[2002]
One more time about r2 measures of fit in logistic regression.NESUG 15 proceedings, 15:222– 226. John R. Sims, Jennifer A. Zimmer, Cynthia D. Evans, Ming Lu, Paul Ardayfio, JonDavid Sparks, Alette M. Wessels, Sergey Shcherbinin, Hong Wang, Emel Serap Monkul Nery, Emily C. Collins, Paul Solomon, Stephen Salloway, Liana G. Apos- tolova, Oskar Hansson, Craig...
work page 2023
-
[2003]
Reducing stigma and discrimination against older people with mental disorders: a technical con- sensus statement.International journal of geriatric psychiatry, 18(8):670–678. Dean Anthony Granitsas. 2020. All laughter is nervous: An anxiety-based understanding of incongruous hu- mor.HUMOR, 33(4):625–643. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri,...
arXiv 2020
-
[2009]
“out of sight, out of mind”: a qualitative study of visual impairment and dementia from three per- spectives.International psychogeriatrics, 21(3):511– 518. Changye Li, Jacob Solinsky, Trevor Cohen, and Serguei Pakhomov. 2024. A curious case of retrogenesis in language: Automated analysis of language patterns observed in dementia patients and young childr...
arXiv 2024
Show all 13 references
-
[2015]
Catarina Botelho, John Mendonça, Anna Pompili, Tanja Schultz, Alberto Abad, and Isabel Trancoso
Theory of mind in behavioural-variant fron- totemporal dementia and alzheimer’s disease: a meta- analysis.Journal of Neurology, Neurosurgery & Psy- chiatry, 86(7):714–719. Catarina Botelho, John Mendonça, Anna Pompili, Tanja Schultz, Alberto Abad, and Isabel Trancoso. 2024. Ma...
2024 arXiv
-
[2016]
a stool which is about and he he is getting a cookie
Speech recognition in alzheimer’s disease and in its assessment. InInterspeech, volume 2016, pages 1948–1952. A Full Feature List and Prompts Following is the complete list of 38 binary features we defined, divided per category. For each feature, we present its relevant prompt...
1993
-
[2018]
Sunghye Cho, Katheryn Alexandra Quilico Cousins, Sanjana Shellikeri, Sharon Ash, David John Irwin, Mark Yoffe Liberman, Murray Grossman, and Naomi Nevler
Why is my classifier discriminatory?Ad- vances in neural information processing systems, 31. Sunghye Cho, Katheryn Alexandra Quilico Cousins, Sanjana Shellikeri, Sharon Ash, David John Irwin, Mark Yoffe Liberman, Murray Grossman, and Naomi Nevler. 2022. Lexical and acoustic sp...
2022
-
[2023]
Atif Latif and Jihie Kim
Dementiabank: Theoretical rationale, proto- col, and illustrative analyses.American Journal of Speech-Language Pathology, 32(2):426–438. Atif Latif and Jihie Kim. 2024. Evaluation and analysis of large language models for clinical text augmenta- tion and generation.IEEE Access...
2024
-
[2024]
Gordon C Baylis, Leslie L Baylis, and Christopher L Gore
Alzheimer’s disease recognition from spon- taneous speech using large language models.ETRI Journal, 46(1):96–105. Gordon C Baylis, Leslie L Baylis, and Christopher L Gore. 2004. Visual neglect can be object-based or scene-based depending on task representation.Cor- tex, 40(2):...
2004
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.