Pith. sign in

REVIEW 3 major objections 3 minor 13 references

Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read This paper shows that non-expert humans and large language models both miss many clinically diagnosed dementia cases, while the cues they rely on differ: humans use a narrow, sometimes misleading set, and LLMs use a richer set closer to…

desk verdict A genuinely new perception-angle paper whose central human-vs-LLM comparison is compromised by GPT-4o serving as both feature extractor and one of the perception voters. read the letter →

arxiv 2505.13418 v1 pith:XGJIDKOT submitted 2025-05-19 cs.CL cs.LG

classification cs.CLcs.LG
keywords dementiaperceptionlargelanguagemodelsexplainableNLPCookieTheftpicturedescriptionlogisticregressionLLMasannotatorfalsenegatives
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that dementia is perceived differently by non-expert humans and by large language models when both read the same transcribed speech, and that neither group perceives it the way clinicians diagnose it. Using 514 Cookie Theft picture descriptions, the authors show that human judgments are inconsistent and hinge on a few simple cues, some of which point opposite to clinical diagnosis, while LLMs draw on a richer feature set that aligns more closely with clinical patterns. Still, both humans and LLMs miss a large share of clinically diagnosed dementia cases, mostly by labeling them healthy. This matters because early detection often begins with non-experts, and increasingly with LLM assistants, so knowing which cues each group actually uses is a step toward better public awareness and safer automated monitoring.

What carries the argument

The load-bearing mechanism is a four-step explainable pipeline. First, 38 binary features are defined in consultation with a neurologist and a neuropsychologist, grouped into five categories: Objective Interpretation, Subjective Interpretation, Linguistic, Human Experience, and Interview Context. Second, a large language model (GPT-4o) annotates every transcript for all 38 features, after an Alternative Annotator Test showed a 90% chance its annotations were as good as or better than human annotations. Third, stepwise logistic regression models are fit separately to human perception, LLM perception, and clinical diagnosis using the same features. Fourth, the coefficients are compared to see which cues each judgment type actually relies on; the same feature representation across all three makes the comparison interpretable.

What would settle it

Have a fresh group of non-expert humans annotate the same 38 features on a random sample of transcripts, then fit the same logistic regression using those human feature annotations to predict human perception. If the model fit rises well above the reported McFadden's $R^2=0.058$, the conclusion that human perception is inherently inconsistent would be undermined, because the noisy part would be the original LLM feature annotations rather than human judgment.

Watch

Extended reading notes

Core claim

The paper's central claim is that non-expert human perception of dementia in language is inconsistent and driven by a narrow, partly misleading set of cues, whereas LLM perception is driven by a broader set that overlaps more with clinical diagnosis; nevertheless both are biased toward false negatives. Concretely, human annotators agreed only weakly (Fleiss' $\kappa=0.28$), a logistic regression on the 38 expert-guided features explained almost none of their judgments (McFadden's $R^2=0.058$), and the few cues that mattered included reading short sentences as a sign of health even though clinicians associate short sentences with dementia. LLM judgments were better explained by the same features (McFadden's $R^2=0.527$) and drew on subjective and emotional cues such as Theory of Mind, lightheartedness, and sadness. With 283 clinically diagnosed dementia cases, humans correctly identified 57% and LLMs 60%, and among their errors both groups predominantly missed dementia cases (65% and 70% false negatives, respectively). The authors also report that annotators' self-described reasoning did not match the features that actually predicted their judgments.

Load-bearing premise

The load-bearing premise is that the 38 binary features extracted by a single LLM (GPT-4o) faithfully capture the linguistic cues that actually drive human and LLM judgments; the validation of this feature set was carried out on only 10 transcripts.

Editorial extensions

If this is right

  • If non-experts read short sentences as a sign of health, awareness materials can be aimed directly at that specific misreading.
  • Any LLM-based early-warning tool should be designed around the demonstrated false-negative tendency, since 70% of LLM errors were missed dementia cases.
  • If people's self-reported cues do not match their modeled decision patterns, asking users to reflect on their own reasoning is not a reliable way to audit or improve their judgments.
  • If LLMs already use a feature set closer to clinical diagnosis, their explanations could serve as coaching signals to help non-experts attend to clinically relevant cues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-transcript design likely understates how well humans detect decline in real life, where repeated exposure gives a baseline; the narrow-cue result may not transfer to longitudinal monitoring.
  • A practical screening system would need to track changes over time rather than classify individual transcripts, because the LLM false-negative pattern shows a transcript without overt linguistic problems is almost always labeled healthy.
  • If the 38-feature set omits cues humans actually use, the low human model fit could reflect incomplete measurement rather than human inconsistency; re-annotating a larger sample with human feature labels would settle this.
  • The paper does not test whether LLM explanations improve human accuracy, but its results make that a natural next experiment for public-awareness tools.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper investigates how non-expert humans and LLMs perceive dementia from language, using 514 Cookie Theft picture descriptions from the Pitt corpus. The authors collect intuitive healthy/dementia judgments from 27 non-expert annotators and from three LLMs (GPT-4o, LLaMA 3, Gemini-1.5-Pro), define 38 expert-guided binary features, annotate those features with GPT-4o, and fit stepwise logistic regressions to explain human perception, LLM majority perception, and clinical diagnosis. The central claims are that human perception is inconsistent and relies on a narrow and sometimes misleading cue set, that LLMs draw on a richer feature set more aligned with clinical patterns, and that both groups show a tendency toward false negatives. The paper also analyzes misperceptions and self-reported human rationales.

Significance. If the central claims were fully supported, the paper would make a valuable contribution to dementia awareness and to the design of LLM-based monitoring tools. The research question is timely and understudied, the use of a clinically grounded corpus is appropriate, and the authors are transparent about several limitations. They also apply a statistical test for LLM-as-annotator quality and present inherently interpretable models. However, the main human-vs-LLM asymmetry currently rests on a confounded measurement setup and on an interpretation of a very low human model fit that may not be unique. The paper's headline conclusion is therefore not yet established, although it is plausibly fixable with additional analyses.

major comments (3)
  1. [§4.2, §5.2, Table 3] The main evidence for the claim that LLMs rely on a richer feature set is confounded. GPT-4o is both the annotator that produced all 38 feature values for all 514 transcripts (Section 5.2) and one of the three LLMs whose majority vote defines the LLM perception label (Section 4.2). The very high McFadden R² of 0.527 in Table 3 may therefore reflect GPT-4o's internal consistency rather than a genuinely broader or more clinically aligned cue set. I request that the authors re-estimate the LLM perception model using only the LLaMA-3 and Gemini-1.5-Pro majority, and ideally with features annotated by a model that is not part of the perception panel, and report whether the R² and the coefficient pattern survive.
  2. [§7.1, Table 3] The low human-perception fit (McFadden R²=0.058) is interpreted as evidence of human inconsistency, but it is equally consistent with the alternative that the 38 binary features annotated by GPT-4o do not capture the cues humans actually use. The Alternative Annotator Test in Section 5.2 validates agreement between GPT-4o and human annotators on the feature values for 10 transcripts (380 values); it does not establish that this feature vocabulary is complete or causally aligned with human perception. Without a larger human-annotated feature validation, or an analysis using an expanded feature set derived from the human rationales in Section 7.3, the central asymmetry between narrow human cues and rich LLM cues may be an artifact of measurement rather than a substantive finding.
  3. [§6, Table 3] The set of significant features is obtained through stepwise logistic regression on the full dataset, and the p-values in Table 3 are not adjusted for the 38 candidate predictors. Under this procedure, the number of significant features per judgment (4 for humans, 13 for LLMs, 8 for clinical diagnosis) is not a clean measure of cue 'richness,' since stepwise selection is known to be unstable and to produce inflated significance. I ask for a robustness check, such as bootstrap stability of selected features or a regularized alternative like LASSO, before the richness comparison is used as a central result. In addition, the claim that LLMs align 'more closely' with clinical patterns is not quantified; a formal comparison, such as correlation or overlap of coefficient vectors including signs, would make this claim testable.
minor comments (3)
  1. [§7.3] The statement that 65% of self-reported cues align with the predefined feature set is based on a manual review of responses from 18 of 27 annotators, but the coding procedure and coder agreement are not described. Please provide the coding protocol and report inter-coder reliability.
  2. [§6] Please state explicitly in the modeling section that all logistic regressions use the majority-vote perception labels, and clarify whether the clinical diagnosis model is fit on the same feature matrix as the perception models.
  3. [Appendix C.1] Five features are removed because they are positive in fewer than 5% of samples. Since some of these features (e.g., empathy, irritability) are clinically interesting, please report whether the main conclusions change if these features are merged with related categories or analyzed descriptively.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: GPT-4o both annotates the 38 features and contributes to the LLM perception label, so the LLM-side 'richer feature set' claim is partly a self-consistency artifact.

  1. other [Sections 4.2, 5.2, 7.1]
    "GPT-4o, LLaMA 3, and Gemini-1.5-Pro were provided with the same transcripts and also asked to provide their best judgment ... we used their majority vote in our analysis. ... we used GPT-4o to label all 38 binary features across our 514 descriptions ... a very strong fit for LLM dementia perception (0.527), suggesting that our model and features captures a reliable underlying signal."

    The LLM perception label is a majority vote over three LLMs, one of which is GPT-4o, while the 38 features are generated by GPT-4o on the same 514 transcripts. The logistic-regression fit of LLM perception on these features therefore partly measures GPT-4o's internal consistency between its own feature annotations and its own perception vote, rather than an independent property of 'LLM perception.' The inference that LLMs use a 'richer, more nuanced feature set' is thus partially a self-consistency artifact. The human-perception model (R2=0.058) and clinical-diagnosis model (R2=0.209) use labels independent of GPT-4o, so the central comparison retains some independent content, but the LLM-side richness claim is not independently established by this design.

full rationale

The only substantive circularity concern is GPT-4o's dual role: it is the sole annotator of all 38 features and also one of the three models whose majority vote defines LLM perception. This can inflate the LLM model's McFadden R2 and the number of significant features, making the 'LLMs are richer' finding partly an artifact of same-source measurement. The Alternative Annotator Test (Calderon et al., 2025) is a self-citation, but it is a general statistical method applied to this data rather than a dementia-specific result, and it is supplemented by human agreement (kappa=0.557) on 10 transcripts; it is therefore not a load-bearing circular self-citation. The human-perception and clinical-diagnosis models are grounded in independent labels, so the central human-vs-LLM asymmetry is not wholly reduced to a fitting or self-citation chain. However, because the feature representation used for all models is GPT-4o's output, and GPT-4o also contributes to the LLM perception label, the LLM-side conclusions are partially confounded. Score 4 reflects this partial, localized circularity rather than a fully circular derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the feature representation, the reliability of GPT-4o as annotator, the statistical model, and the data labeling choices. These are reasonable but not all independently verified; the model coefficients and significance patterns are fitted outputs, and several cutoffs are chosen by hand.

free parameters (3)
  • Low-frequency feature removal threshold = 5% (fewer than 25 positive samples)
    Five features (vision difficulties, introduction, naming characters, empathy, irritability) were removed from all analyses because they were positive in fewer than 5% of samples (Section C.1). This changes the feature set and thus the reported significant coefficients.
  • Alternative Annotator Test threshold epsilon = 0.1
    The conservative threshold in the statistical test (Calderon et al., 2025) used to select GPT-4o as the feature annotator (Section 5.2). A different threshold could have led to a different annotator model.
  • Stepwise regression significance level = p < 0.05
    Features with p-values below 0.05 were retained as significant in the logistic regression models (Section 6, Table 3). No multiple-comparison correction was applied across the 33 tested features.
assumptions (5)
  • domain assumption The 38 expert-guided binary features capture the linguistically relevant cues in Cookie Theft picture descriptions that drive perceptions of dementia.
    The entire coefficient analysis and the conclusions about which cues humans and LLMs use rest on this representational assumption. Introduced in Section 5.1.
  • domain assumption GPT-4o's binary feature annotations are as reliable as human annotations for all 514 transcripts.
    The Alternative Annotator Test (Calderon et al., 2025) was run on only 10 descriptions (N=380 values) and gives a probabilistic guarantee; the features for the full corpus come from GPT-4o alone (Section 5.2).
  • domain assumption Logistic regression coefficients approximate the cues that drive perceptions.
    Section 6 states that under standard assumptions, particularly with relevant confounders included, coefficients can approximate causal effects; no confounders or causal graph are provided.
  • domain assumption Binarizing MCI and 'Other' diagnoses into the Dementia class is appropriate for perception modeling.
    Section 3.2 collapses all non-healthy labels into Dementia, which mixes MCI and other dementias with AD and may affect feature associations.
  • domain assumption The Pitt corpus Cookie Theft transcripts are representative of dementia-related language.
    The study relies entirely on this single corpus and task (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness." pith.science (2026). https://pith.science/paper/XGJIDKOT

@misc{pith2026250513418,
  author       = {Pith},
  title        = {Pith review of: Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGJIDKOT}},
  note         = {Machine review of arXiv:2505.13418}
}
read the original abstract

Cognitive decline often surfaces in language years before diagnosis. It is frequently non-experts, such as those closest to the patient, who first sense a change and raise concern. As LLMs become integrated into daily communication and used over prolonged periods, it may even be an LLM that notices something is off. But what exactly do they notice--and should be noticing--when making that judgment? This paper investigates how dementia is perceived through language by non-experts. We presented transcribed picture descriptions to non-expert humans and LLMs, asking them to intuitively judge whether each text was produced by someone healthy or with dementia. We introduce an explainable method that uses LLMs to extract high-level, expert-guided features representing these picture descriptions, and use logistic regression to model human and LLM perceptions and compare with clinical diagnoses. Our analysis reveals that human perception of dementia is inconsistent and relies on a narrow, and sometimes misleading, set of cues. LLMs, by contrast, draw on a richer, more nuanced feature set that aligns more closely with clinical patterns. Still, both groups show a tendency toward false negatives, frequently overlooking dementia cases. Through our interpretable framework and the insights it provides, we hope to help non-experts better recognize the linguistic signs that matter.

Figures

Figures reproduced from arXiv: 2505.13418 by the authors.

Figure 1
Figure 1. Illustration of the end-to-end methodological process. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Feature categories, definitions, and examples. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Main results from the logistic regression coefficient analysis, and perception disagreements. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cookie Theft Picture. roundabout way without clearly naming it (e.g., “the thing you use to dry stuff” instead of “towel”). * Over-explaining simple observations (e.g., “and then I guess what she’s doing is maybe she’s hold￾ing something and it looks like it could be” …
Figure 5
Figure 5. Figure 5: Value distribution across features. Features marked as ‘yes’ in fewer than 5% of samples (indicated by the [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Heatmap of feature correlations (Pearson): red indicates positive correlation, and blue indicates negative. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of Pearson correlations between features and Human perceptions, LLM majority-vote perceptions, [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

  1. [8]

    Douglas W Scharre

    The optimization of a natural language pro- cessing approach for the automatic detection of alzheimer’s disease using gpt embeddings.Brain Sciences, 14(3):211. Douglas W Scharre. 2019. Preclinical, prodromal, and dementia stages of alzheimer’s disease.Pract Neurol, 15:36–47. Ernest S Shtatland, Ken Kleinman, and Emily M Cain

  2. [10]

    Argonde C van Harten, Michelle M Mielke, Dana M Swenson-Dravis, Clinton E Hagen, Kelly K Edwards, Rosebud O Roberts, Yonas E Geda, David S Knop- man, and Ronald C Petersen

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.arXiv preprint arXiv:2403.05530. Argonde C van Harten, Michelle M Mielke, Dana M Swenson-Dravis, Clinton E Hagen, Kelly K Edwards, Rosebud O Roberts, Yonas E Geda, David S Knop- man, and Ronald C Petersen. 2018. Subjective cog- nitive decline and risk of mci: The mayo cl...

  3. [11]

    Vimbi Viswan, Noushath Shaffi, Mufti Mahmud, Karthikeyan Subramanian, and Faizal Hajamo- hideen

    Interpreting artificial intelligence models: a systematic review on the application of lime and shap in alzheimer’s disease detection.Brain Informatics, 11(1):10. Vimbi Viswan, Noushath Shaffi, Mufti Mahmud, Karthikeyan Subramanian, and Faizal Hajamo- hideen. 2024. Explainable artificial intelligence in alzheimer’s disease classification: A systematic re-...

  4. [12]

    Kathryn M

    Lexical-semantic properties of verbs and nouns used in conversation by people with Alzheimer’s disease.PLOS ONE, 18(8):e0288556. Kathryn M. Yorkston and David R. Beukelman. 1980. An Analysis of Connected Speech Samples of Apha- sic and Normal Speakers.Journal of Speech and Hearing Disorders, 45(1):27–36. Jonathan Adrian Zegarra-Valdivia, Myrthe Gwen Ri- j...

  5. [1983]

    Anat Goren, Carol Swindell, and Arifulla Khan

    BDAE: The Boston Diagnostic Aphasia Exam- ination.PA: Lea & Febiger. Anat Goren, Carol Swindell, and Arifulla Khan. 1992. Expressive language characteristics of schizophrenic subjects with different medication histories.Journal of Neurolinguistics, 7(1):67–90. Dianne Gove, Murna Downs, MJFJ Vernooij-Dassen, and Neil Small. 2016. Stigma and gps’ perception...

  6. [2002]

    One more time about r2 measures of fit in logistic regression.NESUG 15 proceedings, 15:222– 226. John R. Sims, Jennifer A. Zimmer, Cynthia D. Evans, Ming Lu, Paul Ardayfio, JonDavid Sparks, Alette M. Wessels, Sergey Shcherbinin, Hong Wang, Emel Serap Monkul Nery, Emily C. Collins, Paul Solomon, Stephen Salloway, Liana G. Apos- tolova, Oskar Hansson, Craig...

  7. [2003]

    Dean Anthony Granitsas

    Reducing stigma and discrimination against older people with mental disorders: a technical con- sensus statement.International journal of geriatric psychiatry, 18(8):670–678. Dean Anthony Granitsas. 2020. All laughter is nervous: An anxiety-based understanding of incongruous hu- mor.HUMOR, 33(4):625–643. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri,...

  8. [2009]

    out of sight, out of mind

    “out of sight, out of mind”: a qualitative study of visual impairment and dementia from three per- spectives.International psychogeriatrics, 21(3):511– 518. Changye Li, Jacob Solinsky, Trevor Cohen, and Serguei Pakhomov. 2024. A curious case of retrogenesis in language: Automated analysis of language patterns observed in dementia patients and young childr...

Show all 13 references
  1. [2015]

    Catarina Botelho, John Mendonça, Anna Pompili, Tanja Schultz, Alberto Abad, and Isabel Trancoso

    Theory of mind in behavioural-variant fron- totemporal dementia and alzheimer’s disease: a meta- analysis.Journal of Neurology, Neurosurgery & Psy- chiatry, 86(7):714–719. Catarina Botelho, John Mendonça, Anna Pompili, Tanja Schultz, Alberto Abad, and Isabel Trancoso. 2024. Ma...

  2. [2016]

    a stool which is about and he he is getting a cookie

    Speech recognition in alzheimer’s disease and in its assessment. InInterspeech, volume 2016, pages 1948–1952. A Full Feature List and Prompts Following is the complete list of 38 binary features we defined, divided per category. For each feature, we present its relevant prompt...

  3. [2018]

    Sunghye Cho, Katheryn Alexandra Quilico Cousins, Sanjana Shellikeri, Sharon Ash, David John Irwin, Mark Yoffe Liberman, Murray Grossman, and Naomi Nevler

    Why is my classifier discriminatory?Ad- vances in neural information processing systems, 31. Sunghye Cho, Katheryn Alexandra Quilico Cousins, Sanjana Shellikeri, Sharon Ash, David John Irwin, Mark Yoffe Liberman, Murray Grossman, and Naomi Nevler. 2022. Lexical and acoustic sp...

  4. [2023]

    Atif Latif and Jihie Kim

    Dementiabank: Theoretical rationale, proto- col, and illustrative analyses.American Journal of Speech-Language Pathology, 32(2):426–438. Atif Latif and Jihie Kim. 2024. Evaluation and analysis of large language models for clinical text augmenta- tion and generation.IEEE Access...

  5. [2024]

    Gordon C Baylis, Leslie L Baylis, and Christopher L Gore

    Alzheimer’s disease recognition from spon- taneous speech using large language models.ETRI Journal, 46(1):96–105. Gordon C Baylis, Leslie L Baylis, and Christopher L Gore. 2004. Visual neglect can be object-based or scene-based depending on task representation.Cor- tex, 40(2):...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.