REVIEW 3 major objections 6 minor 26 references
Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Fifteen explainable features outperform the 40-feature linguistic baseline in Alzheimer's screening.
desk verdict A genuinely new LLM-vision feature-generation idea for AD screening, but the headline comparison is undermined by label-derived TF-IDF features and a 48-sample test set with no significance testing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the 15-dimensional feature vector itself, with two novel components: (1) content-coverage features generated by prompting a multimodal LLM (GPT-4o) with cropped sub-images of the Cookie Theft picture to produce topic keyword sets and full descriptions, scored by keyword hit rate, BLEU, and METEOR; and (2) TF-IDF reference-vector similarity features, where per-subject transcripts are embedded in TF-IDF space and compared by cosine similarity to an HC reference vector and an AD reference vector, each built by averaging training-document vectors within a class. The TF-IDF keyword hit rate counts how many of the top-30 HC-reference terms appear in a subject's transcript. These features make the content of what a speaker says—not just how they say it—the signal for screening.
What would settle it
Recompute the three TF-IDF-based features (similarity to HC, similarity to AD, and keyword hit rate) under nested cross-validation or with reference vectors built from a held-out development set, so that no test transcript's label contributes to the reference vectors; if the accuracy gap over the 40-feature baseline shrinks or disappears, the central claim depends on target encoding.
Extended reading notes
Core claim
The central claim is that the 15 new features consistently outperform 40 traditional linguistic features across two classifiers (Random Forest and XGBoost) on the ADReSS benchmark. The best result, 85.4% accuracy, is achieved by the new features alone in a random forest. The feature set comprises three LLM-generated topic keyword hit rates, averaged BLEU-1 through BLEU-4 and METEOR scores against 15 LLM-generated golden descriptions, two TF-IDF cosine similarities to HC and AD reference vectors, a TF-IDF keyword hit rate, plus averaged parse tree depth, filled pause count and ratio, and word error rate. The authors argue the features are explainable because each step—picture segmentation, keyword generation, reference construction—is traceable.
Load-bearing premise
The load-bearing premise is that constructing the HC and AD reference vectors from the training labels and then training a classifier on those same labels is a legitimate feature-engineering step rather than a form of target leakage; if the TF-IDF similarity features merely encode the training labels, the reported consistent outperformance would not hold on unseen data.
Editorial extensions
If this is right
- A 15-dimensional feature vector can serve as a lightweight, interpretable alternative to fine-tuned BERT for AD screening, at roughly a tenth of the feature count.
- Naively concatenating the new features with traditional linguistic features can hurt accuracy, so feature selection is needed when combining feature families.
- Because the features are tied to the Cookie Theft task, they transfer to other picture-description cognitive assessments only if equivalent content-coverage pipelines are built for those stimuli.
- The TF-IDF reference vectors give a quantitative measure of how a speaker's language resembles the prototypical healthy or AD profile, which may help clinicians see why a screening decision was made.
Reading between the lines
- If the TF-IDF reference vectors are a form of target encoding, then the feature-importance ranking (TF-IDF similarity HC in the top five) will not survive a nested cross-validation evaluation; testing that is the direct next step.
- The LLM-generated references are a substitute for human gold-standard descriptions; replacing them with human references in the same pipeline could isolate how much of the gain comes from the LLM versus from the content-coverage idea itself.
- The topic keyword sets are LLM-version-dependent; scores may drift across GPT-4o versions, so a reproducibility check with another multimodal LLM or a fixed LLM version would clarify how much of the result is tied to the specific model.
- Because the ADReSS set is small (156 samples), the reported margins may not be stable; a paired bootstrap or confidence interval on the accuracy difference would show whether the gap over the baseline is robust.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a compact, explainable 15-dimensional feature set for Alzheimer's disease (AD) screening from Cookie Theft picture descriptions. The features include LLM-generated topic keyword hit rates, average BLEU and METEOR scores against LLM-generated golden-standard descriptions, TF-IDF similarity to class-specific reference vectors, a TF-IDF keyword hit rate, and four conventional linguistic features (parse-tree depth, filled pauses, filled-pause ratio, and word error rate). On the ADReSS 2020 challenge data, the authors report 85.4% accuracy with random forest and 83.3% with XGBoost using the new features, compared with 75.0% and 72.9% for a 40-feature linguistic baseline, and conclude that the new features consistently outperform traditional linguistic features while being more interpretable and dimensionally efficient.
Significance. If the reported performance is reliable, a 15-dimensional, explainable feature set that approaches fine-tuned BERT accuracy on ADReSS would be a useful contribution to explainable AD screening. The use of a multimodal LLM for content-coverage features is creative, and the proposed features are more transparent than black-box embeddings. The paper also provides a clear feature-generation pipeline with prompts and examples, which facilitates reproducibility. However, the headline claim of consistent outperformance is currently not statistically supported, and the TF-IDF reference features are constructed from training labels in a way that risks target-encoding leakage. No code or data are provided, so the results cannot be independently checked. These issues must be addressed before the central claim can be accepted.
major comments (3)
- [Section 2.2.2, Eqs. (5)-(7)] The HC and AD reference vectors vHC and vAD are computed by averaging TF-IDF vectors over training documents grouped by their labels. Consequently, the features 'TF-IDF similarity HC', 'TF-IDF similarity AD', and 'TF-IDF Keywords Hit Rate' are derived from the training labels. When the classifier is trained on these features, it receives a target-encoded representation, and the Bayesian optimization with cross-validation described in Section 3.1 will produce optimistic validation scores whenever validation-fold subjects contribute to the reference vectors. The paper must either compute the reference vectors in a strictly leave-one-out/out-of-fold fashion for every training and validation prediction, or justify why this is unnecessary. As written, the comparison against label-free linguistic features is not a fair test of the new feature set.
- [Table 3 and Section 3.2] The reported accuracy differences (85.4% vs 75.0% for RF; 83.3% vs 72.9% for XGBoost) are based on a single 48-sample test set with no confidence intervals, bootstrap resampling, or paired significance testing. On n=48, the 10.4-point gap corresponds to roughly 5 correct classifications, and an exact McNemar test on the discordant pairs would not reach p<0.05. The paper should provide paired significance tests (e.g., McNemar) or confidence intervals for all metrics, and the abstract's 'consistently outperform' claim should be tempered unless such evidence is provided.
- [Section 4.2 and Figure 3] The ablation study adds features in order of ANOVA F-values computed on the training set. Since the TF-IDF reference vectors already encode training labels, the ANOVA F-values for those features are inflated by the same target-encoding mechanism. The non-monotonic accuracy curve in Figure 3 is therefore not a reliable guide to feature importance, and the conclusion that 'feature selection based on ANOVA F-values may not be suitable' is confounded by the leakage in the feature construction. An ablation that separates the label-derived features from the LLM-based content features would be more informative.
minor comments (6)
- [Abstract] The phrase 'due to their scalability due to their scalability' is duplicated and should be corrected.
- [Table 1] The table title 'Fifteen proposed features description' is inconsistent with the text: Section 2.2 states that 11 new features are proposed, and the remaining four are conventional linguistic features added from prior work. Please clarify the composition of the 15-feature set.
- [Introduction] The claim 'we pioneered the breakdown of the Cookie Theft picture' overstates novelty; a more cautious phrasing such as 'we propose' or 'we introduce' would be appropriate.
- [Section 2.2.2] There are several typos and grammatical errors in this section, including 'calculating the the number', 'coresponds', and 'evalaution' in the footnote. Equation (4) is also difficult to parse in the provided text and should be typeset more clearly.
- [References] Reference [3] is malformed ('V . L, R. SH, R. M, P. M, L. J, C. M, and L. G.'), and several other references (e.g., [9], [19]) lack full publication details. Please ensure all references follow a consistent, complete format.
- [Section 2.2.2, WER feature] The WER feature uses Whisper-large-v3, but it is not specified whether the word error rate is computed against the manual transcripts of the same audio or against some other reference. Please clarify the reference transcript used.
Circularity Check
No circularity found; the reported ADReSS test-set results are out-of-sample and not forced by construction.
full rationale
The paper's central empirical claim is evaluated on the fixed ADReSS train/test split, which is an external benchmark with a separate 48-sample test set. The TF-IDF reference vectors in Eqs. (5)-(7) are computed only from training-set documents and their training labels; test transcripts are not included in the reference-vector averages and their labels are not used to extract the test features. The two TF-IDF similarity features are therefore supervised embeddings, but the final accuracy numbers are genuine out-of-sample predictions rather than identities obtained by construction. The topic, BLEU/METEOR, and linguistic features are also label-free. Self-citations to the authors' prior work appear only as background context in the introduction and are not used to justify the derivation of the new features or to rule out alternatives. The absence of significance testing and the potential target-encoding risk in hyperparameter selection are legitimate methodological concerns about robustness and fair comparison, but they are not circularity as defined by the requested patterns: no quoted equation reduces a predicted quantity to a fitted input, and no load-bearing argument rests on a self-citation. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- TF-IDF class reference vectors (vHC, vAD) =
Computed from training set via Eq. 5; not released
- Top-30 TF-IDF keyword set =
Top 30 terms in vHC selected by authors
- Three-topic segmentation =
boy/girl; mother/sink; window/curtain
- LLM-generated keyword sets and 15 reference descriptions =
Generated by GPT-4o, 2024-05-13 with unspecified sampling parameters
assumptions (3)
- domain assumption ADReSS labels are accurate ground truth for AD status
- ad hoc to paper GPT-4o's visual descriptions of the Cookie Theft picture are a valid golden standard for content coverage
- ad hoc to paper The fixed three-topic decomposition captures the clinically relevant content of the picture
Cite this review
Pith. "Pith review of Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease." pith.science (2026). https://pith.science/paper/6PLFR4UF
@misc{pith2026241118922,
author = {Pith},
title = {Pith review of: Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease},
year = {2026},
howpublished = {\url{https://pith.science/paper/6PLFR4UF}},
note = {Machine review of arXiv:2411.18922}
}
read the original abstract
Alzheimer's disease (AD) has become one of the most significant health challenges in an aging society. The use of spoken language-based AD detection methods has gained prevalence due to their scalability due to their scalability. Based on the Cookie Theft picture description task, we devised an explainable and effective feature set that leverages the visual capabilities of a large language model (LLM) and the Term Frequency-Inverse Document Frequency (TF-IDF) model. Our experimental results show that the newly proposed features consistently outperform traditional linguistic features across two different classifiers with high dimension efficiency. Our new features can be well explained and interpreted step by step which enhance the interpretability of automatic AD screening.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Alzheimer’s disease (AD) detection presents a significant and growing challenge to healthcare and economic systems due to costly and complex diagnoses [1, 2, 3]. Current research un- derscores the importance of early intervention and the need for economically accessible, non-invasive and affordable alterna- tives for AD detection [4, 5, 6, 7]...
work page Pith review arXiv 2024
-
[2]
Method 2.1. Dataset The dataset utilized in this study is derived from the ADReSS Challenge 2020 [17], which represents a curated subset of the Pitt Corpus within the DementiaBank database [18]. It com- prises 156 speech samples and their corresponding transcripts from English-speaking participants engaged in the Cookie Theft picture description task. The...
work page 2020
-
[3]
Experiment 3.1. Experiment settings We constructed classifiers based on two widely recognized methods: Random Forest (RF) and XGBoost. To ensure op- timal performance, we employed Bayesian Optimization [20] to determine the appropriate set of hyperparameters for each model. The hyperparameters identified through this process were kept fixed across all set...
-
[4]
Discussion 4.1. Feature Importance and ANOV A F-values To further substantiate the effectiveness of our features, we ex- tracted the feature importance from the RF model. Fig. 2(a) presents the top 15 important features in the RF. Notably, four of our new features ranking in the top fifteen. Additionally, we plotted the top 15 features with the highest AN...
-
[5]
Conclusion In conclusion, we have proposed a compact set of features that are both more explainable and more effective for AD detection. We introduced the concept of leveraging TF-IDF alongside ad- vanced LLMs’ viusal processing ability to generate useful fea- tures. Our experiments demonstrate that our new features out- perform the traditional features a...
-
[6]
Acknowledgements This work is supported by the HKSARG Research Grants Council’s Theme-based Research Grant Scheme (Project No.T45- 407/19N) and the CUHK Stanley Ho Big Data De- cision Research Centre
-
[7]
Comprehensive review on alzheimer’s disease: causes and treatment,
Z. Breijyeh and R. Karaman, “Comprehensive review on alzheimer’s disease: causes and treatment,” Molecules, vol. 25, no. 24, p. 5789, 2020
work page 2020
-
[8]
An estimate of the to- tal worldwide societal costs of dementia in 2005,
A. Wimo, B. Winblad, and L. J ¨onsson, “An estimate of the to- tal worldwide societal costs of dementia in 2005,” Alzheimer’s & Dementia, vol. 3, no. 2, pp. 81–91, 2007
work page 2005
Show all 26 references
-
[9]
Review of brief cognitive tests for patients with suspected dementia,
V . L, R. SH, R. M, P. M, L. J, C. M, and L. G., “Review of brief cognitive tests for patients with suspected dementia,” Int Psychogeriatr . 2014 Aug;26(8):1247-62. doi:, vol. 10., 2014
2014
-
[10]
Diagnosis and management of dementia,
Z. Arvanitakis, R. C. Shah, and D. A. Bennett, “Diagnosis and management of dementia,”Jama, vol. 322, no. 16, pp. 1589–1599, 2019
2019
-
[11]
Dig- ital biomarkers for alzheimer’s disease: the mobile/wearable de- vices opportunity,
L. C. Kourtis, O. B. Regele, J. M. Wright, and G. B. Jones, “Dig- ital biomarkers for alzheimer’s disease: the mobile/wearable de- vices opportunity,” NPJ digital medicine, vol. 2, no. 1, p. 9, 2019
2019
-
[12]
Speaking in alzheimer’s dis- ease,
S. G, H. I, V . V , K. J, and P. M., “Speaking in alzheimer’s dis- ease,” is That an Early Sign? Importance of Changes in Language Abilities in Alzheimer’s Disease. Front Aging Neurosci. 2015 Oct 20;7:195. doi:, vol. 10., 2015
2015
-
[13]
Speech reveals future risk of developing dementia: Predictive dementia screening from biographic interviews,
J. Weiner, C. Frankenberg, J. Schr ¨oder, and T. Schultz, “Speech reveals future risk of developing dementia: Predictive dementia screening from biographic interviews,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 674–681
2019
-
[14]
Linguis- tic features and automatic classifiers for identifying mild cogni- tive impairment and dementia,
L. Calz `a, G. Gagliardi, R. R. Favretti, and F. Tamburini, “Linguis- tic features and automatic classifiers for identifying mild cogni- tive impairment and dementia,” Computer Speech & Language , vol. 65, p. 101113, 2021
2021
-
[15]
Cross-lingual features for alzheimer’s dementia detection from speech
T. Melistas, L. Kapelonis, N. Antoniou, P. Mitseas, D. Sgouropou- los, T. Giannakopoulos, A. Katsamanis, S. Narayanan, and N. Demokritos, “Cross-lingual features for alzheimer’s dementia detection from speech.”
-
[16]
Careful whisper– leveraging advances in automatic speech recognition for robust and interpretable aphasia subtype classification,
L. Wagner, M. Zusag, and T. Bloder, “Careful whisper– leveraging advances in automatic speech recognition for robust and interpretable aphasia subtype classification,” arXiv preprint arXiv:2308.01327, 2023
2023 arXiv
-
[17]
A comparative study of acoustic and linguistic fea- tures classification for alzheimer’s disease detection,
J. Li, J. Yu, Z. Ye, S. Wong, M. Mak, B. Mak, X. Liu, and H. Meng, “A comparative study of acoustic and linguistic fea- tures classification for alzheimer’s disease detection,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP...
2021
-
[18]
Exploiting prompt learning with pre-trained language models for alzheimer’s disease detection,
Y . Wang, J. Deng, T. Wang, B. Zheng, S. Hu, X. Liu, and H. Meng, “Exploiting prompt learning with pre-trained language models for alzheimer’s disease detection,” inICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023...
2023
-
[19]
Leveraging pretrained representations with task-related key- words for alzheimer’s disease detection,
J. Li, K. Song, J. Li, B. Zheng, D. Li, X. Wu, X. Liu, and H. Meng, “Leveraging pretrained representations with task-related key- words for alzheimer’s disease detection,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IE...
2023
-
[20]
The influence of au- tomatic speech recognition on linguistic features and automatic alzheimer’s disease detection from spontaneous speech,
J. Heitz, G. Schneider, and N. Langer, “The influence of au- tomatic speech recognition on linguistic features and automatic alzheimer’s disease detection from spontaneous speech,” in Pro- ceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Lang...
2024
-
[21]
Text dialogue analysis for pri- mary screening of mild cognitive impairment: Development and validation study,
C. Wang, S. Liu, A. Li, and J. Liu, “Text dialogue analysis for pri- mary screening of mild cognitive impairment: Development and validation study,” Journal of Medical Internet Research , vol. 25, p. e51501, 2023
2023
-
[22]
Can llms like gpt-4 outper- form traditional ai tools in dementia diagnosis? maybe, but not today,
Z. Wang, R. Li, B. Dong, J. Wang, X. Li, N. Liu, C. Mao, W. Zhang, L. Dong, J. Gao et al. , “Can llms like gpt-4 outper- form traditional ai tools in dementia diagnosis? maybe, but not today,” arXiv preprint arXiv:2306.01499, 2023
2023 arXiv
-
[23]
Alzheimer’s dementia recognition through spontaneous speech,
S. Luz, F. Haider, S. de la Fuente Garcia, D. Fromm, and B. MacWhinney, “Alzheimer’s dementia recognition through spontaneous speech,” Frontiers in computer science , vol. 3, p. 780169, 2021
2021
-
[24]
The natural history of alzheimer’s disease: description of study cohort and accuracy of diagnosis,
J. T. Becker, F. Boiler, O. L. Lopez, J. Saxton, and K. L. McGo- nigle, “The natural history of alzheimer’s disease: description of study cohort and accuracy of diagnosis,” Archives of neurology , vol. 51, no. 6, pp. 585–594, 1994
1994
-
[25]
C. D. Manning, P. Raghavan, and H. Sch ¨utze, Introduction to In- formation Retrieval. Cambridge University Press, 2008
2008
-
[26]
Efficient global op- timization of expensive black-box functions,
D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient global op- timization of expensive black-box functions,” Journal of Global optimization, vol. 13, pp. 455–492, 1998
1998
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.