{"id":"72588d84-d86d-49f2-8408-4ca93cae59ad","arxiv_id":"2608.07378","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LSEAD reports 90.0% accuracy for AD versus control classification using local LLM embeddings, PCA, and logistic regression on combined ADReSS20 and ADReSSo2021 test data, and claims about 5% gains over prior published results.","lead":"This paper evaluates LSEAD, a speech-based Alzheimer's screening pipeline that combines Whisper transcription, Zephyr-7B-beta text embeddings, PCA, and logistic regression on two public benchmark datasets. A generalist might read it to see whether locally deployed large language models can detect early cognitive decline without sending patient speech to the cloud.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90.0% headline is selected after evaluating four classifiers and a PCA variance grid on the same 119-sample test set; without nested validation the number is not an unbiased performance estimate, so the SOTA comparison is not established.","rationale":"The paper's central claim is an empirical benchmark result, and its most load-bearing component is that the 90.0% accuracy is a trustworthy estimate of LSEAD's performance on the combined ADReSS20 and ADReSSo2021 test set. The paper evaluates four classifiers on the same test set and then selects the best (LR), and it grid-searches the PCA variance threshold without revealing the chosen value or whether selection used the test set. Because the test set has only 119 samples, even a few misclassifications move the headline by several percentage points. This means the reported 90.0% is an upper bound of the selection procedure rather than an unbiased expected performance, and the claimed 5% improvement over prior work is not reliable. The reader identified related issues (no error bars, unreported PCA threshold, no nested validation) in the rationale, but did not make test-set-driven model selection the central concern; the reader's stated weakest assumption concerned acoustic versus textual signal content. I partially agree with the reader because that content assumption is also relevant, but the more immediate threat to the claim is evaluation validity. A nested cross-validation check would settle the question directly. The paper has strengths: a locally deployable, open-source pipeline is a meaningful contribution; cross-dataset experiments are useful; and independent ASR comparison is a reasonable robustness check. But without a valid performance estimate, the 'state-of-the-art' and 'consistently outperforms existing approaches' statements are not supported. The reader's CONDITIONAL verdict is therefore the right category; my concern refines the conditions under which the paper should be accepted, and does not require changing the verdict label.","tokens_in":19274,"tokens_out":8683,"duration_ms":79544,"concrete_test":"Rerun the complete pipeline with nested cross-validation on the combined training set: for each outer fold, select the classifier and PCA variance threshold using only the inner training fold, then evaluate the chosen configuration on the held-out outer fold. Report mean accuracy with a 95% confidence interval for the full test set. If the selected classifier varies across folds or the mean accuracy falls materially below 90.0%, the headline is an artefact of test-set peeking and the SOTA claim does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is that the headline 90.0% is produced by test-set selection, not by a pre-specified evaluation protocol. In Section 4.2/Table 2, four classifiers (NNs, SVC, XGBoost, LR) are all evaluated on the combined 119-sample test set, and the best test-set result (LR, 90.0%) is cited as the method's performance. The PCA retention threshold in Section 3.2.4/Eq. (13) is also chosen by grid search over {0.9, 0.95, 0.97, 0.99, 0.999}, with neither the selected value nor the data used for selection reported. With 119 test samples, one sample is roughly 0.84 percentage points, so the reported 5.1% margin over Mortensen et al. corresponds to about six samples. The benchmark comparison in Table 3 additionally mixes a combined-dataset result with published per-dataset results, so 'outperforms existing approaches' is not established even if the combined number is reproducible. This is not an accusation of misconduct; it is a question of whether the central empirical claim is an unbiased estimate or an artifact of unblinded model selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LSEAD, a privacy-preserving speech-based framework for early Alzheimer's disease (AD) screening. Raw audio is transcribed with an ASR system (Whisper or Wav2Vec 2.0), transcript-level embeddings are extracted from the penultimate layer of locally deployed open-source LLMs (Zephyr-7B-beta, with Llama 2 and Qwen3-30B as baselines), PCA reduces dimensionality, and a downstream classifier (logistic regression, SVC, XGBoost, or a neural network) performs binary AD/CN classification. The method is evaluated on ADReSS20 and ADReSSo2021 under a combined setting and cross-dataset transfer settings. The central claim is that this pipeline achieves 90.0% accuracy on the combined test set, outperforming published methods by up to about 5 percentage points, while keeping all data processing on-premises.","tokens_in":19520,"tokens_out":3222,"duration_ms":30175,"significance":"If the 90.0% accuracy claim were supported by an unbiased evaluation, this would be a practically useful contribution: it combines open-source, locally deployable components, addresses an important privacy constraint in clinical deployment, and shows systematic robustness across two ASR systems and two datasets. The cross-dataset experiments and the comparison of multiple LLM backbones are also useful empirical evidence. However, the headline performance is currently not an unbiased estimate because the best classifier and the PCA variance threshold appear to be selected using the test set; the comparison with prior work also mixes incompatible evaluation protocols. The paper has strengths in scope and reproducibility (a code link is provided) but needs substantial additional experimental rigor before its central claims can be accepted.","major_comments":[{"comment":"The headline 90.0% accuracy is the best result among four classifiers evaluated on the same 119-sample combined test set, and the PCA variance threshold V_a is selected by a grid search over {0.9, 0.95, 0.97, 0.99, 0.999} without reporting the selected value or reserving a validation split for this choice. Because each test sample is approximately 0.84 percentage points, the reported 5.1% margin over Mortensen et al. corresponds to about six samples. To support the claim, the authors should use nested cross-validation or a pre-specified protocol in which model selection (classifier and V_a) is performed only on training folds, and should report the selected V_a and K as well as confidence intervals or a significance test for the 90.0% accuracy.","section":"§4.2 (Table 2) and §3.2.4 (Eq. 13)"},{"comment":"The comparative claim 'consistently outperforms existing approaches' is not established because the comparison mixes a combined-dataset result (LSEAD, 90.0%) with published per-dataset results from different protocols. The differences may reflect evaluation setup rather than algorithmic superiority. The authors should either re-implement the baselines under the same combined training/test protocol, or restrict the comparison to published results obtained under the same protocol, and state explicitly which protocol each row follows.","section":"§4.4 (Table 3)"},{"comment":"The paper asserts that removing PCA 'consistently degrades performance across all models' and that PCA is 'a crucial step in the proposed framework,' but no results for the no-PCA condition are reported in any table or figure. Given that the PCA variance threshold is a free parameter and that the performance of the downstream classifier depends on K, this assertion is not currently supported by the evidence. The authors should include a table or plot comparing classifiers with and without PCA, using a fixed model-selection protocol.","section":"§4.2 and §3.2.4"},{"comment":"There are citation and identification errors that affect the comparison. In Table 3, both Mortensen et al. and Bang et al. are cited as [36], while the text cites Mortensen et al. as [61]; the Zephyr-7B-beta model is introduced with citation [44] (the GPT-3 paper) rather than the Zephyr paper [65]; and 'Kheirkhahzadeh et al. [62]' refers to a single-author work. These errors need to be corrected, especially because the comparison with locally deployable LLMs (Mortensen et al./ADetectoLocum) is central to the contribution.","section":"§4.4 (Table 3) and §2 (references [36], [61], [44], [65])"}],"minor_comments":[{"comment":"The notation is used inconsistently: 'V AD' appears with a space in Eq. (3), 'ADReSSo21' and 'ADReSSo2021' are both used, and 'Wav2Vec 2.0' is written as 'Wav2Vec2.0' in Table 6. The text should be unified.","section":"Throughout"},{"comment":"The 5-fold cross-validation rows report only point estimates; standard deviations or confidence intervals over folds and over repeated runs would help assess the stability of the classifier ranking, especially given the small dataset.","section":"Table 2"},{"comment":"The PCA variance threshold is described as determined by grid search 'based on the suggested PCA variance [0.9,0.95,0.97,0.99,0.999] [51]'; the reference is a lecture note and the selection criterion is not defined. The authors should specify the objective used to choose among these values (e.g., validation accuracy) and report the chosen value.","section":"§3.2.4"},{"comment":"The means and standard deviations for MMSE scores of true positives and false negatives are reported without sample sizes or confidence intervals; reporting counts per MMSE band would make the early-detection analysis more informative.","section":"§4.3 (Figure 3)"},{"comment":"The conclusions state that the framework 'consistently achieves the best performance' and 'strongly generalizes' across datasets, but the cross-dataset results in Tables 7 and 8 also appear to involve selecting the best classifier on the target test set; this should be acknowledged or the protocol should be clarified.","section":"§5 (Conclusions)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is plausible and the framework is relevant to the journal's readership, but the evaluation protocol needs to be fixed before the paper can be accepted: model selection must be separated from test-set evaluation, the PCA variance threshold must be reported and selected without test-set information, and the comparison with prior methods must be placed on a common evaluation footing. I did not see evidence of deliberate data manipulation, but the reporting as it stands overstates the reliability of the 90.0% result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, incremental engineering paper with one statistically sloppy load-bearing result. The pipeline — Whisper transcription, Zephyr embeddings, mean pooling, PCA, logistic regression — is reasonable, and the authors do more systematic component comparisons than most papers in this niche. But the 90.0% headline was produced by evaluating four classifiers and a grid of PCA variance thresholds on the same 119-sample test set and taking the best result. That makes the number an upper bound, not an unbiased estimate, and the 5-point margin over Mortensen et al. could easily be selection noise.\n\nWhat is genuinely useful here is the breadth of the evaluation. The paper compares Whisper against Wav2Vec 2.0, Zephyr against Llama 2 and Qwen3-30B, and reports cross-dataset transfer in both directions. The cross-dataset numbers (87.4% and 87.3% for LR) are less vulnerable to the test-set selection critique, since the model is trained on one dataset and tested on the other, though the same classifier-selection issue applies within those tables. The authors are also honest about the limitations: small datasets, text-only pipeline, no interpretability. That counts for something.\n\nThe soft spots, in order of seriousness. First, the headline result has no confidence interval or significance test; with 119 test samples, each sample is about 0.84 percentage points, so the 5.1% margin over Mortensen is roughly six samples. Second, the PCA variance threshold is chosen by grid search over the test set, and the selected value is not reported. Third, the code is promised but not shipped, which makes it impossible to verify the pipeline. Fourth, there are citation errors — Mortensen et al. is listed as reference [36] in Table 3 but appears as [61] in the text — and the table mixes a combined-dataset result with per-dataset results from the literature, so the 'outperforms existing approaches' claim is not yet established. These are all fixable.\n\nI disagree with the harshest reading that the whole approach is circular. The embeddings are from a pretrained frozen LLM, the classifier is linear, and the cross-dataset results are real generalization checks. The core idea is not at fault; the evaluation protocol is.\n\nWho is this for? Researchers working on speech biomarkers for dementia, especially those interested in local deployment. The paper is a useful data point, not a decisive one. I would send it to peer review — a good referee can force the authors to re-analyze with nested validation and report the selected threshold — but I would not cite it in its current form.\n\nRecommendation: major revision. The experiments should be redone with a proper inner/outer loop, or at minimum the model selection should be constrained and reported.","headline":"Useful incremental engineering, but the 90% headline is an unblinded test-set selection bound; the cross-dataset study is the more solid part.","tokens_in":20112,"tokens_out":3018,"would_cite":false,"duration_ms":25904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LSEAD claims that a fully local, text-only pipeline—Whisper transcription, Zephyr-7B-β mean-pooled embeddings, PCA, and logistic regression—reaches 90.0% accuracy on the combined ADReSS20 and ADReSSo2021 Alzheimer's speech test set…","keywords":["Alzheimer's disease","speech analysis","large language models","privacy-preserving healthcare","ADReSS20","ADReSSo2021","Whisper","logistic regression"],"falsifier":"Run an acoustic-only classifier using prosody, pause structure, and voice quality on the same combined ADReSS20 and ADReSSo2021 test set; if acoustics alone matches or exceeds the reported 90.0% accuracy, the paper's text-only premise is false. Alternatively, report a 95% confidence interval for the 119-sample test accuracy; a confidence interval spanning the 84.9% comparator would mean the headline margin is not statistically established.","tokens_in":19056,"feed_emoji":"🧠","tokens_out":6006,"duration_ms":50239,"temperature":0.7,"pith_summary":"This paper tries to establish that a privacy-preserving, text-only pipeline can detect Alzheimer's disease from spontaneous speech as accurately as any published method on two standard benchmarks. The proposed LSEAD system transcribes Cookie Theft picture descriptions with Whisper, embeds the transcripts with the open-source Zephyr-7B-β language model, compresses the embeddings with PCA, and classifies with logistic regression. On the combined ADReSS20 and ADReSSo2021 test sets it reports 90.0% accuracy and 89.7% F1, about five points above the strongest published comparator. Because every component can run on local infrastructure without sending patient recordings or transcripts to external services, the claim, if true, would make state-of-the-art linguistic screening compatible with clinical privacy requirements.","feed_headline":"Words alone spot Alzheimer's at 90 percent","feed_subtitle":"A local 7B language model reads Cookie Theft transcripts and beats cloud-based baselines without sending patient data off-site.","key_machinery":"The load-bearing mechanism is the combination of attention-mask-aware mean pooling of the penultimate-layer hidden states of Zephyr-7B-β, PCA dimensionality reduction, and logistic regression. Mean pooling turns variable-length transcripts into a fixed vector while ignoring padding tokens; PCA removes redundant dimensions so a linear classifier can find separating directions; logistic regression makes the final AD-versus-CN decision. The paper argues that this specific chain, rather than the largest model, is what yields the reported accuracy, since removing PCA degrades all classifiers and both Llama 2-7B and Qwen3-30B underperform Zephyr-7B-β in the same pipeline.","core_discovery":"The central claim is that diagnostic information in Alzheimer's speech is recoverable from the lexical-semantic content of automatically transcribed text, and that a modest open-source LLM embedding plus PCA plus a linear classifier extracts it better than larger or cloud-based models. On the combined benchmark, logistic regression achieves 90.0% accuracy (precision 91.2%, recall 88.1%, F1 89.7%), surpassing the 84.9% of the best compared method, with cross-dataset transfer between ADReSS20 and ADReSSo2021 staying near 87% in both directions, and Whisper outperforming Wav2Vec 2.0 transcripts. The paper further claims that the embedding, not model scale, drives performance: Zephyr-7B-β beats Qwen3-30B (87.4%) and Llama 2-7B (83.2%) within the same pipeline. It also claims early-stage sensitivity, with correctly classified AD subjects averaging MMSE 19.4±7.3 and a substantial share of true positives falling in the mild impairment range of 19–23.","pith_inferences":["Going beyond the paper, an immediate testable extension is to report bootstrap confidence intervals around the 90.0% figure; with only 119 test samples the 5.1-point gap over the closest comparator could be within sampling noise.","Going beyond the paper, running the same pipeline on manual ADReSS20 transcripts would isolate how much information is lost by automatic transcription, and comparing the Zephyr pipeline with an acoustic-only classifier would test whether the text-only premise omits a large part of the signal.","Going beyond the paper, the concentration of false negatives near the MMSE 24–30 boundary suggests that adding prosodic or acoustic features to the text embeddings is the most direct route to improving detection of preclinical and very early Alzheimer's disease."],"forward_implications":["A clinic could run the entire screening loop on its own servers and still get benchmark-competitive accuracy, removing a major privacy obstacle to using LLMs on patient speech.","Automatic Whisper transcription is sufficient; manual transcripts and specialized recording hardware are not required, which widens the population that could be screened.","Because a 7B-parameter model with PCA and logistic regression beats a 30B model in the same pipeline, model scale is not the main driver of accuracy, and small instruction-tuned models may be enough for this task.","Cross-dataset transfer staying near 87% in both directions suggests the learned linguistic markers generalize across recording protocols, a precondition for real-world deployment.","The MMSE-stratified analysis indicates mild cases with scores 19–23 are captured, supporting use of the framework as an early-screening trigger rather than only a confirmatory test.","Going beyond the paper, an immediate testable extension is to report bootstrap confidence intervals around the 90.0% figure; with only 119 test samples the 5.1-point gap over the closest comparator could be within sampling noise.","Going beyond the paper, running the same pipeline on manual ADReSS20 transcripts would isolate how much information is lost by automatic transcription, and comparing the Zephyr pipeline with an acoustic-only classifier would test whether the text-only premise omits a large part of the signal.","Going beyond the paper, the concentration of false negatives near the MMSE 24–30 boundary suggests that adding prosodic or acoustic features to the text embeddings is the most direct route to improving detection of preclinical and very early Alzheimer's disease."],"supporting_citations":[{"why":"Supplies the ADReSS20 dataset and the standardized Cookie Theft evaluation protocol on which the headline comparison is built.","marker":"[18]"},{"why":"Supplies the ADReSSo2021 dataset and the 78.9% challenge baseline accuracy that the framework must beat.","marker":"[45]"},{"why":"The Whisper ASR model that produces the transcripts; the paper attributes its 90.0% accuracy partly to Whisper's robustness in clinical speech conditions.","marker":"[70]"},{"why":"Defines Zephyr-7B-β, the embedding backbone whose penultimate-layer representations carry the diagnostic signal in the pipeline.","marker":"[65]"},{"why":"The closest prior privacy-preserving local-LLM system, ADetectoLocum, whose 84.9% accuracy is the main comparator LSEAD claims to exceed.","marker":"[61]"},{"why":"The LLM-based speech AD recognition method reported at 83.1% accuracy, used as the main language-model comparison.","marker":"[36]"},{"why":"The end-to-end voice AD detection system reported at 80.3% accuracy, compared on accuracy, precision, recall, and F1.","marker":"[30]"},{"why":"The Wav2Vec 2.0 ASR baseline in the ASR-robustness study, used to show that the pipeline is not tightly coupled to a single transcription system.","marker":"[69]"},{"why":"Supports the design premise that textual content captures salient AD-related cognitive-linguistic impairments better than acoustic features.","marker":"[48]"},{"why":"The GPT-3.5 reference used to argue that Zephyr-7B-β approaches commercial-level capability while remaining locally deployable.","marker":"[44]"}],"fun_headline_variants":["Privacy-preserving LLM listens for Alzheimer's in speech","Open-source LLM detects early Alzheimer's from speech transcripts","Local LLM beats cloud for Alzheimer's screening from words","Speech analysis with LLM aids early Alzheimer's detection privately","LLM embeddings from speech transcripts spot Alzheimer's early"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's accuracy depends on the assumption that the Alzheimer's signal in the Cookie Theft recordings survives automatic transcription and is carried by the words and their meaning, rather than by how they are spoken.","fun_headline_variants_meta":{"raw":{"variants":["Privacy-preserving LLM listens for Alzheimer's in speech","Open-source LLM detects early Alzheimer's from speech transcripts","Local LLM beats cloud for Alzheimer's screening from words","Speech analysis with LLM aids early Alzheimer's detection privately","LLM embeddings from speech transcripts spot Alzheimer's early"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3062,"prompt_tokens":1002,"completion_tokens":2060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":1994}},"tokens_in":618,"tokens_out":2060,"duration_ms":12572,"temperature":1.0,"reasoning_tokens":1994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:27:55.332278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an acoustic-only classifier using prosody, pause structure, and voice quality on the same combined ADReSS20 and ADReSSo2021 test set; if acoustics alone matches or exceeds the reported 90.0% accuracy, the paper's text-only premise is false. Alternatively, report a 95% confidence interval for the 119-sample test accuracy; a confidence interval spanning the 84.9% comparator would mean the headline margin is not statistically established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest prior privacy-preserving local-LLM system, ADetectoLocum, whose 84.9% accuracy is the main comparator LSEAD claims to exceed."},{"cited_title":"Baevski, Y","cited_arxiv_id":null,"evidence_quote":"The Wav2Vec 2.0 ASR baseline in the ASR-robustness study, used to show that the pipeline is not tightly coupled to a single transcription system."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the design premise that textual content captures salient AD-related cognitive-linguistic impairments better than acoustic features."}],"review_version":2}