{"id":"637f7a5e-fd9e-4c03-a8c3-f4166807073f","arxiv_id":"2506.11119","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On PREPARE spontaneous speech, Whisper-medium audio embeddings achieved the best three-way ADRD classification (0.731 accuracy, 0.802 AUC), outperforming text-based and traditional acoustic pipelines.","lead":"This paper compares 15 speech and several text models for detecting Alzheimer's and related dementias from spontaneous speech recordings. On the PREPARE dataset, Whisper-medium audio embeddings did best, with 73% accuracy and 0.80 AUC in three-way classification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Traditional-feature results in Table 4 appear to come from a binary HC-vs-AD task per Supplementary S2, while Tables 2-3 report three-class HC/MCI/AD metrics; the paper's claim that ASR embeddings outperform traditional feature pipelines may therefore rest on an invalid comparison.","rationale":"The reader's concern about spontaneous-speech filtering is legitimate but secondary: even if the filtering were perfect, the central comparative claim against traditional feature pipelines could still be invalid if Table 4 was binary. I therefore treat the S2 inconsistency as the most load-bearing issue. This is a concrete, checkable flaw rather than a subjective worry; it is internal to the manuscript and does not require assumptions about intent. The rest of the benchmark, including Whisper-medium's rank among the audio models, may still survive, so the appropriate outcome remains conditional acceptance pending recomputation or clarification rather than rejection. I do not see an internal inconsistency in the main model-ranking logic itself; the main tables are structurally reproducible, and the paper is appropriately cautious about data diversity and interpretability.","tokens_in":15725,"tokens_out":11603,"duration_ms":149702,"concrete_test":"Obtain the data-processing and training scripts (or the authors' clarification) for Table 4, verify whether labels were binary HC-vs-AD or three-class, then rerun each traditional feature set (eGeMAPS, ComParE, MFCC, LIWC) with the same 80/20 stratified splits, classifier budget, and three-class accuracy/AUC used in Tables 2-3; if the three-class traditional numbers remain below Whisper-medium's 0.7307/0.8024 in a paired comparison, the concern is resolved, otherwise the 'outperforms traditional feature pipelines' claim in the abstract and Section 4 must be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Supplementary S2 states: \"Each feature set was used independently to train binary classification models distinguishing Alzheimer's disease (AD) patients from cognitively normal controls.\" Table 4 then reports eGeMAPS (accuracy 0.6427, AUC 0.7669), ComParE, MFCC, and LIWC results, and Section 4 uses these values to argue that traditional acoustic features are competitive with several foundation models and that \"the Whisper-medium model significantly outperformed both the traditional feature-based models and other embedding models.\" If Table 4 was produced on the HC/AD subset with binary labels (excluding the 81 MCI samples), those numbers are not comparable to the three-class accuracy and AUC in Tables 2-3: the sample, class priors, and metric definition differ. The abstract's conclusion that acoustic/ASR-derived embeddings outperform traditional feature pipelines depends directly on this comparison, so the manuscript needs to state the task and label set for Table 4 and, if it was binary, recompute the traditional baselines under the same three-class protocol. The absence of released code makes the ambiguity unresolvable from the paper as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks a range of open-source foundation speech and language models for three-class (HC/MCI/AD) classification of spontaneous speech from the PREPARE dataset. It reports that Whisper-medium achieves the best audio-based performance (accuracy 0.731, AUC 0.802), that BERT with pause annotation is the best text-based model (accuracy 0.662, AUC 0.744), and that acoustic/ASR-derived embeddings outperform text-based and traditional feature pipelines. The experiments use five stratified train-test splits with a frozen embedding extractor and a trained classification layer, which is a sound protocol. However, the comparison with traditional feature pipelines is undermined by a mismatch in classification tasks, and one claim about pause annotation is contradicted by the paper's own table.","tokens_in":15943,"tokens_out":3684,"duration_ms":40388,"significance":"If the results hold, the paper provides a useful benchmark of foundation models on a large, publicly available dataset for ADRD detection, with practical implications for scalable screening. The multi-seed evaluation, frozen-embedding design, and use of demographic covariates are strengths. The comparison against traditional features is important for establishing the value of foundation models, but the current evidence for that comparison is invalid as presented. The paper also offers a clear description of the data filtering pipeline, though the human-evaluation step needs more transparency.","major_comments":[{"comment":"The traditional feature results in Table 4 are produced by binary HC-vs-AD classification, as stated in Supplementary S2: \"Each feature set was used independently to train binary classification models distinguishing Alzheimer's disease (AD) patients from cognitively normal controls.\" In contrast, Tables 2 and 3 report three-class HC/MCI/AD metrics. The accuracy and AUC values are therefore not comparable across these tables because the label sets, sample compositions, and class priors differ. The Discussion (Section 4) uses Table 4 to claim that \"The Whisper-medium model significantly outperformed both the traditional feature-based models and other embedding models,\" which is a central conclusion of the paper. Please recompute the traditional baselines under the identical three-class protocol, or explicitly label Table 4 as a binary task and avoid direct numerical comparison. Without this correction, the claim that ASR embeddings outperform traditional feature pipelines is not supported.","section":"Section 3, Table 4; Supplementary S2; Section 4"},{"comment":"The text states that \"incorporating pause annotations consistently led to improved performance—in terms of both accuracy and AUC—compared to their counterparts without pause annotations.\" This is contradicted by the BioClinicalBERT row for Whisper-small transcriptions: with pause, AUC is 0.7118, while without pause it is 0.7126. This is the only exception in Table 2, but it invalidates the word \"consistently\" as used. Please qualify the claim, for example by noting that pause annotation improved both metrics in 11 of 12 configurations, or describe the exception explicitly.","section":"Section 3, text around Table 2"},{"comment":"The identification of spontaneous speech samples relies on DBSCAN clustering of BERT embeddings followed by \"human evaluation to classify clusters to language tasks,\" with 83 samples excluded as non-spontaneous. No details are given on the number of human evaluators, the annotation protocol, inter-rater agreement, or any validation of the cluster-to-task assignment. Because this filtering determines the composition of the final 1,189-sample benchmark, it is load-bearing for every model comparison. Please provide a more detailed description of the human evaluation step, including agreement measures, and consider a sensitivity analysis or a reproducible rule that does not depend on subjective judgment.","section":"Section 2.1, Figure S1"},{"comment":"The paper reports Area Under the Curve (AUC) for a three-class classification problem without specifying how multiclass AUC is computed. Please state whether macro-averaged one-vs-rest AUC (or another definition) is used. This is important for interpreting the absolute AUC values and for comparing with other studies.","section":"Section 2.3.2, Evaluation"}],"minor_comments":[{"comment":"There is a typo: \"A coustic-based approaches\" should read \"Acoustic-based approaches.\"","section":"Abstract"},{"comment":"The figure caption refers to panels \"2(a)\" and \"2(b)\" but the figure is numbered 3; this should be corrected to avoid confusion.","section":"Figure 3 caption"},{"comment":"The row label \"Whisper-larger\" in the text should be \"Whisper-large\" to match the model names and avoid ambiguity.","section":"Section 3, Table 3 row names"},{"comment":"The column labeled \"Data sizea\" is described as \"additional words trained\" but the values for BERT (3.3B), BioBERT (18B), and BioClinicalBERT (0.5B) are not clearly sourced; please clarify whether these are pretraining corpus sizes and cite the corresponding references.","section":"Table 2, column header"},{"comment":"The paper states that data is publicly available but does not mention code. Given the complexity of the preprocessing and evaluation pipeline, releasing code would substantially improve reproducibility; please add a code availability statement or explain why it is omitted.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the invalid comparison between the binary traditional-feature results and the three-class foundation-model results. This is fixable but requires rerunning the traditional baselines under the three-class protocol or substantially softening the conclusions. The claim about consistent benefit of pause annotation is also overstated relative to the paper's own Table 2. The paper is otherwise a reasonable benchmark study; however, the current presentation overstates the evidence in two load-bearing places."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Takeaway: this is a genuinely useful large-scale benchmark of open speech and text models for Alzheimer's detection on PREPARE, and the finding that Whisper-medium is top is credible. But the abstract's claim that ASR embeddings outperform traditional feature pipelines is not supported, because the traditional-feature numbers in Table 4 come from a binary HC-vs-AD analysis (per S2) while everything else is three-class HC/MCI/AD. That stress-test concern holds up. The paper needs to recompute the traditional baselines under the same three-class protocol, or clearly present them as a separate binary analysis.\n\nWhat's new and good: 15 speech models plus three BERT-family models benchmarked on 1,189 spontaneous-speech samples with five stratified replications and frozen encoders. The ranking among Whisper variants is a real result — Whisper-medium at 0.731/0.802 beats small and large, which is an interesting non-monotonic scaling pattern. The pause-annotation comparison for BERT models is also a useful practical detail. Including age and sex as covariates is sensible.\n\nSoft spots, in order of severity. First, the Table 4/S2 mismatch is load-bearing for the headline conclusion about traditional features; fixing it requires rerunning the same three-class pipeline. Second, the claim that pause annotation consistently improves accuracy and AUC is contradicted by their own Table 2: BioClinicalBERT on Whisper-small transcripts gets higher accuracy with pause but lower AUC (0.7118 vs 0.7126). That is not a fatal flaw, but the wording needs to be softened. Third, the multiclass AUC is not defined anywhere, no code is released, and no significance tests are reported, so differences between models like Whisper-base and Whisper-medium could be noise. Fourth, the DBSCAN-plus-human-evaluation filtering of spontaneous speech is subjective; it deserves a sensitivity check or at least a note on inter-rater reliability.\n\nThis paper is for researchers building ADRD screening tools who need a model ranking on a realistic dataset. It deserves a serious referee; the empirical scope is valuable and the issues are fixable. I would recommend peer review with major revisions.","headline":"A useful foundation-model benchmark on a real-world ADRD dataset, but the comparison against traditional features is undermined by a binary-vs-three-class mismatch.","tokens_in":16497,"tokens_out":2582,"would_cite":false,"duration_ms":32719,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Whisper-medium embeddings outperform text and engineered features for three-way dementia classification from spontaneous speech.","keywords":["Alzheimer's disease","dementia detection","foundation models","speech embeddings","automatic speech recognition","Whisper","spontaneous speech","benchmarking"],"falsifier":"Re-run the identical frozen-embedding classifiers on all 1,646 PREPARE training recordings without the spontaneous-speech filter, or on an independent spontaneous-speech corpus with age- and sex-matched groups; if Whisper-medium's margin over BERT-with-pause disappears or flips, the acoustic-superiority claim fails. Alternatively, report per-class recall on the 81 MCI samples: if MCI recall is near zero, the reported accuracy is carried almost entirely by HC/AD separation.","tokens_in":15530,"feed_emoji":"🧠","tokens_out":6608,"duration_ms":72337,"temperature":0.7,"pith_summary":"This paper argues that frozen embeddings from open-source foundation speech models can serve as practical biomarkers for Alzheimer's disease and related dementias. On 1,189 spontaneous-speech recordings from the PREPARE dataset, it reports that Whisper-medium achieves the best three-way classification of healthy control, mild cognitive impairment, and Alzheimer's disease, at 0.731 accuracy and 0.802 AUC. It further claims that acoustic embeddings from automatic speech recognition models outperform text-embedding pipelines and handcrafted acoustic features, while adding pause annotations consistently improves text models. The result matters because it points to a scalable, non-invasive screening route that avoids manual transcription and feature engineering.","feed_headline":"Whisper-medium beats text models at spotting dementia in speech","feed_subtitle":"Frozen audio embeddings reach 0.731 accuracy and 0.802 AUC on a 1,189-sample HC/MCI/AD benchmark.","key_machinery":"The load-bearing object is the frozen, mean-pooled encoder embedding of a pre-trained foundation model, treated as a feature vector for a small trained classifier. For the audio pipeline, Whisper converts 16 kHz audio into log-Mel spectrograms and its transformer encoder produces clip-level embeddings (1024 dimensions for Whisper-medium); for the text pipeline, BERT-family models embed transcripts in which pauses are encoded as punctuation marks (comma, period, ellipsis for short, medium, long pauses). What carries the argument is the claim that large-scale pre-training on diverse speech lets Whisper encode both semantic content and paralinguistic timing cues, so a single frozen representation outperforms task-specific feature engineering.","core_discovery":"The central claim is that among the benchmarked models, Whisper-medium's encoder embeddings—mean-pooled across the clip and concatenated with age and sex before a single-layer feedforward classifier—give the best performance for HC/MCI/AD classification, with accuracy $0.7307 \\pm 0.0202$ and AUC $0.8024 \\pm 0.0143$ over five random splits. Among language models, BERT with pause markers reaches the top accuracy of $0.6622 \\pm 0.0131$ and AUC of $0.7444 \\pm 0.0136$, and adding pause markers improves every text-based configuration. The paper also shows that Whisper-medium beats both Whisper-large and Whisper-tiny, that traditional feature sets such as eGeMAPS reach levels comparable to several smaller speech models but not to Whisper-medium, and that the advantage of speech models is attributed to their ability to retain non-semantic cues such as pauses and prosody without explicit annotation.","pith_inferences":["The paper reports accuracy and AUC but not per-class recall; with MCI at only 6.81% of the sample, per-class sensitivity would clarify whether the model actually distinguishes MCI or mostly separates HC from AD.","The 83-sample exclusion by DBSCAN and human review means the benchmark covers a curated spontaneous-speech subset; running the same pipeline on the full multilingual set would test whether the ranking holds outside English Cookie Theft descriptions.","The pause-annotation gain in text models and Whisper's acoustic advantage may be two views of the same timing signal; an experiment with deliberately pause-perturbed audio could separate semantic from timing contributions.","A longitudinal extension is the natural next test: if embedding trajectories shift with disease progression, the same frozen encoder could track within-person change rather than only cross-sectional status."],"forward_implications":["Acoustic-only screening is feasible: the best speech model needs no transcript and no manual pause annotation to reach 0.731 accuracy and 0.802 AUC on a three-way clinical task.","Model scale is not monotonic: Whisper-medium outperforms Whisper-large, suggesting an optimal size near 769M parameters for this task and dataset.","Non-semantic information carries signal: adding pause markers to transcripts improves every BERT-family configuration tested.","A practical telehealth or mobile workflow is within reach: frozen embeddings plus a 128-unit classification layer is computationally light enough for deployment.","Traditional feature pipelines still compete: eGeMAPS ($0.643$ accuracy, $0.767$ AUC) matches several self-supervised speech models but falls short of Whisper-medium."],"supporting_citations":[{"why":"supplies the PREPARE Phase 2 training audio and the HC/MCI/AD labels that define the benchmark","marker":"[31]"},{"why":"provides the Whisper family of ASR models and the spectrogram preprocessing used for both transcription and audio embeddings","marker":"[16]"},{"why":"introduces the pause-annotation scheme and the prior binary ADReSS result whose 0.896 accuracy the paper compares against","marker":"[29]"},{"why":"provides BERT, the text-embedding backbone, and the embedding representation used in DBSCAN clustering for spontaneous-speech filtering","marker":"[35]"},{"why":"describes DementiaBank, the corpus from which PREPARE recordings were curated and which motivates the spontaneous-speech task","marker":"[32-34]"},{"why":"supplies wav2vec2, a comparison speech model representing self-supervised representations trained on LibriSpeech","marker":"[43]"},{"why":"supplies WavLM, the comparison self-supervised speech model with denoising and masked speech objectives","marker":"[46]"},{"why":"defines the ADReSS spontaneous-speech benchmark and the balanced small-data setting the paper contrasts with its larger, imbalanced sample","marker":"[52]"}],"fun_headline_variants":["Whisper-medium wins dementia detection with 0.73 accuracy","Speech AI beats language AI for dementia screening","Pauses and prosody power speech models in dementia detection","Whisper-medium tops dementia detection from speech, hits 0.80 AUC","Speech-based AI outperforms text for early dementia detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the DBSCAN clustering of BERT embeddings plus human review correctly identifies the 1,189 spontaneous-speech samples; if that curation is biased, every model comparison in the paper describes a different, filtered task rather than spontaneous speech detection.","fun_headline_variants_meta":{"raw":{"variants":["Whisper-medium wins dementia detection with 0.73 accuracy","Speech AI beats language AI for dementia screening","Pauses and prosody power speech models in dementia detection","Whisper-medium tops dementia detection from speech, hits 0.80 AUC","Speech-based AI outperforms text for early dementia detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2983,"prompt_tokens":1068,"completion_tokens":1915,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":684,"tokens_out":1915,"duration_ms":16197,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:21:03.795025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the identical frozen-embedding classifiers on all 1,646 PREPARE training recordings without the spontaneous-speech filter, or on an independent spontaneous-speech corpus with age- and sex-matched groups; if Whisper-medium's margin over BERT-with-pause disappears or flips, the acoustic-superiority claim fails. Alternatively, report per-class recall on the 81 MCI samples: if MCI recall is near zero, the reported accuracy is carried almost entirely by HC/AD separation.","supporting_citations":[{"cited_title":"2024 [accessed April 15 2024]","cited_arxiv_id":null,"evidence_quote":"supplies the PREPARE Phase 2 training audio and the HC/MCI/AD labels that define the benchmark"},{"cited_title":"Radford, J.W","cited_arxiv_id":null,"evidence_quote":"provides the Whisper family of ASR models and the spectrogram preprocessing used for both transcription and audio embeddings"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces the pause-annotation scheme and the prior binary ADReSS result whose 0.896 accuracy the paper compares against"},{"cited_title":"Devlin, M.-W","cited_arxiv_id":null,"evidence_quote":"provides BERT, the text-embedding backbone, and the embedding representation used in DBSCAN clustering for spontaneous-speech filtering"},{"cited_title":"Baevski, Y","cited_arxiv_id":null,"evidence_quote":"supplies wav2vec2, a comparison speech model representing self-supervised representations trained on LibriSpeech"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the ADReSS spontaneous-speech benchmark and the balanced small-data setting the paper contrasts with its larger, imbalanced sample"}],"review_version":1}