{"id":"797082b1-2715-4c41-a396-94b40ebb4f80","arxiv_id":"2505.19448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Certain ASR transcripts and speech synthesized from them outperform manual transcripts in Alzheimer's disease detection, suggesting ASR errors can serve as useful diagnostic cues.","lead":"This study tested whether automatic speech recognition (ASR) errors can help detect Alzheimer's disease and found that some ASR transcripts, and speech synthesized from them, yield higher detection accuracy than careful manual transcripts. The authors propose a cross-attention model to identify which linguistic cues ASR errors amplify, with the goal of making automated dementia screening cheaper and more practical.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's headline comparisons are not tested for significance; on 48 test samples, the reported margins are consistent with seed and sampling noise, so the central claim that ASR errors help AD detection is not yet supported.","rationale":"The paper has real strengths: it fine-tunes 18 ASR variants, evaluates 36 transcripts plus TTS-synthesized versions on the standard ADReSS benchmark, and reports mean accuracies over 10 random seeds. The cross-attention interpretability model is a reasonable extension of existing baselines, and the observation that some ASR-based inputs numerically beat manual transcripts is worth reporting. However, the central claim in the abstract is comparative: certain ASR transcripts and ASR-synthesized speech 'outperform' manual counterparts. That claim requires the observed accuracy differences to be distinguishable from noise. The data do not currently show this. The test set has 48 subjects, the reported differences are often only a few percentage points, and no significance testing or confidence intervals are provided. With 108 comparisons (36 ASR conditions times three detection methods), chance findings are virtually guaranteed under the null. This is more load-bearing than the TTS prompt-matching concern raised by the reader, because it affects both the transcript and synthesized-speech halves of the central claim; the TTS confound affects only the synthesized-speech half and would be moot if the transcript differences are not reliable. The TTS prompt concern is nevertheless real and should be reported: Section 3.2 says a random segment is selected as the prompt for each transcript, so ASR- and manual-synthesized speech may differ in prompt acoustics unrelated to transcript content. The interpretability analysis in Section 4.4 also selects the best-performing random seed on the test set before computing attention scores, which is a form of test-set selection and weakens the causal interpretation. The unshown 'additional statistical experiments' in Section 4.4 are a further missing-support issue that the manuscript itself flags by omission. None of these objections proves the claim false; they show the evidence is currently insufficient. A conditional verdict with a demand for significance testing, prompt matching, and released predictions is therefore appropriate, which matches the reader's conditional verdict.","tokens_in":10408,"tokens_out":4133,"duration_ms":36358,"concrete_test":"Recompute Table 2 using per-seed test predictions for the 48 ADReSS subjects. For each ASR transcript and each synthesized-speech condition, compare against the corresponding manual condition with a paired McNemar/bootstrap test (or paired t-test across seeds if the same 10 seeds were used), then apply Benjamini-Hochberg correction across 36x3=108 comparisons. Also report 95% bootstrap CIs for the top accuracy differences. If no comparison survives FDR at q=0.05, or if CIs for the top differences include zero, the headline claim should be downgraded and the interpretation section restricted to a hypothesis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 2's mean accuracies over 10 seeds on the 48-sample ADReSS test set. No confidence intervals, paired significance tests, or multiple-comparison corrections are reported. With n=48, a one-sample standard error for accuracy is about sqrt(0.5*0.5/48)=7.2%, and the largest transcript advantage (fine-tuned whisper large v2, 85.42% vs manual 81.67% in the embedding self-attention row) is 3.75 percentage points, i.e., roughly 1.8 subjects; most highlighted advantages are smaller (e.g., whisper medium 82.92% vs 81.67%). Given 36 ASR variants and three detection methods, some 'certain' winners are expected by chance even if ASR and manual transcripts carry identical information. The same issue affects the synthesized-speech comparison (best ASR-synth 77.50% vs manual-synth 74.58%). The paper's mechanism claim ('asymmetric biases') is supported only by 'additional statistical experiments' that are not reported (Section 4.4). This is a load-bearing gap because the central assertion is comparative and the differences are within plausible noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether automatic speech recognition (ASR) transcripts, and speech synthesized from them, can outperform manual transcripts for Alzheimer's disease (AD) detection on the ADReSS dataset. The authors fine-tune 18 ASR model variants on DementiaBank data, produce 36 ASR transcripts per ADReSS subject, and compare AD detection accuracy using BERT/Wav2Vec2 embeddings and knowledge-based features in self-attention and cross-attention models. They report that certain ASR transcripts and ASR-synthesized speech yield higher accuracy than manual counterparts, and they propose a cross-attention interpretability model to identify which knowledge-based features are amplified by ASR errors. The paper also claims that ASR errors introduce asymmetric biases between AD and healthy control groups that help detection.","tokens_in":10610,"tokens_out":2859,"duration_ms":28050,"significance":"If the central empirical claim is robust, the finding is practically significant: ASR transcripts are cheaper to obtain than manual transcripts, and the possibility that ASR errors carry informative cues about AD could improve scalable screening. The study is also unusually broad, covering 36 ASR variants and two input modalities (text and synthesized speech), and the proposed cross-attention model offers a concrete interpretability mechanism. However, the significance hinges entirely on whether the reported accuracy differences are statistically meaningful and on whether the synthesized-speech comparison controls for acoustic confounds. The paper does not currently provide the needed evidence for these load-bearing points.","major_comments":[{"comment":"The central claim that certain ASR transcripts and ASR-synthesized speech outperform manual counterparts is not supported by significance testing. The table reports mean accuracy over 10 seeds on a 48-sample test set, but no standard deviations, confidence intervals, or paired significance tests are given. With n=48, the standard error of an accuracy estimate is roughly 7 percentage points, and the largest headline margin (85.42% vs. 81.67% for fine-tuned whisper large v2 in the embedding self-attention row) is 3.75 points, i.e., about 1.8 subjects. Since 36 ASR variants and three detection methods are compared, some apparent winners are expected by chance. Please report per-seed variance, perform paired tests (e.g., McNemar or bootstrap) for the manual-vs-ASR comparisons, and correct for multiple comparisons, or the abstract's assertion is not established.","section":"Section 4.3, Table 2"},{"comment":"The synthesized-speech comparison is potentially confounded by the choice of TTS prompt. The paper states that for each transcript the authors 'randomly selected a segment from the subject's original speech as a prompt' for CosyVoice2. If the prompt segment is not matched across the manual and ASR conditions, the synthesized audio may differ in speaker identity, recording conditions, or segment-specific acoustic content that is unrelated to whether the transcript came from ASR or manual transcription. The measured accuracy difference between ASR-synthesized and manual-synthesized speech cannot then be attributed purely to transcript content. Please specify whether the same prompt segment was used for all transcript conditions of a given subject, and if not, re-run the comparison with matched prompts or otherwise control for this variable.","section":"Section 3.2, Section 4.3"},{"comment":"The interpretability analysis selects the best-performing random seed on the test set and then analyzes attention scores from that same test set. This is circular: the model is chosen for its test-set accuracy, so the subsequent attention analysis is conditioned on the test set and may reflect chance patterns rather than stable ASR-related cues. A proper protocol would select hyperparameters and seeds on a validation split, or report the distribution of attention patterns across seeds. Additionally, the claim that 'additional statistical experiments support the hypothesis' is not backed by any reported results; the experiments are mentioned but not described or shown, so the asymmetric-bias explanation is currently unsupported.","section":"Section 4.4"},{"comment":"The paper's repeated use of 'certain ASR transcripts' is insufficiently precise. Across 36 ASR variants and three detection methods, the identity of the 'certain' winners changes by row, and no criterion is given for what counts as a systematic advantage. A reader cannot tell whether the reported superiority is a consistent property of specific ASR error patterns or a set of isolated high-scoring configurations. Please define the comparison criterion a priori, and report the number of variants/methods that exceed manual performance and whether this number is itself significant relative to the 36-by-3 comparison space.","section":"Section 4.3, observation (a)"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'withih' should be 'within', and 'manual transcripts is labor-intensive' should be 'manual transcripts are labor-intensive'.","section":"Section 1"},{"comment":"The paper says the attention heatmap for original speech is 'omitted due to space limitations', but the corresponding analysis is a substantive part of the claimed contribution. Please include the figure or a quantitative summary, otherwise Section 4.5 cannot be verified.","section":"Section 4.5"},{"comment":"The use of an identity projection Wk so that knowledge-based features directly serve as K is an interesting design choice, but the paper should clarify whether this restricts the learnable interaction between embeddings and features, and whether alternative projections were considered in preliminary experiments.","section":"Section 3.4.2, Equation (1)"},{"comment":"The table caption defines the three slash-separated values, but the layout makes it difficult to compare rows quickly. Consider restructuring the table so that manual-transcript and manual-synthesized baselines are repeated or clearly marked in each block, and add a column for the WER of each ASR model to help the reader connect transcription quality with detection accuracy.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"Please consider whether the relationship between this submission and the authors' ICASSP 2025 paper (reference [21]) is sufficiently disclosed. If the earlier paper already reports the same core empirical finding that ASR errors can improve AD detection, the novelty of the present submission may need to be clarified. Also, the absence of release for code, seeds, and matched-prompt metadata will make it difficult for reviewers or readers to verify the central claims; I would encourage the editor to request these as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper is a broad empirical mapping of ASR transcripts for AD detection, and it confirms — with 18 model variants and a TTS twist — a result already present in the authors' own ICASSP paper and in Li et al. (2024). The most useful contribution is the systematic WER-accuracy sweep, plus the observation that the effect survives when transcripts are rendered as synthetic speech. If you work on automated dementia screening, this is a handy reference for which ASR models to try.\n\nWhat it does well: the 36-transcript comparison is genuinely wide, the fine-tuning setup is standard, and the paper is honest that synthesized speech underperforms original speech and that manual transcripts beat original audio. Those observations are real controls, even if they are not the headline. The citation pattern is appropriate: prior related work is cited, including the authors' own ICASSP paper, and the paper frames itself as an extension rather than a first report.\n\nThe soft spots are in the statistics and the interpretability section. Table 2 reports mean accuracy over 10 seeds on 48 test samples, with no standard deviations, confidence intervals, or paired tests. The largest highlighted advantage is about 3.75 points, roughly two subjects; most are smaller. On 48 samples the standard error for accuracy is around 7 points, so the headline claim — that certain ASR transcripts outperform manual ones — is not supported by the numbers as reported. With 36 ASR variants and three detection methods, a few winners are expected by chance. Separately, the cross-attention model uses the knowledge-based features directly as the key (identity projection), so the attention analysis partly measures how much the model uses the features it was given, not independent evidence that ASR errors created those cues. The claimed 'additional statistical experiments' in Section 4.4 are not shown; that is the only support for the asymmetric-bias mechanism, and its absence is a real gap.\n\nOne more concern: the TTS comparison selects a random segment from the subject's original speech as the prompt for each transcript. If the prompt segments were not matched across ASR and manual conditions, the synthesized-audio comparison is confounded by acoustic properties of the prompt. The paper does not say whether they were matched.\n\nWho this is for: people building ASR-based screening pipelines, and methodologists who want a case study in how easy it is to over-read accuracy differences on small clinical datasets. I would send it to review rather than desk-reject, because the breadth is useful and the question matters, but the author team should be asked to supply significance tests, matched prompts, and the missing statistics before the strong conclusions can stand.","headline":"Broad ASR survey that extends a known result, but the headline comparisons sit inside seed noise on a 48-sample test set.","tokens_in":11235,"tokens_out":2900,"would_cite":false,"duration_ms":26117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatically transcribed speech, errors included, can outperform human transcripts for detecting Alzheimer's disease.","keywords":["Alzheimer's disease detection","automatic speech recognition","ASR errors","cross-attention","interpretability","speech synthesis","ADReSS","dementia screening"],"falsifier":"Re-run the synthesized-speech comparison using the exact same prompt segment from the original recording for both the ASR transcript and the manual transcript of each subject; if the ASR-synthesized advantage disappears, the reported gain is an artifact of unequal prompts rather than evidence about ASR errors.","tokens_in":10152,"feed_emoji":"🧠","tokens_out":8029,"duration_ms":79410,"temperature":0.7,"pith_summary":"This paper tries to establish that errors introduced by automatic speech recognition are not merely noise to be removed before screening for Alzheimer's disease; in some cases they are diagnostic cues in their own right. On the ADReSS benchmark, certain ASR transcripts and speech re-synthesized from those transcripts classify Alzheimer's versus healthy controls more accurately than manual transcripts and their synthesized counterparts. The authors attribute the gain to asymmetric biases: recognizer mistakes mirror AD speakers' indistinct pronunciation and disfluencies, amplifying group differences in lexical and timing features. The paper also contributes a cross-attention model that performs the classification while exposing which features the errors strengthen. If the claim holds, fully automated, low-cost dementia screening could rely directly on recognizer output rather than expensive human transcription.","feed_headline":"Speech-recognition mistakes can aid Alzheimer's screening","feed_subtitle":"Some automated transcripts beat human-made ones at diagnosis, a sign that recognition errors carry useful cues.","key_machinery":"The machinery is a cross-attention interpretability model that fuses two input types: knowledge-based features (35 textual or 60 acoustic descriptors previously linked to AD) and embeddings from a pre-trained language or speech model. The key-projection matrix is fixed to identity, so the knowledge-based feature vector acts directly as the keys while BERT or Wav2Vec2 embeddings supply queries and values; the resulting attention matrix drives the classification and, averaged over correctly predicted samples, reveals which features receive heightened attention. This design converts the model's decision into a readable map of which cues—such as disfluency ratios or pause durations—are emphasized when ASR output is used, which is how the paper identifies the error-derived signals.","core_discovery":"The central discovery is that noisy machine transcription can beat the gold-standard human transcript for detecting Alzheimer's disease. After fine-tuning eighteen variants of four recognizer families and producing thirty-six transcripts with a wide range of word error rates, the paper finds that several ASR transcripts exceed manual transcripts in mean classification accuracy, and the same pattern appears when the transcripts are converted to speech by a neural text-to-speech model. The proposed cross-attention model reproduces the advantage and traces it to specific knowledge-based features—syllable and lexicon counts, average sentence length, filler-pause and repetition ratios, readability indices, and prosodic timing variables—that ASR errors amplify in the Alzheimer's group compared with healthy controls. These results support the paper's hypothesis that ASR errors introduce asymmetric, group-dependent biases that function as usable diagnostic signals.","pith_inferences":["An operational screener could select the recognizer that maximizes downstream detection accuracy on a held-out set, even if that recognizer is not the most accurate transcriber; the paper's numbers show accuracy and word error rate do not move together.","The cross-attention diagnostic could be transferred to other conditions marked by speech disfluency, such as Parkinson's disease or primary progressive aphasia, to test whether recognition bias is a general clinical signal.","A natural follow-up is to correlate error subtypes (insertions, deletions, substitutions) with cognitive severity scores, since the feature analysis suggests errors concentrate in disfluent, low-diversity stretches of speech."],"forward_implications":["End-to-end Alzheimer's screening can run from raw audio to diagnosis without human transcription, since recognizer output is sufficient and sometimes more accurate.","Word error rate is not a reliable guide to diagnostic value: the best-performing detector in the tables is not the lowest-error recognizer.","ASR errors that reflect articulation and disfluency can be treated as deliberate biomarkers instead of artifacts to be discarded.","Because synthesized speech scores below original audio, TTS-based analysis is best used as an experimental tool for isolating transcript content, not as a clinical replacement for real recordings.","The same cross-attention classifier can be applied to other transcript- or speech-based cognitive assessments, carrying interpretability with it."],"supporting_citations":[{"why":"supplies the ADReSS benchmark, its manual transcripts, and the train/test split used for all detection experiments.","marker":"[8]"},{"why":"supplies the Wav2Vec2 self-supervised speech representations used for classification and synthesized-speech analysis.","marker":"[14]"},{"why":"supplies the Whisper recognizer family, whose fine-tuned variants produce the low-WER transcripts that outperform manual ones.","marker":"[17]"},{"why":"documents prior evidence that ASR errors affect AD detection, the baseline finding this paper overturns by showing errors can help.","marker":"[19]"},{"why":"gives earlier evidence that ASR errors can improve dementia classification, which this study broadens across many models and modalities.","marker":"[20]"},{"why":"provides the CosyVoice2 text-to-speech engine that renders transcripts into speech for the synthesized-speech comparison.","marker":"[25]"},{"why":"justifies using an intermediate Wav2Vec2 layer, guiding the embedding choice in the cross-attention model.","marker":"[26]"}],"fun_headline_variants":["ASR slips rival human transcripts for Alzheimer's detection","When speech AI errs, it spots Alzheimer's better","Machine transcription errors sharpen Alzheimer's screening","Robot speech mistakes outdo manual notes for Alzheimer's"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthesized-speech result assumes the random voice prompt chosen for each transcript does not differ between ASR-derived and manually derived conditions in ways that affect detection; if it does, the apparent accuracy gain may come from acoustics rather than from transcript errors.","fun_headline_variants_meta":{"raw":{"variants":["ASR slips rival human transcripts for Alzheimer's detection","When speech AI errs, it spots Alzheimer's better","Machine transcription errors sharpen Alzheimer's screening","Robot speech mistakes outdo manual notes for Alzheimer's"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000126,"raw_usage":{"total_tokens":1065,"prompt_tokens":856,"completion_tokens":209,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":148}},"tokens_in":472,"tokens_out":209,"duration_ms":2959,"temperature":1.0,"reasoning_tokens":148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:14:00.565995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the synthesized-speech comparison using the exact same prompt segment from the original recording for both the ASR transcript and the manual transcript of each subject; if the ASR-synthesized advantage disappears, the reported gain is an artifact of unequal prompts rather than evidence about ASR errors.","supporting_citations":[{"cited_title":"Impact of ASR on Alzheimer's Disease Detection: All Errors are Equal, but Deletions are More Equal than Others","cited_arxiv_id":"1904.01684","evidence_quote":"provides the CosyVoice2 text-to-speech engine that renders transcripts into speech for the synthesized-speech comparison."},{"cited_title":"Useful blunders: Can automated speech recognition errors improve downstream demen- tia classification?","cited_arxiv_id":null,"evidence_quote":"justifies using an intermediate Wav2Vec2 layer, guiding the embedding choice in the cross-attention model."}],"review_version":1}