{"id":"9013f908-389d-49ab-99b5-de8648f74806","arxiv_id":"2412.06332","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The analysis of ASR errors in BERT-based Alzheimer's detection shows that stopwords dominate the errors but barely affect classification, while task-related keywords are transcribed accurately and are the decisive factor.","lead":"An automatic speech recognition system transcribing elderly speech with a 33.9% word error rate still enables the same Alzheimer's detection accuracy as human transcription, because most errors are small function words. The words that matter for diagnosis, such as picture-description keywords, are mostly transcribed correctly and carry most of the classification weight.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Word-type importance is inferred from random edits on manual transcripts and only from the 40/48 ASR-robust cases; a direct counterfactual correction of actual ASR errors is needed before concluding keyword errors are pivotal.","rationale":"The reader's weakest assumption correctly identifies the random-edit perturbation as a proxy that may not mimic real ASR errors. I agree with that concern, but I see an additional load-bearing issue: the empirical error statistics that motivate the claim are computed only on the ASR-robust subset. Because those cases are selected by the outcome being explained, the association between stopword errors and harmless errors is partly built into the sample. This does not make the paper internally inconsistent, and the descriptive observations are informative: keyword WER is much lower than overall WER, stopwords account for 60% of errors in robust cases, and the offset curves in Figure 4 show a sensible gradient. However, the central causal claim about which actual ASR errors matter downstream requires a direct intervention on the ASR hypotheses, not only on manual transcripts. The counterfactual correction experiment I propose is exactly that intervention and would settle the concern either way. The reader's conditional verdict remains appropriate; I am not moving the verdict to accept or reject, but the revision should add this direct test or explicitly weaken the causal wording.","tokens_in":8552,"tokens_out":5025,"duration_ms":57751,"concrete_test":"On the actual test-set ASR transcriptions, construct three edited versions for each of the 48 speakers: (A) correct every ASR error whose reference word in the manual transcript is a stopword, leaving all other errors; (B) correct every ASR error whose reference word is a task-related keyword, leaving all other errors; (C) correct all errors, i.e. the manual transcript. Recompute BERT+SVM predictions for A, B, and C and compare them with the original ASR-based and manual-based predictions. If correcting keyword errors moves predictions toward the manual-transcription decisions substantially more than correcting stopword errors, the paper's ranking is confirmed; if correcting stopword errors also flips cases, or if neither correction recovers the manual labels, the word-type explanation is not the mechanism. Report these results separately for the 8 non-robust cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that stopword errors are harmless while keyword errors are pivotal rests on two indirect steps. First, Tables 4 and 5 compute error-composition statistics only on the 40/48 ASR-robust cases where ASR did not change the prediction. This is close to circular: a robust case is by definition one where the errors did not flip the classifier, so finding that those errors are mostly stopwords does not establish that stopword errors are generally harmless. The 8 non-robust cases could be dominated by keyword or non-stopword errors, which would support the claim, or by stopword errors, which would refute it. Second, the ablation in Section 4.2 and Figure 4 randomly removes or substitutes words in manual transcripts and measures the signed distance to the SVM hyperplane (Eq. 1). Random edits are not observed ASR errors: replacing a stopword with an unrelated content word changes text naturalness and length in ways that BERT embeddings may reflect, and this does not mimic the systematic acoustic confusions of a real ASR front-end. The paper therefore shows a potential effect of hypothetical edits, not an actual effect of ASR errors on downstream decisions. A direct counterfactual intervention on the real ASR transcriptions is needed to support the causal ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how automatic speech recognition (ASR) errors affect BERT-based Alzheimer's disease (AD) detection. Using a custom ASR system with 33.9% WER on the ADReSS-2020 test set, the authors reproduce the previous observation that ASR and manual transcriptions yield equivalent detection accuracy (88%). They then analyze error composition and report that, within the 40 ASR-robust cases where predictions did not change, stopwords comprise 60% of errors while task-related keywords comprise only 9%. Through word-ablation experiments on manual transcripts, they show that randomly removing or substituting stopwords shifts BERT embeddings toward the SVM decision boundary without crossing it, whereas removing or substituting keywords causes the embeddings to cross the boundary. The paper concludes that not all ASR errors are equally detrimental: stopword errors are largely harmless while keyword errors are pivotal for downstream AD detection.","tokens_in":8845,"tokens_out":5709,"duration_ms":55841,"significance":"The paper makes a useful descriptive contribution by documenting the non-uniform distribution of ASR errors across word types and by demonstrating that the BERT+SVM classifier's embedding is less sensitive to stopword edits than to keyword edits. It also reproduces the known non-linear WER-accuracy relationship and provides transparent error accounting with an externally sourced keyword list, which aids reproducibility. If the causal interpretation were fully supported, the findings would motivate ASR evaluation metrics that weight content-word errors more heavily in AD screening contexts. However, the causal claim currently rests on a proxy ablation rather than on a direct analysis of real ASR errors, so the significance is conditional on additional validation.","major_comments":[{"comment":"The error-composition statistics (60% stopwords, 9% keywords) are computed only on the 40/48 ASR-robust cases, i.e., cases where ASR errors did not change the prediction. This is close to circular for the central claim: by construction, these are cases where errors were not harmful. The 8 non-robust cases are excluded from the analysis and their error composition is not reported. If those 8 cases contain a high proportion of keyword or non-stopword errors, the conclusion would be strengthened; if they contain mostly stopword errors, the conclusion would be undermined. The paper should report the error composition for all 48 test cases, and specifically for the 8 non-robust cases, or justify why the restriction to robust cases is appropriate.","section":"Section 4.1 (Tables 4 and 5)"},{"comment":"The word-ablation experiments randomly remove or substitute words in manual transcripts, not actual ASR errors. Random edits do not mimic systematic ASR confusions (e.g., phonetic similarity, deletions and insertions correlated with word frequency and speaking style), and substituting a stopword with an unrelated content word changes text naturalness in ways that BERT embeddings may reflect. Moreover, the ablation uses edit ratios up to 100% of a word class, which far exceeds the per-speaker ASR error rate of roughly 34%. The resulting offset curves therefore measure the sensitivity of the fitted BERT+SVM decision boundary to arbitrary word-level perturbations, not the actual effect of ASR errors on downstream decisions. A direct counterfactual analysis on the real ASR transcriptions—for instance, correcting or perturbing actual recognized words by error type, or comparing manual vs. ASR embeddings for the same speaker—is needed to support the causal ranking of stopword versus keyword errors.","section":"Section 4.2 and Figure 4"},{"comment":"The claim that keyword errors are 'pivotal' is based on the ablation crossing the SVM boundary when keywords are edited. However, the 40 ASR-robust cases contain 130 keyword errors (9% of all errors) yet yield unchanged predictions. If keyword errors were as pivotal as the ablation suggests, one would expect the robust cases to contain few or no keyword errors, or the errors to occur in non-predictive positions. The paper does not reconcile the frequency of keyword errors in robust cases with the strong sensitivity shown in Figure 4(b). Please quantify per-speaker keyword error counts and their relation to prediction stability, or provide an explanation for why the ablation's boundary crossing does not translate into real ASR-induced flips.","section":"Section 4.3 (Table 5) and Conclusions"}],"minor_comments":[{"comment":"The heading contains a typo: 'Anayisis' should be 'Analysis'.","section":"Section 4.2 heading"},{"comment":"The abstract and introduction state that 'stopwords constitute 60% of errors' without noting that this figure comes from the ASR-robust subset; please qualify this claim in the abstract or clearly in the text.","section":"Abstract and Section 4.2"},{"comment":"Table 3 lists 36 keywords but the text says the list contains 39 task-related keywords; please verify the count or include the missing entries.","section":"Table 3"},{"comment":"The x-axis label 'Edit Ratio (x10%)' is ambiguous; consider labeling it as 'Edit ratio (×10)' or 'Edit ratio (1–10 = 10%–100%)'.","section":"Figure 4"},{"comment":"The PCA target dimensionality is set to 108 because it equals the training set size; this choice is mentioned but not justified as a design decision, and the sensitivity of the results to this hyperparameter is not explored.","section":"Section 2.3"},{"comment":"Reference [30] lists the title 'The far side of failure: Investigating the impact of speech recognition errors on subsequent dementia classification' but the cited work appears to have been published under a different title; please check the bibliographic details.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical analysis that fits the journal's scope. The descriptive findings and the sensitive yet clear ablation experiments are useful. The main risk is that the causal claim is overstated relative to the evidence: the error-composition analysis is restricted to robust cases, and the ablation uses random edits rather than real ASR errors. Both concerns are addressable within the manuscript's scope by adding the 8-case breakdown and a direct counterfactual analysis on the existing ASR transcriptions. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives a clear, quantitative answer to a question the field has been circling: when a 33.9% WER ASR transcript yields the same AD detection accuracy as a manual transcript, where do the errors actually land? The descriptive core is solid. Stopwords make up 60% of ASR errors, task keywords only 9%, and keyword WER is 14.3% versus 33.9% overall. That is a genuinely informative, reproducible observation that goes beyond prior claims that errors can be harmless or helpful. The alignment maps and the long-tail error distribution are good illustrations. Building on [31, 32] by explicitly separating stopwords from task keywords is a real extension, not just a rehash.\n\nThe soft spots are real but not fatal. First, the error-composition tables are computed only on the 40/48 ASR-robust cases, the ones where ASR did not flip the prediction. That is selection on the outcome: you are describing errors in cases where they had no impact, so concluding that those errors are harmless is somewhat circular. The 8 non-robust cases could be dominated by keyword errors, or even by stopword errors that happened to push the embedding across the boundary; we simply do not see them. The paper should at least report the error composition for those 8 cases separately.\n\nSecond, the ablation experiments randomly remove or substitute words in manual transcripts and measure the SVM hyperplane offset. That measures the classifier's sensitivity to hypothetical edits, not the actual effect of real ASR confusions. Real ASR errors are systematic and correlate with word acoustics and speaker characteristics. The paper is careful with its language, saying the results \"suggest\" rather than \"prove,\" but the abstract's wording that stopwords \"are shown to play a limited role\" is a bit stronger than the evidence supports. A direct counterfactual on the actual ASR hypotheses—e.g., replacing the recognized keyword with the manual keyword and measuring the decision change—would nail the causal claim.\n\nThat said, the central argument holds up as an analysis rather than a definitive experiment. The finding that task keywords are both low-WER and high-importance is a useful practical hint for building ASR-robust AD screening: optimize ASR for content words, not for stopword accuracy. The paper is concise, honest about its assumptions, and the descriptive statistics are reproducible. I would send it to peer review. A solid revision would address the 8 excluded cases and add a direct ASR-error simulation. For a reading group on clinical speech analytics, it is worth a slot.","headline":"Useful word-level breakdown of ASR errors in AD detection, but the causal importance ranking rests on a perturbation proxy and an analysis restricted to ASR-robust cases; still worth a referee.","tokens_in":9302,"tokens_out":2145,"would_cite":true,"duration_ms":24535,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ASR errors are not equal: stopwords contribute 60% of transcription errors but little discriminative signal, while task-related keywords—only 9% of errors—are pivotal for BERT-based Alzheimer's detection.","keywords":["Alzheimer's disease","speech recognition","pre-trained language model","spoken language processing","ASR error analysis","stopwords","task-related keywords","ADReSS"],"falsifier":"A direct test would be to train two ASR variants, one whose errors fall mainly on stopwords and one whose errors fall mainly on task keywords at the same overall WER; if the keyword-error variant does not degrade AD detection accuracy more than the stopword-error variant, the central claim that keywords are pivotal would be falsified.","tokens_in":8385,"feed_emoji":"🧠","tokens_out":5909,"duration_ms":55681,"temperature":0.7,"pith_summary":"This paper tries to explain why automatic transcriptions with a high word error rate can still support Alzheimer's disease detection as accurately as manual transcripts. Using the ADReSS Cookie Theft picture-description corpus, a BERT-based detector, and a tailor-made ASR system, it finds that a 33.9% word error rate leaves detection accuracy unchanged at 88%. The reason is that errors are not distributed evenly across word types: stopwords make up 60% of errors but carry little discriminative weight, while diagnosis-related keywords account for only 9% of errors and are the words that move BERT embeddings across the classification boundary. The implication is that for clinical screening, the quality of ASR should be measured by how well task-critical words survive, not by overall word error rate.","feed_headline":"Stopword errors are noise; keyword errors matter in AD screening","feed_subtitle":"A 33.9% word error rate keeps AD detection at 88% because most errors fall on low-information words.","key_machinery":"The central object is the signed hyperplane offset $d = (w^T x + b)/\\|w\\|$ of the BERT [CLS] embedding (PCA-reduced to 108 dimensions) from the linear SVM decision boundary. The argument works by taking manual transcripts, randomly removing or substituting stopwords versus keywords, recomputing BERT embeddings, and tracking whether the average offset crosses zero, the decision boundary. This turns the qualitative question of which ASR errors matter into a quantitative measure of how far different word-level edits push the classifier's input.","core_discovery":"On the paper's own terms, the discovery is that ASR errors are strongly biased toward semantically light words, and the downstream BERT-based classifier is correspondingly insensitive to those errors. In the ASR-robust cases, stopwords are 24% of error types but 60% of error tokens, with a stopword WER of 32.4%, whereas task-related keywords such as 'cookie', 'jar', 'boy', 'girl', and 'water' have a WER of only 14.3% and account for just 9% of all errors. Controlled ablation experiments on manual transcripts show that randomly removing or substituting stopwords shifts BERT embeddings toward the SVM decision boundary without crossing it, while removing or substituting keywords crosses the boundary and flips the classification. The paper concludes that stopword errors are largely harmless to AD detection, keyword errors are pivotal, and the apparent paradox of high WER coexisting with unchanged accuracy is resolved by this asymmetry.","pith_inferences":["A natural extension the paper does not draw: because ASR errors push embeddings toward the AD side of the boundary, stopword-heavy errors may systematically bias screening toward false positives.","The asymmetry suggests ASR confidence scores could be used to down-weight stopword errors when computing transcript-based features for clinical decision support.","The same word-class ablation could be applied to other clinical NLP tasks with a task-specific vocabulary; if the pattern generalizes, task-word preservation rather than overall WER should guide ASR development.","A testable prediction follows: an ASR language model biased toward Cookie Theft keywords should improve AD detection robustness more than an equal-WER reduction on general vocabulary."],"forward_implications":["If the central claim is right, ASR benchmarking for AD screening should report word error rates separately for task keywords and stopwords rather than a single overall WER.","A stopword-heavy ASR error profile can sustain 88% detection accuracy, so improving ASR on stopwords alone will not improve screening.","Recognition of the task-related keywords is the bottleneck; keyword WER at 14.3% is already low, so further gains should focus on the remaining keyword misses.","Because non-keyword and non-stopword errors also cross the boundary, the classifier is sensitive to semantic content broadly, meaning ASR evaluation should weight content words by their contribution to the downstream task.","Deletion or substitution of keywords, even a small fraction of errors, can flip a healthy-control transcript into the AD region, so error analyses should track keyword-level confusion rather than aggregate rates."],"supporting_citations":[{"why":"Supplies the 156-speaker Cookie Theft corpus with balanced train/test splits used for all detection and error analyses.","marker":"[33]"},{"why":"BERT supplies the [CLS] token embeddings that feed the SVM classifier.","marker":"[16]"},{"why":"Provides the BERT plus PCA plus SVM detection pipeline whose accuracy is the paper's starting point.","marker":"[19]"},{"why":"The adapted TDNN ASR system that produces the 33.9% WER transcriptions.","marker":"[25]"},{"why":"Source of the 39 task-related keyword list and prior evidence that ASR errors can aid dementia classification.","marker":"[31]"},{"why":"Prior work showing deletion errors affect AD detection, motivating the word-type error analysis.","marker":"[32]"},{"why":"Defines the stopword list used to partition errors into stopword and non-stopword sets.","marker":"[36]"}],"fun_headline_variants":["ASR errors hit stopwords, not keywords, so AD detection survives","Word error rate misleads when errors land on stopwords","Why high ASR error still yields 88% AD accuracy: stopwords absorb it","Stopwords absorb ASR errors, keeping AD detection resilient","Keyword errors flip AD detection; stopword errors are harmless"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that stopwords are unimportant rests on random word-level edits to manual transcripts standing in for real ASR errors; if real errors are systematic, correlated with speaker traits, word position, or acoustic difficulty, their effect on the classifier could differ from what the ablation measures.","fun_headline_variants_meta":{"raw":{"variants":["ASR errors hit stopwords, not keywords, so AD detection survives","Word error rate misleads when errors land on stopwords","Why high ASR error still yields 88% AD accuracy: stopwords absorb it","Stopwords absorb ASR errors, keeping AD detection resilient","Keyword errors flip AD detection; stopword errors are harmless"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1303,"prompt_tokens":899,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":313}},"tokens_in":515,"tokens_out":404,"duration_ms":4779,"temperature":1.0,"reasoning_tokens":313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:45:18.056022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to train two ASR variants, one whose errors fall mainly on stopwords and one whose errors fall mainly on task keywords at the same overall WER; if the keyword-error variant does not degrade AD detection accuracy more than the stopword-error variant, the central claim that keywords are pivotal would be falsified.","supporting_citations":[{"cited_title":"Per- sonalized adversarial data augmentation for dysarthric and elderly speech recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the 156-speaker Cookie Theft corpus with balanced train/test splits used for all detection and error analyses."},{"cited_title":"Exploiting multi- modal features from pre-trained networks for alzheimer’s demen- tia recognition,","cited_arxiv_id":null,"evidence_quote":"BERT supplies the [CLS] token embeddings that feed the SVM classifier."},{"cited_title":"A comparative study of acoustic and linguistic fea- tures classification for alzheimer’s disease detection,","cited_arxiv_id":null,"evidence_quote":"The adapted TDNN ASR system that produces the 33.9% WER transcriptions."},{"cited_title":"Development of the cuhk elderly speech recognition system for neurocognitive disorder detection using the dementia- bank corpus,","cited_arxiv_id":null,"evidence_quote":"Source of the 39 task-related keyword list and prior evidence that ASR errors can aid dementia classification."},{"cited_title":"The Far Side of Failure: Investigating the Impact of Speech Recognition Errors on Subsequent Dementia Classification","cited_arxiv_id":"2211.07430","evidence_quote":"Defines the stopword list used to partition errors into stopword and non-stopword sets."}],"review_version":1}