{"id":"02cbaba0-b1cd-4d0c-8e89-81aa436cddb3","arxiv_id":"2506.08846","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Across six ASR services, speakers with aphasia receive worse transcriptions than controls, and standard audit methods mask within-group disparities and hallucination risks.","lead":"This paper audits six commercial speech-to-text services on speech from people with aphasia and finds consistently worse transcription for aphasia speakers than for controls. It argues standard ASR audits hide these harms through single text-cleaning methods, broad group comparisons, and a single error metric, and proposes a more inclusive audit framework.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'hallucinations unique to Whisper' claim rests on spot checks of the other five ASR services, not on the systematic manual review applied to Whisper; this is the least secure part of the strongest claim.","rationale":"The reader's weakest assumption—that the cleaned AphasiaBank ground truth is unbiased—is real but largely conservative for the paper's headline claim. Removing segments containing unintelligible tokens (Appendix A.1.1) deletes the most severely affected aphasia speech; if anything, including those segments would raise aphasia WERs and widen the measured gap, so the central result of 'worse for aphasia' is unlikely to be an artifact of this cleaning. The same cannot be said for the hallucination exclusivity claim. It is a universal negative about five commercial systems, and the evidence offered is spot checks rather than the systematic, metric-screened review used for Whisper. The selection thresholds in Appendix A.8.2 create a blind spot: hallucinations that coexist with otherwise high similarity to the reference, or that occur in files below the 90th percentile of WER, would not be manually inspected. The paper is careful to call the Whisper rate a lower bound, but it does not apply symmetric caution to the other services. This asymmetry is the most load-bearing weakness in the strongest claim. A service-blind replication of the hallucination protocol, or even a random-sample version not filtered by metric thresholds, would settle it. If the test finds hallucinations in other services, the paper's contribution becomes the distribution of Whisper hallucinations rather than the uniqueness of Whisper; the WER findings and the three-pitfall framework survive, but the '53 of 56' claim and the 'no instances' statement need revision. Because the reader's verdict was already CONDITIONAL and this concern supports rather than overturns that conditionality, no verdict change is needed.","tokens_in":36415,"tokens_out":5573,"duration_ms":69100,"concrete_test":"Run the Appendix A.8.2 manual hallucination protocol on a random sample of 1,000 audio files per ASR service (6,000 total), stratified by aphasia/control and drawn independently of the original metric thresholds, with annotators blind to which service produced each transcript. If the confirmed hallucination rate for every non-Whisper service is 0/1,000, the exclusivity claim survives; if any service shows a nonzero rate, the paper must revise the 'no instances' statement and re-estimate the 53-of-56 concentration. As a power note, a true rate of 0.3% would be missed with probability ~5% in a 1,000-file sample, so a null result should be reported with an upper confidence bound rather than as proof of absence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 and Appendix A.8.2 establish the Whisper hallucination result by manually reviewing 1,198 metric-selected candidate files, but the claim that no other ASR service hallucinated is supported only by 'spot checks conducted throughout the cleaning process.' The candidate-selection procedure uses percentile thresholds on WER, BLEU, ROUGE, METEOR, WIL, RIL, CER, and Insertion Rate to pick high-risk files; a hallucination in Amazon, Google, Microsoft, AssemblyAI, or Rev AI that does not push a file past those thresholds would not enter the reviewed set. The paper also does not report a service-blind protocol for the spot checks, so differential review effort between Whisper and the other services is a plausible source of the asymmetry. Because the strongest claim includes 'Whisper hallucinations occurred almost exclusively for aphasia speakers (53 of 56 confirmed)' and the abstract's Pitfall 3 argument treats hallucinations as a distinct class of generative-AI errors, the exclusivity result is load-bearing: if another service hallucinates at a comparable rate, the empirical contribution narrows from 'only Whisper hallucinates' to 'Whisper hallucinates, and its hallucinations concentrate in aphasia speakers.' The WER disparity itself is not threatened by this concern; the issue is specifically the universal negative about the other five services.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a community-driven auditing framework for ASR systems and demonstrates it in a case study comparing six commercial ASR services on AphasiaBank speech from people with aphasia and a control group. It identifies three pitfalls in standard ASR audits—fixed text standardization, aggregate-only demographic comparisons, and reliance on a single metric (WER)—and presents evidence for each: WER disparities between aphasia and control speakers across all six services, heterogeneity of performance across aphasia subtypes and acoustic covariates, and Whisper-specific hallucinations that are largely concentrated in aphasia speakers. The paper also reports a small community survey indicating that aphasia speakers prefer heavily cleaned transcriptions, and it makes reproducibility claims backed by a public GitHub repository.","tokens_in":36591,"tokens_out":2735,"duration_ms":35398,"significance":"If the findings hold, the paper makes a valuable contribution to fair and accessible ASR auditing. The central WER disparity is supported by multiple robustness checks: it holds across six services, on matched and unmatched samples, under weighted and unweighted aggregation, and in regression models with clustered standard errors. The disaggregation by aphasia type and acoustic features is a useful corrective to monolithic group comparisons, and the community-engagement component is a strength, even if small. The paper also ships code and gives detailed preprocessing descriptions, which supports reproducibility. The main caveat is that one load-bearing claim—that hallucinations occur only in Whisper—rests on a weaker evidentiary basis than the rest of the empirical analysis.","major_comments":[{"comment":"The claim that 'no instances of hallucinations in the other ASR services' were found is not supported by the described methodology. The manual hallucination review was applied only to 1,198 Whisper candidate files selected via percentile thresholds on WER, BLEU, ROUGE, METEOR, WIL, RIL, CER, and Insertion Rate; the other five services were only spot-checked 'throughout the cleaning process.' A hallucination in Amazon, Google, Microsoft, AssemblyAI, or Rev AI that does not push a file past those thresholds would not enter the reviewed set, and the spot checks are not described as service-blind or systematic. Because the abstract and Pitfall 3 treat hallucinations as a distinct class of generative-AI errors, the exclusivity result is load-bearing. I recommend either conducting the same systematic manual review on the other services' candidate files or explicitly weakening the claim to 'Whisper hallucinated in our data; we did not systematically verify absence in other services.'","section":"§6 and Appendix A.8.2"},{"comment":"The ground-truth cleaning pipeline removes any word marked unintelligible ('xxx') and then drops every audio segment containing such a token. This is likely to exclude the most severely affected aphasia speech, since unintelligible tokens are a hallmark of severe aphasia. The paper does not report how many segments were excluded per group or provide a sensitivity analysis that retains or imputes these segments. This selection could attenuate or otherwise distort the measured WER disparity and the hallucination concentration. I ask the authors to quantify the exclusion counts by group and, if feasible, show that the main results are robust to an alternative treatment of unintelligible segments (e.g., retaining them as error tokens or analyzing the excluded subset separately).","section":"Appendix A.1.1"},{"comment":"The 'standard audit' results in §3 are not based on default configurations for two of the six services: Google uses the Chirp model and Microsoft uses the Azure Continuous model, both chosen after initial testing revealed poor default performance. This is a reasonable engineering decision, but it means the headline comparison is not strictly a comparison of out-of-the-box systems. The paper should state more prominently that the results are model-specific rather than service-default-specific, and should report the default-model WERs (or at least the observed failure modes) so readers can assess how much the exceptions affect the rank ordering and the disparity estimates.","section":"§2.2 and §3"}],"minor_comments":[{"comment":"The checkmark notation in Table 1 is hard to parse (e.g., '✓(✓)✓' appears as a single cell entry); a legend or separate columns for 'default' versus 'available but not default' would improve clarity.","section":"Table 1"},{"comment":"The text says 'Wilcoxon signed rank tests' in one place and 'Wilcoxon rank-sum tests' in another; these are different tests, and the figure caption should specify which was used for the pairwise comparisons.","section":"§4.2 and Figure 2"},{"comment":"The description of the whisper_normalizer-based cleaning is detailed, but the statement that 'we remove additional filler words not removed by the whisper_normalizer' lists filler tokens in a comma-separated inline list; presenting them as a table or code block would make the exact token set easier to reproduce.","section":"Appendix A.2.2"},{"comment":"References [49] and [50] appear to be the same survey paper duplicated with different formatting; please merge or distinguish them.","section":"References"},{"comment":"The sentence describing hallucination traits lists 'repetitions not present in the audio file' as a hallucination indicator; since WER insertions from stutters are explicitly distinguished from hallucinations earlier in §6, please clarify how this indicator was applied to avoid overlap with ordinary disfluency insertions.","section":"Appendix A.8.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is well suited to FAccT and the core WER-disparity result is convincing. The revision should focus on the hallucination-exclusivity claim, which currently exceeds the evidence, and on the unintelligible-segment exclusion, which is a plausible source of selection bias. If the authors can either systematically review the other services or soften the claim, and add a sensitivity analysis for UNK segments, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2506.08846. First, the main empirical result is solid: across six ASR services, aphasia speakers get WERs 6-10 points worse than matched controls, and the pattern holds under matched and unmatched samples, weighted and unweighted aggregation, and several standardization variants. Second, the more dramatic claim—that Whisper is the only service that hallucinates and that hallucinations concentrate in aphasia speech—is less secure than it looks.\n\nWhat's genuinely new: this is the first multi-service ASR audit on AphasiaBank data, and the subgroup breakdown by aphasia type is useful. The three-pitfall framework is not conceptually novel—disaggregated evaluation, metric suites, and community input all have prior art—but the paper makes those abstract recommendations concrete and shows they matter empirically. They also release code on GitHub, which is good practice.\n\nThe soft spots. The hallucination exclusivity claim rests on spot checks for five of the six services, not the systematic manual review done for Whisper. The candidate-selection thresholds could miss hallucinations that don't push a file to extreme metric values, so the asymmetry may be an artifact of differential review effort. The stress-test note is right about this. Also, the ground-truth cleaning drops all segments containing unintelligible tokens ('xxx'), which likely excludes the most severely affected aphasia speech; that could understate the true disparity rather than inflate it, but it's a limitation the authors should state plainly. The community survey is n=7, which they acknowledge; it supports the standardization preference but doesn't generalize. The non-default models for Google and Microsoft are a reasonable engineering choice, but readers should know the 'default' label is approximate.\n\nNone of these sink the central WER result, which is robust. But the hallucination part of the abstract overstates the evidence. The paper deserves a serious referee; I'd accept it conditionally and ask for either systematic hallucination review of all services or a clearly hedged claim.","headline":"Robust WER audit of six ASR systems on aphasia speech shows a real disparity, but the 'only Whisper hallucinates' claim rests on spot checks and needs to be hedged or systematically verified.","tokens_in":37218,"tokens_out":2000,"would_cite":true,"duration_ms":22756,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard speech-to-text audits systematically understate how badly aphasia speech is transcribed.","keywords":["automatic speech recognition","algorithmic auditing","aphasia","word error rate","hallucination","text standardization","disaggregated evaluation","speech accessibility"],"falsifier":"Re-run the audit keeping the segments the pipeline dropped (those containing unintelligible words), score them against a reference that preserves or clinically adjudicates those words, and check whether the 6–10 percentage point WER gap and the 53-of-56 hallucination concentration persist. If the gap shrinks substantially, the reported disparity is an artifact of ground-truth cleaning rather than a property of the ASR services.","tokens_in":36145,"feed_emoji":"🎙️","tokens_out":5385,"duration_ms":63633,"temperature":0.7,"pith_summary":"This paper argues that standard audits of automatic speech recognition (ASR) have three blind spots: they fix a single text-cleaning method, report only broad group averages, and rely on a single error metric, Word Error Rate (WER). Using clinical interview speech from people with aphasia and a control group, it shows that all six tested services transcribe aphasia speech with WERs 6–10 percentage points worse, with every difference statistically significant, and that fabricated “hallucinations” occurred almost exclusively for aphasia speakers. The paper proposes a community-driven audit framework: choose text standardization based on what affected speakers actually prefer, report performance for subgroups and acoustic conditions, and use a suite of metrics that includes hallucination rate. If the paper is right, standard audits both understate harm to speech-disordered users and can mislead about which service is best for them.","feed_headline":"Speech-to-text errors jump 6-10 points for aphasia speakers","feed_subtitle":"Six services all transcribe aphasia speech worse, and nearly all hallucinations hit those speakers; the paper names three audit fixes.","key_machinery":"The carrying mechanism is a three-part audit design. First, text standardization is varied across five levels, from the original transcript through removal of fillers, fragments, repeated words, and repeated phrases; the paper shows WER and even service rankings change with that choice, and it uses the aphasia community’s stated preference for the most cleaned version as the primary reference. Second, performance is disaggregated by aphasia fluency, Boston-classification type, gender, race, and acoustic covariates—nonvocal duration share (a dysfluency proxy from voice activity detection) and background noise energy—analysed with regressions clustered on speakers. Third, evaluation uses a metric suite beyond WER$, = (S+I+D)/N$: CER, BLEU, ROUGE-1/2/L, METEOR, WIL, RIL, insertion rate, and a manually verified binary hallucination indicator. Each part is designed to catch a class of error that the other parts miss.","core_discovery":"Across six commercial ASR services, transcriptions of speech from people with aphasia consistently have WERs 6–10 percentage points higher than control speakers (for example, 0.17 versus 0.09 for the worst-performing service and 0.12 versus 0.06 for the best), with all differences significant at $p<0.001$. Among 56 confirmed hallucinations in the open-weight Whisper model, 53 occurred for aphasia speakers, and an audio-manipulation experiment produced more Whisper hallucinations for aphasia speech than for control speech. The methodological claim is that standard audit practices hide this harm: a single text-standardization choice can reverse which service ranks best, treating “aphasia” as one group hides that non-fluent and Global aphasia are far worse (average WER 0.21 and 0.305, versus 0.07 for controls), and WER cannot distinguish fabricated hallucinated content from ordinary insertion errors. The paper concludes that audits should vary standardization in line with community preferences, disaggregate subgroups and acoustic covariates, and report a metric suite that includes hallucination rate.","pith_inferences":["The three pitfalls likely generalize beyond aphasia: any speech community with disfluencies, non-standard dialects, or clinician-specific transcription norms should be audited with multiple standardizations, disaggregated subgroups, and hallucination metrics, but that extension is ours, not the paper’s.","Because the paper’s ground-truth cleaning removes every segment containing an unintelligible word, the reported WER gap is probably a lower bound; an audit that retains or clinically adjudicates those segments could find an even larger disparity.","A testable extension of the hallucination result is to apply the paper’s audio-manipulation treatments (leading silence, white noise, early cutoff) to other large ASR models; the paper shows these manipulations raise Whisper hallucination rates and affect aphasia speech more, but whether other models behave the same is unknown.","The community-preference finding suggests ASR standardization should become a user-facing option rather than a hidden default, since preferences varied with recovery stage and purpose; this design implication goes beyond what the paper demonstrates."],"forward_implications":["If the central claim holds, single-metric, single-standardization audits can produce unstable service rankings: the paper shows Whisper significantly outperforming Amazon under minimal cleaning but Amazon significantly outperforming Whisper under the community-preferred cleaning.","Disaggregation by aphasia type changes the conclusion: non-fluent and Global aphasia show much higher WERs, so any audit that reports only “aphasia versus control” conceals the speakers who need the most accurate transcription.","Hallucination rate should be a standard reporting metric, because WER and insertion rate cannot distinguish fabricated content from stutter-like repetitions, and in this study hallucinations were nearly exclusive to aphasia speakers.","Acoustic covariates (nonvocal pause share and background noise) are measurable confounders that raise both WER and hallucination likelihood, but the aphasia indicator remains the dominant factor after adjustment.","Community-preferred cleaning is not in tension with competitive WER: for half of the services, removing repeated words made no significant WER difference, so audits need not sacrifice comparability to honor user preferences."],"supporting_citations":[{"why":"Supplies the standard audit procedure and text-cleaning conventions that the paper then critiques and extends.","marker":"[57]"},{"why":"Defines ASR hallucinations and their harms, provides the hallucination taxonomy used for manual review, and supplies the prior Whisper hallucination rate the paper compares against.","marker":"[56]"},{"why":"Supplies the clinical interview audio and human reference transcriptions that form the ground truth for both aphasia and control speakers.","marker":"[70]"},{"why":"Extends the audit method to another clinical speech population, establishing the comparison baseline for disabled-speaker ASR evaluation.","marker":"[113]"},{"why":"Introduces the Whisper model evaluated in the case study and the whisper_normalizer text-cleaning package used for standardization.","marker":"[86]"},{"why":"Provides the MAPSSWE statistical test used to determine whether service rankings change across text-standardization levels.","marker":"[36]"}],"fun_headline_variants":["Aphasia speech errors 6–10 WER points higher than controls","Standard ASR audits mask worse aphasia transcription quality","Whisper hallucinations target aphasia speech: 53 of 56","One-size-fits-all ASR audits hide aphasia disparities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cleaned human transcriptions used as the reference for both groups are correct and unbiased, even though every audio segment containing a word marked unintelligible was thrown out, and that removal may exclude the most severe aphasia speech and distort the measured disparity.","fun_headline_variants_meta":{"raw":{"variants":["Aphasia speech errors 6–10 WER points higher than controls","Standard ASR audits mask worse aphasia transcription quality","Whisper hallucinations target aphasia speech: 53 of 56","One-size-fits-all ASR audits hide aphasia disparities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00061,"raw_usage":{"total_tokens":2860,"prompt_tokens":986,"completion_tokens":1874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1797}},"tokens_in":602,"tokens_out":1874,"duration_ms":14179,"temperature":1.0,"reasoning_tokens":1797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:00:08.342421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the audit keeping the segments the pipeline dropped (those containing unintelligible words), score them against a reference that preserves or clinically adjudicates those words, and check whether the 6–10 percentage point WER gap and the 53-of-56 hallucination concentration persist. If the gap shrinks substantially, the reported disparity is an artifact of ground-truth cleaning rather than a property of the ASR services.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standard audit procedure and text-cleaning conventions that the paper then critiques and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ASR hallucinations and their harms, provides the hallucination taxonomy used for manual review, and supplies the prior Whisper hallucination rate the paper compares against."},{"cited_title":"MacWhinney, D","cited_arxiv_id":null,"evidence_quote":"Supplies the clinical interview audio and human reference transcriptions that form the ground truth for both aphasia and control speakers."},{"cited_title":"Speaker 0: xxx. Speaker 1:xxx","cited_arxiv_id":null,"evidence_quote":"Extends the audit method to another clinical speech population, establishing the comparison baseline for disabled-speaker ASR evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Whisper model evaluated in the case study and the whisper_normalizer text-cleaning package used for standardization."}],"review_version":1}