REVIEW 4 major objections 5 minor 13 references
Feasibility of Using Automatic Speech Recognition with Voices of Deaf and Hard-of-Hearing Individuals
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper finds that commercial speech recognition mis-transcribes roughly three of every four words spoken by deaf and hard-of-hearing users, about 78% word error rate, versus 18% for hearing speech, making current voice interfaces…
desk verdict A useful benchmark of one ASR API on deaf speech, undermined by an overbroad 'interfaces are unusable' claim that the data don't support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a measurement pipeline: recordings of deaf speakers reading Clarke Sentences, a standardized intelligibility test, are fed into a commercial speech-recognition API, and the resulting transcripts are scored with a standard word error rate toolkit. The Clarke score provides a human intelligibility baseline that separates 'good,' 'mediocre,' and 'poor' deaf speech, and the WER comparison quantifies how far commercial ASR lags behind a human listener.
What would settle it
Run a larger random sample of deaf speakers through the same commercial ASR under identical recording conditions as hearing speakers and compare word error rates. If the average deaf WER drops below roughly 40% for intelligible speakers, or if any large deaf cohort matches hearing-level WER, the claim that voice interfaces are unusable by DHH people would be weakened; if the 78% versus 18% gap persists across services and newer models, the claim is strengthened.
Extended reading notes
Core claim
The central discovery is that commercial automatic speech recognition, trained mostly on hearing voices, does not transfer to deaf speech. Across 45 deaf speech samples the word error rate averaged 77% (which the paper reports as approximately 78%), while five hearing speakers in a noisier environment had an 18% average. Performance varied widely among deaf speakers, and even the 'good' group with intelligibility scores of 40 and above averaged 53% WER. The authors interpret this as showing that current voice interfaces are effectively unusable for DHH users, and that a much larger and more varied database of deaf speech would be needed to train ASR to an acceptable level.
Load-bearing premise
The conclusion depends on the 45 deaf speech samples selected by one naive listener being representative of DHH speakers generally, and on the deaf and hearing recordings being comparable even though the hearing group was recorded in a noisy lab with only five subjects.
Editorial extensions
If this is right
- Voice-only devices such as smart speakers and phone assistants will continue to mishear deaf users, so they cannot serve as a primary interface without dedicated training on deaf speech.
- Producing deaf-inclusive ASR training data is a hard practical problem because the DHH population is small and hugely varied, so per-user adaptation or alternative input pathways may be necessary.
- Human intelligibility ratings do not predict ASR accuracy at the top end (53% WER for 'good' speech), so usability testing for voice interfaces must use actual ASR output, not human ratings.
- An automated WER-based feedback tool could replace the Clarke test for practical purposes, giving DHH users a realistic measure of how well voice interfaces understand them.
Reading between the lines
- The 78% WER probably understates the functional problem for command interfaces, because a mis-recognized command may still be a recognized word but the wrong one; task-completion rates might be even lower than word accuracy suggests.
- Testing the same deaf speakers across several commercial ASR services (phone assistants, smart speakers, dictation) could reveal whether the gap is intrinsic to deaf speech or exaggerated by one vendor's training data.
- A plausible path forward, not explored in the paper, is few-shot personalization: modern ASR models could adapt to an individual deaf speaker with a small amount of their own speech during device setup.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a single empirical measurement of automatic speech recognition (ASR) performance on deaf and hard-of-hearing (DHH) speech. The authors sampled 45 Clarke Sentences recordings from a 650-speaker DHH dataset, stratified by a naive listener into good, mediocre, and poor intelligibility groups, and transcribed them with the Microsoft Translator Speech API. They also transcribed recordings from five hearing speakers reading the same sentence lists, recorded in a noisy lab. Word error rates were computed with the NIST SCTK toolkit. The reported results are an average 77% WER for the DHH samples (53% for the 'good' subgroup) versus 18% for the hearing baseline. The authors conclude that current speech-controlled interfaces such as Amazon Echo and Apple's Siri are not usable by DHH people and that alternative input methods or substantial ASR improvements are needed.
Significance. If the reported measurement is valid, the finding that a mainstream commercial ASR system mis-transcribes about three out of four words of DHH speech is an important accessibility data point. The paper's strengths are its use of a real DHH speech corpus with clinician-assigned intelligibility scores, a standard scoring tool (SCTK), and a stratified attempt to include good, mediocre, and poor speakers. The comparison between intelligibility ratings and WER is a useful contribution. However, the paper's central conclusion is broader than the experiment supports: only one specific API (Microsoft Translator) is tested on sentence-length read speech, while the conclusion is about all current speech-controlled interfaces. The sampling of 45 speakers by a single naive listener and the unmatched hearing baseline also limit the quantitative claim. The main value of the paper is as a preliminary negative result that motivates more careful evaluation of ASR accessibility for DHH users.
major comments (4)
- [Abstract and Section 4] The conclusion that 'current speech-controlled interfaces are not usable by DHH people' is not supported by the data. The experiment tests only the Microsoft Translator Speech API on 10-syllable Clarke Sentences, not commercial voice interfaces such as Amazon Echo or Siri, which typically use short command grammars, command-specific language models, and confirmation/repair loops. The authors' own cited reference [3] reports a 13% WER for a constrained single-digit task with deaf speakers who have poor intelligibility, demonstrating that narrow recognition tasks can perform far better than free-form sentence transcription. The paper should either test actual voice-command interfaces or explicitly limit the claim to sentence-level dictation with the tested API.
- [Section 3.2] The 45 DHH samples were chosen by one naive listener rather than randomly sampled from the 650-speaker dataset, with 15 samples labeled good (intelligibility 40+), 16 mediocre (30-40), and 14 poor (10-30). This selection protocol may bias the sample toward or away from speakers who are typical of the broader DHH population, and no inter-rater reliability or selection criteria are reported. Because the paper's headline 77% WER depends on the composition of these 45 samples, the authors should describe the selection procedure in detail, provide a random or systematic sampling strategy, or at least give the intelligibility distribution of the full dataset to allow readers to judge representativeness.
- [Section 3.1] The hearing baseline is based on only five speakers recorded with a cell phone in a noisy lab, whereas the DHH recordings come from an existing corpus recorded under different, unspecified conditions. The authors state that if the hearing speakers were recorded in the same setting as the DHH samples, they would expect an even lower WER, but this does not remove the recording-condition mismatch from the central 77% versus 18% comparison. The paper should report the recording conditions of both sets, ideally use matched recordings, and provide confidence intervals or a statistical comparison for the main difference, given the small n and large variance among DHH speakers.
- [Sections 3.1 and 3.2] No measures of statistical uncertainty are given for the central estimates. The paper reports only the average WER for each group and a t-test between the good and mediocre DHH subgroups; it does not report confidence intervals for the 77% overall DHH WER or the 18% hearing WER. Given the small sample sizes and the high variability evident in Figure 2, the authors should provide confidence intervals or standard errors for all reported means, ideally with a bootstrap or similar resampling analysis.
minor comments (5)
- [Section 2.2] The statement that the Microsoft Translator API 'matches other similar transcription software' is supported only by press announcements in references [6-8], not by comparative evaluations. The authors should either cite peer-reviewed comparisons or soften the claim.
- [Section 3.2] The text says 'as shown in Figure 3.2' but the figure is labeled 'Figure 3'; the cross-reference should be corrected.
- [Abstract] The sentence 'We show that current speech-controlled interfaces are not usable by DHH people' appears twice in the abstract; one occurrence should be removed.
- [Section 5] References [6], [7], and [8] are web/press items with incomplete bibliographic details; full access dates and publication venues should be added.
- [Figure 2] The axes and point markers in Figure 2 would benefit from explicit labels and a legend, since the reader must infer which markers correspond to 'good', 'mediocre', and 'poor' groups from the text.
Circularity Check
No circularity: the paper reports an empirical WER measurement using an external commercial ASR and an independent scoring toolkit, with no fitted parameters or self-citation chain that forces the result.
full rationale
The paper's central claim is a measured word error rate: deaf speech samples from an existing Clarke Sentences intelligibility dataset are fed to the Microsoft Translator Speech API, and the WER is computed with the NIST SCTK toolkit. There is no derivation or model whose output is fed back into its inputs. The intelligibility scores come from a speech pathologist's independent ratings, and the ASR transcriptions are produced by an external system trained on data unrelated to this study. The comparison group of hearing speakers is measured with the same external ASR, so the 78% versus 18% gap is an empirical observation rather than a quantity implied by construction. The main weaknesses are external-validity concerns: generalizing from one cloud transcription API to all consumer voice interfaces, and from 10-syllable Clarke Sentences to short command interactions. Those concerns attack the strength of the conclusion, not the independence of the measurement, and the paper itself even cites prior work showing a constrained digit task at 13% WER for deaf speech. No self-citation is load-bearing, no parameter is fitted and renamed as a prediction, and no known result is repackaged under new coordinates. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Microsoft Translator Speech API is representative of Siri, Alexa, and other speech-controlled interfaces
- domain assumption The 45-sample subset chosen by one naive listener represents the DHH speaker population in the 650-person dataset
- domain assumption The Clarke Sentences deaf recordings and the noisy-lab hearing recordings are comparable for WER comparison
- domain assumption Word error rate on read sentences is a valid proxy for usability of interactive voice commands
Cite this review
Pith. "Pith review of Feasibility of Using Automatic Speech Recognition with Voices of Deaf and Hard-of-Hearing Individuals." pith.science (2026). https://pith.science/paper/5Y7TE555
@misc{pith2026190901167,
author = {Pith},
title = {Pith review of: Feasibility of Using Automatic Speech Recognition with Voices of Deaf and Hard-of-Hearing Individuals},
year = {2026},
howpublished = {\url{https://pith.science/paper/5Y7TE555}},
note = {Machine review of arXiv:1909.01167}
}
read the original abstract
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce device costs and interactive friction. This transition has been hastened by the increasing capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Our findings show that current speech-controlled interfaces are not usable by deaf and hard of hearing people. Therefore, it might be wise to pursue other methods for deaf persons to deliver natural commands to computers.
Figures
Reference graph
Works this paper leans on
-
[3]
ANALYSIS RESULTS 3.1 WER for Hearing Speakers We calculated the ASR transcription and WER analysis results for five hearing subjects who read various lists from the same Clarke Sentences database, The speech samples were recorded with a cell phone in a noisy environment with background noise in a lab with many people speaking and computers, which is simila...
-
[1]
RELA TED WORK Prior research has investigated how the lack of a feedback loop for deaf people who cannot hear their own speaking results in poor speech quality due vowel errors, intonation errors, and length errors [1, 2]. ASR software is generally trained with speech samples from hearing people, which re- sults in very poor recognition of deaf speech. Ev...
-
[2]
METHODOLOGY 2.1 Deaf Speech Dataset We sampled from a subset of a large speech dataset of 650 deaf and hard of hearing (DHH) individuals at the Na- tional Technical Institute for the Deaf at Rochester Insti- tute of Technology, which has an enrollment of around 1100 deaf and hard of hearing students. The dataset consisted of samples taken from DHH individ...
work page Pith review arXiv 1909
-
[4]
They were significantly poorer in perfor- mance compared to hearing people under similar conditions
CONCLUSIONS The current WER of Microsoft Translator was too high for comfortable use. They were significantly poorer in perfor- mance compared to hearing people under similar conditions. There are a number of factors that have an impact on the accuracy of automatic speech recognition systems with deaf Figure 3: Group vs average WER speech. The results also...
-
[5]
M. S. Thirumalai and S. G. Gayathri. Speech of the hearing impaired. Central Institute of Indian Languages, 2004
work page 2004
-
[6]
Mary J. Osberger and Nancy S. McGarr. Speech production characteristics of the hearing impaired. Journal of Speech, Language and Hearing Research , 8:221–283, 1982
work page 1982
-
[7]
Deaf speech assessment using digital processing techniques
C Jeyalakshmi, V Krishnamurthi, and A Revathy. Deaf speech assessment using digital processing techniques. Signal & Image Processing: An International Journal (SIPIJ), 1(1):14–25, 2010
work page 2010
-
[8]
Phonetically sensitive discriminants for improved speech recognition
George R Doddington. Phonetically sensitive discriminants for improved speech recognition. In Acoustics, Speech, and Signal Processing, 1989. ICASSP-89., 1989 International Conference on , pages 556–559. IEEE, 1989
work page 1989
Show all 13 references
-
[9]
Samar and Dale Evan Metz
Vincent J. Samar and Dale Evan Metz. Criterion validity of speech intelligibility rating-scale procedures for the hearing-impaired population. Journal of Speech, Language, and Hearing Research , 31(3):307–316, 1988
1988
-
[10]
Microsoft researchers achieve speech recognition milestone, Sep 2016
Richard Eckel. Microsoft researchers achieve speech recognition milestone, Sep 2016
2016
-
[11]
Google’s speech recognition technology now has a 4.9% word error rate, May 2017
2017
-
[12]
Reaching new records in speech recognition, Mar 2017
George Saon. Reaching new records in speech recognition, Mar 2017
2017
-
[13]
National institute of standards and technology, June 2017
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.