Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Fine-Tuning ASR for Stuttered Speech: Personalized vs. Generalized Approaches

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning Whisper on even small amounts of a single speaker's stuttered speech cuts word error rates roughly in half compared with a generalized model, bringing transcription accuracy close to that of fluent speech.

desk verdict A useful new stuttered-speech corpus and a clear within-domain personalization result, but the generalized-vs-personalized comparison is confounded by corpus mismatch and the paper overstates what E5 shows. read the letter →

arxiv 2506.00853 v2 pith:7KO6VRZB submitted 2025-06-01 cs.SD eess.AS

Dena Mujtaba , Nihar Mahapatra This is my paper
classification cs.SDeess.AS
keywords stutteringautomaticspeechrecognitionpersonalizationWhisperLoRAlow-rankadaptationworderrorratespontaneous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether automatic speech recognition for stuttered speech works better as one model for all users or as models shaped to each individual's voice. The authors fine-tune Whisper-small with low-rank adaptation (LoRA), comparing a generalized model trained on the FluencyBank corpus with personalized models trained on a new dataset, HeardAI, of read and spontaneous speech from 18 people who stutter across 18 voice-AI use categories. Their central finding is that the personalized models win: using small amounts of a participant's own speech cuts word error rate from 16.12% to 9.39% on read speech and from 18.39% to 9.81% on spontaneous speech relative to the generalized model, with the read-plus-spontaneous combination working best. The point matters because voice assistants, captioning, and automated interviews remain largely inaccessible to people who stutter, and a short per-user enrollment could close most of the accuracy gap.

What carries the argument

Low-rank adaptation (LoRA) is the mechanism that makes the comparison feasible: instead of updating Whisper's weights, the method adds a trainable low-rank correction $\Delta W = BA$ with $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ to the frozen pre-trained weights, using rank $r = 128$, scaling $\alpha = 256$, and dropout $p = 0.05$ on all linear layers. This keeps the number of trainable parameters low enough that dozens of per-participant models can be trained on a single GPU and reduces the overfitting risk of fine-tuning on tiny per-speaker datasets. The other load-bearing piece is the HeardAI dataset itself: read prompts and spontaneous job-interview answers from 18 participants across 18 voice-AI categories, transcribed by speech-language pathologists, which lets the authors test read-to-spontaneous generalization and per-category effects.

What would settle it

Run the same Experiment 5 protocol on an independent or public cohort of speakers who stutter: if per-speaker fine-tuned models fail to beat a generalized model trained on that cohort's combined speech, the central claim falls. A sharper check is to give the generalized model the full HeardAI dataset, all 18 speakers and both speech types, and compare per-speaker word error rates; if the all-speaker model matches or beats the personalized models, the reported gap is an artifact of the 2.21-hour FluencyBank baseline.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that parameter-efficient fine-tuning of a pre-trained ASR model on a single speaker's stuttered speech yields substantially better transcription than fine-tuning on a multi-speaker corpus. In Experiment 5, a personalized Whisper-small model trained with LoRA on each participant's read and spontaneous HeardAI speech lowered word error rate to 9.39% on held-out read prompts and 9.81% on held-out spontaneous speech, down from 16.12% and 18.39% for the generalized FluencyBank model, with character error rate dropping to about half. The improvement appears with as little as one quarter of a participant's read prompts and then tapers, and it holds across every prompt category while shrinking the spread of error rates across participants. The authors also report that models fine-tuned on spontaneous speech alone, despite having fewer training examples, perform comparably to models fine-tuned on read speech alone, which they attribute to stuttering being more frequent and variable in spontaneous speech.

Load-bearing premise

The load-bearing premise is that the 18 HeardAI participants and the 12 FluencyBank speakers, after excluding problematic recordings, represent the broader population of people who stutter in everyday voice-AI settings, so if this sample is not representative, the measured advantage of personalization may not transfer to other speakers.

Editorial extensions

If this is right

  • A voice assistant could halve its error rate for a user who stutters after only a few minutes of enrollment prompts from that user.
  • Personalized fine-tuning lowers both the average word error rate and the variation across speakers, so the users with the highest initial error rates gain the most.
  • Training on a mix of read and spontaneous speech works best, and each minute of spontaneous speech contributes more to accuracy than a minute of read speech.
  • The best personalized models reach word error rates near those reported for non-stuttered speech, indicating that personalization can close most of the documented accessibility gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A deployment path the paper leaves implicit: because LoRA adapters are small and the base Whisper weights stay frozen, personalization could be offered as a brief per-user enrollment whose adaptation is stored locally rather than uploading everyone's stuttered speech to a server.
  • The same per-speaker low-rank recipe is a natural next test for other speech conditions with scarce per-user data, such as dysarthria, cluttering, or strong regional accents.
  • The cross-validation already holds out whole voice-AI categories per participant, so what remains untested is temporal and environmental stability; a longitudinal follow-up would show whether one adapter keeps working across months, microphones, and settings.
  • Because the HeardAI dataset is not publicly released, the reported effect sizes rest on a single 18-participant study; a public replication with a larger, more diverse cohort would establish how well the roughly halved word error rate generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper fine-tunes Whisper-small with LoRA on stuttered speech, comparing generalized models (FluencyBank) with personalized models trained on a newly collected HeardAI dataset of read and spontaneous speech from 18 participants. The authors report that personalized models reduce WER on held-out HeardAI eval, with Experiment 5 (read+spontaneous personalized fine-tuning) achieving the largest gains (16.12% to 9.39% on read; 18.39% to 9.81% on spontaneous). They conclude that small amounts of participant-specific speech significantly improve personalized ASR over generalized approaches.

Significance. If the main comparison were clean, the paper would provide useful evidence that small personalized datasets can improve ASR for stuttered speech, and the newly collected HeardAI dataset is a valuable community resource despite not being publicly released. The authors also deserve credit for using held-out evaluation, per-participant analysis, LoRA fine-tuning, and multiple speech contexts. However, the central claim as stated is not fully supported by the experiments: the principal comparison is confounded by training-domain mismatch, and the paper's own Experiment 3 contradicts the blanket wording of the abstract and conclusion. The significance of the work therefore depends on an additional controlled baseline that the current manuscript lacks.

major comments (3)
  1. [Section 2.5, Experiment 1 vs. Experiment 5; Table 1] The headline comparison between Gen-E1 and WS-P-E5 varies two factors at once: personalization and training-domain match. Gen-E1 is fine-tuned only on FluencyBank, while WS-P-E5 is fine-tuned on the same HeardAI prompts and recording conditions used for evaluation. Part or all of the WER gap from 16.12% to 9.39% on read speech and from 18.39% to 9.81% on spontaneous speech could therefore be a corpus/domain effect rather than an effect of adapting to the individual speaker. A generalized model trained on pooled HeardAI data from other speakers (leave-one-speaker-out) is needed to separate speaker-specific benefit from in-domain training benefit. Without such a baseline, the abstract's causal claim that personalization itself improves accuracy is not established.
  2. [Section 3, Table 1; Abstract and Section 4] The paper's own Experiment 3 contradicts the blanket statement that 'fine-tuning ASR models using even small amounts of participant-specific speech data significantly improves transcription accuracy compared to generalized models.' WS-P-E3, trained only on each participant's read prompts, increases spontaneous WER to 31.86% from the Gen-E1 value of 18.39%. The positive conclusion therefore rests entirely on the combined read+spontaneous setup of Experiment 5. The abstract and conclusion should be qualified to state that the improvement is observed when both read and spontaneous personalization data are used, and the read-only case should be discussed as a failure mode rather than glossed over.
  3. [Section 3, Table 1] The word 'significantly' is used without any inferential statistics. Reported values are means across participants, but no confidence intervals, standard errors, paired significance tests, or per-participant distributions are given for the WER/CER comparisons. Given the large inter-participant variability visible in Figure 4, the claim of significance needs a paired analysis across the 18 HeardAI participants (or an equivalent repeated-measures procedure), not just a comparison of mean WER values.
minor comments (5)
  1. [Figure 2 caption] The caption contains apparent typos: 'Hopitality' should be 'Hospitality' and 'Video Conferencing and Remote Work' appears without proper capitalization in the list of categories.
  2. [References] Reference [10] lists 'Whister' but the system is called 'Whisper' in the text; reference [22] contains the typo 'phenomonen' for 'phenomenon'.
  3. [Section 2.5 / Table 1] The table reports means across participants but does not state the number of participants or utterances used for each row; providing N per condition would clarify the comparison, especially for Experiments 3-5 where per-participant cross-validation is used.
  4. [Section 2.2.2] The statement that the HeardAI dataset cannot be publicly released is relevant for reproducibility; please state whether the fine-tuning code and evaluation scripts will be released so that others can re-run the comparisons on their own stuttered-speech data.
  5. [Section 2.3] The LoRA hyperparameters r=128, alpha=256, and dropout p=0.05 are reported as 'previously determined to be effective,' but no ablation or sensitivity analysis is provided; since these affect the central result, a sentence explaining the provenance of these settings would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fine-tuning results come from held-out evaluation, and self-citations are contextual rather than load-bearing.

full rationale

This paper is an empirical benchmarking study, not a derivation chain, so there is no predicted quantity that reduces to a fitted input by construction. The central result is obtained by held-out evaluation: Experiments 2-5 split each participant's HeardAI prompts into training and evaluation folds and report WER/CER on unseen prompts, so the personalized models are not fitted to the test data. The generalized baseline (Gen-E1) is a separately trained model on FluencyBank, and the personalized models are fine-tuned from that baseline with LoRA. The authors' self-citations (e.g., reference [2]) support only contextual background claims about ASR bias against disfluent speech and do not define or justify the personalized-versus-generalized comparison. The LoRA hyperparameters are attributed to external prior work by a different team (reference [15]), not to the authors' own prior results. The most serious limitation is a validity confound: Gen-E1 is trained only on FluencyBank while the personalized E5 model is trained on in-domain HeardAI, so the headline comparison varies both personalization and training corpus simultaneously. That is a potential overstatement of the personalization effect and a legitimate experimental-design concern, but it is not circularity: the reported WER values are not derived from the claim being tested, no equation is equivalent to the hypothesis by construction, and no load-bearing self-citation chain forces the outcome. Therefore, no circular step is identified and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central result is empirical and rests on the accuracy of SLP transcriptions, the representativeness of two small datasets, and the independence of held-out evaluation folds. No new physical or mathematical entities are introduced. LoRA hyperparameters inherited from prior work are listed for completeness.

free parameters (5)
  • LoRA rank r = 128
    Chosen following Müller-Eberstein et al. [15], not fitted to the HeardAI data; held constant across all models.
  • LoRA scaling factor alpha = 256
    Taken from [15]; held constant across all compared models.
  • LoRA dropout p = 0.05
    Taken from [15]; held constant across all compared models.
  • Learning rate = 1e-5
    Chosen by the authors; same for all fine-tuning runs.
  • Training epochs = 20 (experiments 1, 3-5); 7 (experiment 2)
    Chosen by the authors; experiment 2 uses fewer epochs to study data scaling, not final accuracy.
assumptions (5)
  • domain assumption CHAT transcripts and SLP annotations are accurate ground truth for stuttered speech.
    WER and CER are computed against these transcripts (Section 2.2); any annotation noise inflates measured error rates.
  • domain assumption The held-out evaluation prompts are independent of training prompts due to category-based splitting.
    Section 2.5 states splits are by prompt category to prevent context overlap; this assumes category separation prevents content leakage.
  • domain assumption The 18 HeardAI and 12 FluencyBank speakers represent the broader population of people who stutter.
    Section 2.2.2 describes a small, non-public sample with unspecified recruitment and exclusion criteria; the central generalization depends on this.
  • domain assumption Whisper-small is a suitable base model for stuttered-speech fine-tuning.
    The authors fine-tune only Whisper-small; results may not transfer to larger or other architectures.
  • domain assumption Spontaneous speech contains more stuttering than read speech (Rasskazov and Rasskazova, 2007).
    Used in Section 3 to interpret why spontaneous-only training (Experiment 4) is effective; cited to prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-Tuning ASR for Stuttered Speech: Personalized vs. Generalized Approaches." pith.science (2026). https://pith.science/paper/7KO6VRZB

@misc{pith2026250600853,
  author       = {Pith},
  title        = {Pith review of: Fine-Tuning ASR for Stuttered Speech: Personalized vs. Generalized Approaches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KO6VRZB}},
  note         = {Machine review of arXiv:2506.00853}
}
read the original abstract

Stuttering -- characterized by involuntary disfluencies such as blocks, prolongations, and repetitions -- is often misinterpreted by automatic speech recognition (ASR) systems, resulting in elevated word error rates and making voice-driven technologies inaccessible to people who stutter. The variability of disfluencies across speakers and contexts further complicates ASR training, compounded by limited annotated stuttered speech data. In this paper, we investigate fine-tuning ASRs for stuttered speech, comparing generalized models (trained across multiple speakers) to personalized models tailored to individual speech characteristics. Using a diverse range of voice-AI scenarios, including virtual assistants and video interviews, we evaluate how personalization affects transcription accuracy. Our findings show that personalized ASRs significantly reduce word error rates, especially in spontaneous speech, highlighting the potential of tailored models for more inclusive voice technologies.

Figures

Figures reproduced from arXiv: 2506.00853 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 4
Figure 4. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 24 canonical work pages

  1. [1]

    Fine-Tuning ASR for Stuttered Speech: Personalized vs. Generalized Approaches

    Introduction V oice AI, systems driven by speech input, have become ubiqui- tous in contemporary life due to recent significant performance improvements. However, these technologies remain largely in- accessible to people who stutter, who constitute approximately 1% of the global population [1]. Such individuals encounter substantial barriers in routine a...

    work page Pith review arXiv 2025

  2. [2]

    Lock the front door

    Methodology 2.1. Automatic Speech Recognition Model: Whisper We utilize Whisper [19], a sequence-to-sequence transformer- based model capable of multilingual and multitask speech tran- scription. Whisper processes audio in 30-second segments via its encoder, producing a sequential representation used by its decoder for transcription. Trained on over 680k ...

    work page

  3. [3]

    Smart Home Automa- tion,

    Results and Discussion In Table 1, we present the results for the untrained Whisper- small model on the HAI dataset. Both WER and CER are notably high for read and spontaneous speech samples. Fig- ure 2 further illustrates these results by prompt category, high- lighting that certain categories, notably “Smart Home Automa- tion,” “Language Learning,” “Tra...

    work page

  4. [4]

    Conclusion In this paper, we investigated the performance differences be- tween generalized and personalized ASR approaches for stut- tered speech, specifically through fine-tuning large pre-trained models. Using a newly aggregated dataset covering diverse voice-AI scenarios, we analyzed how transcription accuracy varied across different applications and ...

    work page

  5. [5]

    Na- tional Science Foundation under Grant No

    Acknowledgements This material is based upon work supported by the U.S. Na- tional Science Foundation under Grant No. 2345086. We thank the HeardAI team members for their assistance with dataset col- lection and transcription

    work page

  6. [6]

    Epidemiology of stuttering: 21st cen- tury advances,

    E. Yairi and N. Ambrose, “Epidemiology of stuttering: 21st cen- tury advances,” Journal of fluency disorders , vol. 38, no. 2, pp. 66–87, 2013

    work page 2013

  7. [7]

    Lost in transcription: Identifying and quantifying the accuracy biases of automatic speech recogni- tion systems against disfluent speech,

    D. Mujtaba, N. Mahapatra, M. Arney, J. Yaruss, H. Gerlach- Houck, C. Herring, and J. Bin, “Lost in transcription: Identifying and quantifying the accuracy biases of automatic speech recogni- tion systems against disfluent speech,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    work page 2024

  8. [8]

    From user percep- tions to technical improvement: Enabling people who stutter to better use speech recognition,

    C. Lea, Z. Huang, J. Narain, L. Tooley, D. Yee, D. T. Tran, P. Georgiou, J. P. Bigham, and L. Findlater, “From user percep- tions to technical improvement: Enabling people who stutter to better use speech recognition,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , 2023, pp. 1–16

    work page 2023

Show all 27 references
  1. [9]

    Re-envisioning remote meetings: Co-designing inclusive and empowering videoconferencing with people who stutter,

    J. Li, S. Wu, and G. Leshed, “Re-envisioning remote meetings: Co-designing inclusive and empowering videoconferencing with people who stutter,” in Proceedings of the 2024 ACM Designing Interactive Systems Conference, 2024, pp. 1926–1941

    2024

  2. [10]

    Ethical considerations in AI-based recruitment,

    D. F. Mujtaba and N. R. Mahapatra, “Ethical considerations in AI-based recruitment,” in2019 IEEE International Symposium on Technology and Society (ISTAS). IEEE, 2019, pp. 1–7

    2019

  3. [11]

    Stut- tering and labor market outcomes in the United States,

    H. Gerlach, E. Totty, A. Subramanian, and P. Zebrowski, “Stut- tering and labor market outcomes in the United States,” Journal of Speech, Language, and Hearing Research , vol. 61, no. 7, pp. 1649–1663, 2018

    2018

  4. [12]

    The influence of workplace discrimination and vigilance on job satis- faction with people who stutter,

    L. W. Plexico, M.-B. Hamilton, H. Hawkins, and S. Erath, “The influence of workplace discrimination and vigilance on job satis- faction with people who stutter,” Journal of Fluency Disorders , vol. 62, p. 105725, 2019

    2019

  5. [13]

    Disordered speech data collection: Lessons learned at 1 million utterances from Project Euphonia

    R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. C. Nelson et al., “Disordered speech data collection: Lessons learned at 1 million utterances from Project Euphonia.” in Interspeech, vol. 2021, 2021, pp. 4833–4837

    2021

  6. [14]

    Variability of stuttering: Behav- ior and impact,

    S. E. Tichenor and J. S. Yaruss, “Variability of stuttering: Behav- ior and impact,” American Journal of Speech-Language Pathol- ogy, vol. 30, no. 1, pp. 75–88, 2021

    2021

  7. [15]

    Whister: Using Whisper’s repre- sentations for stuttering detection,

    V . Changawala and F. Rudzicz, “Whister: Using Whisper’s repre- sentations for stuttering detection,” inInterspeech 2024, 2024, pp. 897–901

    2024

  8. [16]

    SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter,

    C. Lea, V . Mitra, A. Joshi, S. Kajarekar, and J. P. Bigham, “SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter,” in ICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6798–6802

    2021

  9. [17]

    Enhancing ASR for stuttered speech with limited data using detect and pass,

    O. Shonibare, X. Tong, and V . Ravichandran, “Enhancing ASR for stuttered speech with limited data using detect and pass,”arXiv preprint arXiv:2202.05396, 2022

    2022 arXiv

  10. [18]

    Have disfluency-type measures contributed to the understanding and treatment of developmental stuttering?

    J. Einarsd ´ottir and R. J. Ingham, “Have disfluency-type measures contributed to the understanding and treatment of developmental stuttering?” 2005

    2005

  11. [19]

    Clut- tering updated,

    K. O. St. Louis, L. J. Raphael, F. L. Myers, and K. Bakker, “Clut- tering updated,” The ASHA Leader, vol. 8, no. 21, pp. 4–22, 2003

    2003

  12. [20]

    Hypernetworks for personalizing ASR to atypical speech,

    M. M ¨uller-Eberstein, D. Yee, K. Yang, G. V . Mantena, and C. Lea, “Hypernetworks for personalizing ASR to atypical speech,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 1182–1196, 2024

    2024

  13. [21]

    Assessing ASR model quality on disordered speech using BERTScore,

    J. Tobin, Q. Li, S. Venugopalan, K. Seaver, R. Cave, and K. Tomanek, “Assessing ASR model quality on disordered speech using BERTScore,”arXiv preprint arXiv:2209.10591, 2022

    2022 arXiv

  14. [22]

    LoRA: Low-rank adaptation of large lan- guage models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large lan- guage models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  15. [23]

    FluencyBank: A new resource for fluency research and practice,

    N. B. Ratner and B. MacWhinney, “FluencyBank: A new resource for fluency research and practice,” Journal of fluency disorders , vol. 56, pp. 69–80, 2018

    2018

  16. [24]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in International Conference on Machine Learning . PMLR, 2023, pp. 28 492–28 518

    2023

  17. [25]

    Tools for analyzing talk part 1: The CHAT tran- scription format,

    B. MacWhinney, “Tools for analyzing talk part 1: The CHAT tran- scription format,” Carnegie.[Google Scholar], vol. 16, 2017

    2017

  18. [26]

    Normalization and pre-tokenization,

    HuggingFace, “Normalization and pre-tokenization,” https:// huggingface.co/learn/nlp-course/chapter6/4

  19. [27]

    Why do so many stutterers fail to stutter when alone and how can this phenomonen be used in treatment,

    I. Rasskazov and N. Rasskazova, “Why do so many stutterers fail to stutter when alone and how can this phenomonen be used in treatment,” in International Stuttering Awareness Day Online Conference, 2007

    2007

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.