REVIEW 4 major objections 5 minor 31 references
State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A Mamba-based state-space model plus fine-tuned self-supervised models achieves the lowest word error rates reported on the whispered-speech benchmarks wTIMIT and CHAINS.
desk verdict Useful whispered-ASR results and an interesting zero-shot dialect split, but the printed WERs are internally inconsistent and need fixing before the SOTA claim can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the ConMamba encoder, which replaces self-attention with bidirectional Mamba state-space layers, selective state-space models that process a sequence in linear time, and adds depthwise-separable convolutions to capture local acoustic structure such as phoneme boundaries. A unidirectional Mamba decoder combines the encoder output with autoregressive token predictions. Around this core, the paper's recipe mixes roughly 16 hours of wTIMIT whispered speech, Singaporean dialect, and normal and whispered CHAINS data with a thousand hours of LibriSpeech to train the Mamba model from scratch, and fine-tunes Wav2Vec2, WavLM, HuBERT, and Whisper on the same small multi-dialect corpus. The efficiency claim rests on the Mamba model's linear scaling, which lets a small model train on a low-range dataset and still model long-range dependencies.
What would settle it
Check every utterance in the wTIMIT training portion, the LibriSpeech subset, and the self-supervised pretraining corpora for US-dialect speakers or US transcripts; if any US audio or its text appears in training, a retrained system on the clean split would show sharply higher US WERs, settling the zero-shot cross-dialect claim.
Extended reading notes
Core claim
The paper's central discovery, stated on its own terms, is that a system trained only on Singaporean whispered and normal speech plus normal Irish speech can transcribe unseen US whispered and normal speech and unseen Irish whispered speech at very low error rates. Fine-tuning Whisper reaches WERs of 0.51% on Singaporean whispered, 0.12% on Singaporean normal, 0.40% on US whispered, 0.92% on US normal, and 2.11% on Irish whispered speech. The from-scratch Mamba-Ver2 model reaches 0.56% on Singaporean whispered, 0.63% on Singaporean normal, 1.75% on US whispered, 0.97% on US normal, and 1.19% on Irish whispered speech, beating Whisper on the Irish whispered condition. On the strength of these numbers the paper claims the best reported performance on wTIMIT and CHAINS for whispered speech recognition, with the efficient Mamba model trained on roughly 16 hours of whispered data mixed with LibriSpeech rather than on hundreds of thousands of hours.
Load-bearing premise
Everything depends on the dataset split being exactly as described: only Singaporean speech appears in training, US speech appears only in testing, Irish whispered speech appears only in testing, and no US utterances hide inside the LibriSpeech mix or the pretraining corpora.
Editorial extensions
If this is right
- If the results hold, a from-scratch state-space model trained on about 16 hours of whispered speech plus normal audiobook speech can come within a few tenths of a percent of Whisper, a model pretrained on 680,000 hours.
- Fine-tuned Whisper's near-zero WERs on unseen US dialect suggest that large self-supervised models need only a small amount of whispered target-domain data to adapt across dialects and speaking styles.
- Mamba-Ver2 beating Whisper on Irish whispered speech, 1.19% versus 2.11%, suggests that the state-space model is particularly robust to the unusually high speech rate of the CHAINS corpus, not just to whispering.
- The training strategy removes the need for whisper-to-normal speech conversion or pseudo-whispered data augmentation, replacing them with a simple mix of small whispered and large normal corpora.
Reading between the lines
- The paper leaves implicit that the near-perfect US results imply the model has learned dialect-invariant acoustic representations rather than dialect-specific shortcuts; probing hidden states for dialect information would test this directly.
- A direct extension would be to train the same ConMamba architecture on whispered data from another language family, since the model is trained from scratch without English-specific pretraining, success there would indicate the mechanism generalizes beyond English.
- A controlled next experiment would train a standard transformer or Conformer from scratch on the identical data mix, isolating the efficiency gain attributable to the Mamba layers themselves.
- Reproducing the exact wTIMIT and LibriSpeech split with another whispered corpus would clarify how much of the cross-dialect transfer comes from the small whispered set versus the large normal-speech mix.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a Mamba-based state-space model (ConMamba encoder with Mamba decoder, versions Ver1 and Ver2) and fine-tuned self-supervised models (Wav2Vec2, WavLM, HuBERT, Whisper) for whispered speech recognition across Singaporean, US, and Irish dialects. Training uses wTIMIT Singaporean whispered and normal speech, CHAINS Irish normal speech, and (for the Mamba models) 1000 hours of LibriSpeech; evaluation covers SG whispered/normal, US whispered/normal, and Irish whispered speech. The paper reports very low WERs, e.g., 0.12% for Whisper on SG normal, 0.92% on US normal, and 2.11% on Irish whispered speech, and claims state-of-the-art performance on the wTIMIT and CHAINS datasets. The central technical claim is that a comparatively small Mamba model trained from scratch approaches or beats fine-tuned Whisper on several conditions while using far less whispered data.
Significance. If the reported results hold, they would represent a striking advance in whispered and multi-dialect ASR: current published WERs on whispered CHAINS are around 9%, whereas this paper reports 1.19% for Mamba-Ver2 and 2.11% for Whisper on the Irish whispered condition. The paper also makes a useful methodological contribution by comparing a from-scratch state-space model against fine-tuned self-supervised baselines under a controlled evaluation protocol, and it releases code. The main caveats are that the numerical results contain internal label inconsistencies that must be reconciled before the claims can be evaluated, the claim that the systems were not exposed to US data is contradicted by the use of LibriSpeech in Mamba training, and the state-of-the-art claim is supported by only a single self-cited comparison. These are fixable issues, but they currently prevent acceptance of the headline claims.
major comments (4)
- [Section 4, Tables 1 and 2] The US whispered/normal column assignments are interchanged between the prose and the tables, and this affects the headline numbers. In Table 2, the Whisper row lists Whisper-US=0.40 and Normal-US=0.92, but the prose states that for the US dialect the WER was 0.92% for whispered speech and 0.40% for normal speech. The same swap appears in Table 1: the prose says the US whispered WER is 5.97%, while the table shows Whisper-US=3.4 and Normal-US=5.97, and the statement that the SG normal WER is nearly twice the US value only holds if Normal-US=3.4. Since the abstract's state-of-the-art claim is quantified by these exact WERs, the tables and the prose must be reconciled and the resulting values re-derived before the results can be assessed.
- [Section 3.1 and Section 4] The claim that 'the systems were not provided with data for the US dialect, neither normal nor whispered speech' is contradicted by the training setup described for the Mamba models. Section 3.1 states that the Mamba models are trained from scratch on a mixture that includes 1000 hours of LibriSpeech, and LibriSpeech consists predominantly of US English speakers. Consequently, US normal speech is present in the Mamba training mixture, which invalidates the zero-shot interpretation of the US normal results unless LibriSpeech is explicitly excluded or the claim is restricted to the fine-tuning data of the self-supervised models. This needs to be clarified and, if necessary, the evaluation must be rerun without US English material in the Mamba training set.
- [Abstract and Section 4] The claim of 'best performance reported on the wTIMIT and CHAINS datasets' is supported only by a single prior WER of 9.22% from the authors' own reference [4]. No comparison table of published wTIMIT/CHAINS results is provided, and no confidence intervals or statistical significance tests accompany the reported WERs. Since the central claim is a state-of-the-art statement, the manuscript should include a systematic comparison with all relevant prior published results on these datasets and report the variance across evaluation subsets or multiple runs.
- [Section 3.2 and Section 4] The efficiency claim in the title and abstract is not quantified. The manuscript states that Mamba-Ver2 is 'significantly smaller' and trains with 'tiny data' compared to Whisper, but it does not report parameter counts, training time, FLOPs, or the actual hours of data used by each model. This is particularly important because the Mamba models are trained on 1000 hours of LibriSpeech in addition to the whispered data, so 'low data' refers only to the whispered portion. Please provide concrete efficiency metrics so that the efficiency contribution can be evaluated.
minor comments (5)
- [Throughout] There are several typos and grammatical issues, for example 'autoencode' should be 'autoencoder', and 'Whereby we proposed an efficient model' should be rephrased as 'We therefore propose an efficient model'.
- [Section 1] The statement that 'the Whisper model developed by OpenAI is italicized to differentiate it easily from the concept of whispered speech' is not visibly implemented in the manuscript; the model name appears in the same font as surrounding text. Either apply the italics or remove the note.
- [Table 2 caption] The caption says results are based on greedy and beam searches, but the Whisper and Mamba rows are evaluated only with beam search. Please state this explicitly in the caption or add the missing greedy values.
- [Figure 3] Figure 3 shows training and evaluation losses for Mamba-Ver1 and Ver2, but the text does not discuss the loss curves in detail. Please relate the curves to the reported WERs, especially the plateau behavior and whether early stopping was used.
- [References] Reference [4] is a self-citation of prior work by the same authors; it should be clearly marked as such in the text, and the comparison against it should be presented in a dedicated table rather than only in prose.
Circularity Check
No circularity: WERs are measured on held-out data; self-citation to a prior baseline is not load-bearing.
full rationale
No circularity found. The paper's claims are empirical: WERs are measured on held-out test conditions (Section 4, Tables 1-2) after training on the described partitions, with no fitting of parameters to target error rates. The Mamba model is trained from scratch on a mixture of whispered/normal speech and LibriSpeech, and the self-supervised models are fine-tuned with standard loss; none of the reported WERs is defined as an input or derived from the claim of state-of-the-art performance. The only author self-citation, [4], supplies a previous WavLM CHAINS result (9.22%) used as a comparison baseline; the current WERs do not reduce to that equation or to any self-citation. Internal numerical inconsistencies between Section 4 prose and Table 2 are a correctness/validity concern, not circularity.
Assumptions & free parameters
free parameters (7)
- ConMamba encoder depth =
12 layers
- Mamba decoder depth =
Ver1: 4 layers, Ver2: 6 layers
- Cross-attention heads =
Ver1: 4, Ver2: 8
- Training epochs =
50 epochs for Mamba models, 5 for self-supervised fine-tuning
- Batch size =
8
- SpecAugment policy =
time-masking only
- LibriSpeech/whispered data mixture ratio =
not specified
assumptions (3)
- domain assumption The wTIMIT and CHAINS train/test splits are speaker- and utterance-disjoint as described, and US audio never appears in training or pretraining.
- domain assumption Fine-tuning large self-supervised models for 5 epochs on a few hours of whispered SG data generalizes to unseen US and Irish whispered speech.
- domain assumption Mixing 1,000 hours of normal LibriSpeech speech with small whispered data helps the Mamba model learn both styles instead of biasing it toward normal speech.
Cite this review
Pith. "Pith review of State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition." pith.science (2026). https://pith.science/paper/5UEXK3FB
@misc{pith2026250616969,
author = {Pith},
title = {Pith review of: State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/5UEXK3FB}},
note = {Machine review of arXiv:2506.16969}
}
read the original abstract
Whispered speech recognition presents significant challenges for conventional automatic speech recognition systems, particularly when combined with dialect variation. However, utilizing an efficient method to solve this problem using a low-range dataset and processing load is beneficial. This paper proposes a solution using a Mamba-based state-space model and four fine-tuned self-supervised models consisting of Wav2Vec2, WavLM, HuBERT, and Whisper to address the dual challenges of whispered speech and dialect diversity. Based on our knowledge, this represents the best performance reported on the wTIMIT and CHAINS datasets for whispered speech recognition. We trained the models using whispered and normal speech data across Singaporean, US, and Irish dialects. The findings demonstrated that utilizing the proposed Mamba-based model could work as a highly efficient model trained with low amounts of whispered data to simultaneously work on whispered and normal speech recognition. The code for this work is freely available.
Figures
Reference graph
Works this paper leans on
-
[4]
Experimental Results In this work, the proposed state-space and self-supervised mod- els for whispered speech recognition have been trained to per- form well with both whispered and normal speech. As a base- line, we evaluated the performance of the pre-trained Whisper Large-v2 model on the test set to assess the need for a special- ized system for the pr...
-
[1]
State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition
Introduction Unconventional or challenging speech patterns can alter the acoustic structure of speech, posing difficulties for standard au- tomatic speech recognition systems [1, 2]. Whispered speech is a form of speech production often used voluntarily to main- tain privacy or avoid disturbing others in environments such as libraries. It is sometimes pro...
work page Pith review arXiv 2025
-
[2]
The primary distinction lies in how they are produced
whispered and Multi-dialect speech Whispered speech exhibits significant differences from normal speech in terms of its production mechanisms and acoustic fea- tures. The primary distinction lies in how they are produced. In the whispered speech, the air is exhaled through an adjusted pharynx, ensuring the vocal cords do not vibrate [17]. This leads to a ...
-
[3]
Specifically, we introduce the wTIMIT and CHAINS
methodology This section describes the materials and the proposed sys- tem. Specifically, we introduce the wTIMIT and CHAINS
-
[5]
Conclusion Whispered speech recognition presents a significant challenge for conventional ASR systems, particularly when combined with additional factors such as limited data and dialect varia- tion. These challenges can severely degrade the performance of traditional systems, highlighting the necessity of developing ASR models specifically tailored to ha...
-
[6]
Gener- ative models for improved naturalness, intelligibility, and voicing of whispered speech,
D. Wagner, S. P. Bayerl, H. A. C. Maruri, and T. Bocklet, “Gener- ative models for improved naturalness, intelligibility, and voicing of whispered speech,” in 2022 IEEE Spoken Language Technol- ogy Workshop (SLT). IEEE, 2023, pp. 943–948
work page 2022
-
[7]
A. Farhadipour and H. Veisi, “Gammatonegram representation for end-to-end dysarthric speech processing tasks: Speech recogni- tion, speaker identification, and intelligibility assessment,” Iran Journal of Computer Science, pp. 1–14, 2024
work page 2024
-
[8]
A. Farhadipour, H. Veisi, M. Asgari, and M. A. Keyvan- rad, “Dysarthric speaker identification with different degrees of dysarthria severity using deep belief networks,” Etri Journal , vol. 40, no. 5, pp. 643–652, 2018
work page 2018
Show all 31 references
-
[9]
Analysis of deep generative model impact on feature extraction and dimension reduction for short ut- terance text-independent speaker verification,
A. Farhadipour and H. Veisi, “Analysis of deep generative model impact on feature extraction and dimension reduction for short ut- terance text-independent speaker verification,” Circuits, Systems, and Signal Processing, vol. 43, no. 7, pp. 4547–4564, 2024
2024
-
[10]
Leveraging self- supervised models for automatic whispered speech recognition,
A. Farhadipour, H. Asadi, and V . Dellwo, “Leveraging self- supervised models for automatic whispered speech recognition,” arXiv preprint arXiv:2407.21211, 2024
2024 arXiv
-
[11]
Whisper to normal speech conversion using sequence-to-sequence mapping model with auditory attention,
H. Lian, Y . Hu, W. Yu, J. Zhou, and W. Zheng, “Whisper to normal speech conversion using sequence-to-sequence mapping model with auditory attention,”IEEE Access, vol. 7, pp. 130 495– 130 504, 2019
2019
-
[12]
wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020
2020
-
[13]
End-to- end whispered speech recognition with frequency-weighted ap- proaches and pseudo whisper pre-training,
H.-J. Chang, A. H. Liu, H.-y. Lee, and L.-s. Lee, “End-to- end whispered speech recognition with frequency-weighted ap- proaches and pseudo whisper pre-training,” in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 186– 193
2021
-
[14]
Improving whispered speech recognition performance using pseudo-whispered based data augmentation,
Z. Lin, T. Patel, and O. Scharenborg, “Improving whispered speech recognition performance using pseudo-whispered based data augmentation,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8
2023
-
[15]
Whispered speech recogni- tion using deep denoising autoencoder and inverse filtering,
D. T. Grozdic and S. T. Jovicic, “Whispered speech recogni- tion using deep denoising autoencoder and inverse filtering,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 25, no. 12, pp. 2313–2322, 2017
2017
-
[16]
datasets and provide a brief overview of the mamba-based model engaged with convolutions and four self-supervised models consisting of WavLM, HuBERT, Wav2Vec2, andWhis- per. 3.1. Datasets Whispered speech datasets for the English language are lim- ited, and the wTIMIT dataset ...
-
[17]
Multi-dialect speech recognition with a single sequence-to-sequence model,
B. Li, T. N. Sainath, K. C. Sim, M. Bacchiani, E. Weinstein, P. Nguyen, Z. Chen, Y . Wu, and K. Rao, “Multi-dialect speech recognition with a single sequence-to-sequence model,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 20...
2018
-
[18]
Multi-dialect speech recognition in english using attention on ensemble of experts,
A. Das, K. Kumar, and J. Wu, “Multi-dialect speech recognition in english using attention on ensemble of experts,” inICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6244–6248
2021
-
[19]
Wavlm: Large-scale self- supervised pre-training for full stack speech processing,
S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022
2022
-
[20]
Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,
W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021
2021
-
[21]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518
2023
-
[22]
wtimit whispered timit dataset,
B. P. Lim, “wtimit whispered timit dataset,” 2010. [Online]. Available: http://www.isle.illinois.edu/speech web lg/pubs/2010/ lim10thesis.pdf
2010
-
[23]
Acoustic analysis of consonants in whispered speech,
S. T. Jovi ˇci´c and Z. ˇSari´c, “Acoustic analysis of consonants in whispered speech,” Journal of voice, vol. 22, no. 3, pp. 263–274, 2008
2008
-
[24]
La- ryngeal adjustment in whispering: magnetic resonance imaging study,
K. Tsunoda, Y . Ohta, S. Niimi, Y . Soda, and H. Hirose, “La- ryngeal adjustment in whispering: magnetic resonance imaging study,” Annals of Otology, Rhinology & Laryngology , vol. 106, no. 1, pp. 41–43, 1997
1997
-
[25]
Acoustic differences between voiced and whispered speech in gender diverse speakers,
N. Houle and S. V . Levi, “Acoustic differences between voiced and whispered speech in gender diverse speakers,”The Journal of the Acoustical Society of America, vol. 148, no. 6, pp. 4002–4013, 2020
2020
-
[26]
Lib- rispeech: an asr corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210
2015
-
[27]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023
2023 arXiv
-
[28]
Speech slytherin: Examining the performance and efficiency of mamba for speech separation, recognition, and synthesis,
X. Jiang, Y . A. Li, A. N. Florea, C. Han, and N. Mesgarani, “Speech slytherin: Examining the performance and efficiency of mamba for speech separation, recognition, and synthesis,” arXiv preprint arXiv:2407.09732, 2024
2024 arXiv
-
[29]
Specaugment: A simple data augmen- tation method for automatic speech recognition,
D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmen- tation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019
1904 arXiv
-
[30]
Decoupled weight decay regulariza- tion,
I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” arXiv preprint arXiv:1711.05101, 2017
2017 arXiv
-
[31]
Paddlespeech: An easy-to- use all-in-one speech toolkit,
H. Zhang, T. Yuan, J. Chen, X. Li, R. Zheng, Y . Huang, X. Chen, E. Gong, Z. Chen, X. Hu et al. , “Paddlespeech: An easy-to- use all-in-one speech toolkit,” arXiv preprint arXiv:2205.12007, 2022
2022 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.