REVIEW 3 major objections 4 minor 35 references
Adding uncertainty-aware statistical features from Whisper embeddings and a CNN-sLSTM backbone improves non-intrusive speech intelligibility prediction on TMHINT-QI(S).
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
iMTI-Net combines Whisper embeddings with uncertainty-proxy statistics and a CNN-sLSTM backbone to improve non-intrusive speech intelligibility prediction on TMHINT-QI(S).
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A plausible extension of MTI-Net with a real confound: the baseline is not the original MTI-Net, and no ablation isolates entropy or sLSTM. the 3 major comments →
Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that deriving per-frame uncertainty proxies from Whisper embeddings—the mean, standard deviation, and softmax entropy of each embedding frame—and feeding them through a CNN-sLSTM backbone yields better non-intrusive speech intelligibility prediction than the original MTI-Net. The paper's iMTI-Net concatenates these statistics with the raw Whisper embedding, passes the result through an adapter, concatenates it with CNN features from STFT and learnable filter banks, and models the sequence with sLSTM. A multi-task loss jointly predicts human intelligibility, STOI, and inverted character error rates from Google ASR and Whisper. On TMHINT-QI(S), the CNN-sLSTM variant achiev
What carries the argument
The central object is the feature vector x_t = [E_t; μ_t; σ_t; h_t], where E_t is a Whisper embedding frame and h_t = −Σ p_t,d log p_t,d is the entropy of a softmax applied over the embedding dimension—the paper's proxy for uncertainty. The mean μ_t and standard deviation σ_t capture global per-frame characteristics. These statistics are concatenated with the raw embeddings, passed through an adapter, concatenated with CNN-based acoustic features, and temporally modeled by sLSTM, a scalar-gated LSTM with an extra normalization state n_t that computes a normalized hidden state h_t = c_t/n_t. This mechanism is what lets the model combine the rich pre-trained representation with a compact per-f
Load-bearing premise
The load-bearing premise is that the entropy and statistical features—rather than the other simultaneous changes (Whisper instead of HuBERT, the added Whisper-CER target, feature-dimension instead of temporal concatenation, and sLSTM)—drive the measured gains; the paper gives no ablation that isolates them.
What would settle it
Run the same training pipeline with the entropy and moment features removed (or replaced by a constant) while keeping CNN-sLSTM and the added Whisper-CER target; if the intelligibility LCC/SRCC stays near 0.78 on TMHINT-QI(S), the uncertainty-aware statistics are not the cause. A complementary test would keep Whisper and BLSTM while adding only the entropy features; if they produce no improvement over the baseline, the claimed mechanism is unsupported.
If this is right
- If the improvement generalizes, non-intrusive intelligibility assessment can be built on pre-trained ASR embeddings plus cheap statistical summaries, without needing reference speech.
- Using sLSTM instead of BLSTM yields better correlation on most targets, suggesting that normalized scalar memory helps long-range speech modeling in this task.
- Multi-task prediction of human intelligibility and ASR error rates lets a single model serve both subjective assessment and automatic recognition monitoring.
- The evaluation set includes unseen noise types and unseen speech enhancement systems, so the results speak to robustness under domain mismatch, not just memorized conditions.
- Adding Whisper-CER as an extra training target may allow machine recognition signals to serve as auxiliary supervision when human labels are limited.
Where Pith is reading between the lines
- The largest untested contributor may be the switch from HuBERT to Whisper in the baseline; a controlled ablation keeping Whisper and BLSTM while adding only the entropy/moment features would show whether the uncertainty statistics matter at all.
- The entropy is computed after softmax over Whisper's feature dimension, so it measures how flat a frame's representation is; high entropy may correspond to uncertain or poorly encoded frames, but the paper does not test that interpretation directly.
- The scatter plots suggest the practical benefit is a wider spread of predictions at both extremes; this could be tested on enhancement conditions known to produce very low or very high intelligibility.
- Because the statistics are inexpensive, a natural extension is to apply the same uncertainty-aware scheme to other pre-trained encoders, such as HuBERT or wav2vec, to see whether the benefit transfers across embeddings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes iMTI-Net, an extension of MTI-Net for non-intrusive speech intelligibility prediction. The model extracts Whisper Transformer embeddings, computes per-frame mean, standard deviation, and softmax entropy across embedding dimensions, concatenates these with CNN-based acoustic features, and models temporal structure with an sLSTM. A multi-task loss jointly predicts subjective intelligibility, Google and Whisper CER, and STOI. On the TMHINT-QI(S) benchmark, the best CNN-sLSTM variant achieves LCC 0.7817 and SRCC 0.7622 for intelligibility, outperforming the reported baseline. The authors attribute the gains to uncertainty-aware features and the CNN-sLSTM architecture.
Significance. If the proposed components are indeed responsible for the reported gains, the paper would make a useful contribution to non-intrusive intelligibility assessment: it combines a strong pretrained encoder with a lightweight uncertainty proxy and a modern recurrent cell, and it evaluates on a benchmark that includes unseen noises and enhancement systems. The multi-target setup covering both human and machine intelligibility is also practical. However, the current experimental design does not isolate the proposed mechanisms, so the central claim is not yet supported by the evidence presented.
major comments (3)
- [§III-B, Table I] The baseline used in Table I is not the original MTI-Net. As stated in §III-B, the baseline replaces HuBERT with Whisper, adds Whisper CER as a training target, and concatenates features along the temporal dimension, whereas iMTI-Net concatenates along the feature dimension. The comparison therefore bundles at least four changes: embedding model, extra target, concatenation strategy, and the proposed statistical features/sLSTM. This cannot support the abstract's claim that iMTI-Net 'outperforms the original MTI-Net' or that gains come specifically from uncertainty-aware features. Please report the true original MTI-Net result and/or a factorial ablation.
- [§III-B, Tables I–IV] The contribution of entropy is never isolated. No experiment removes entropy while keeping mean/std, and no experiment replaces Whisper with HuBERT within the iMTI-Net pipeline. The iMTI-Net CNN-BLSTM vs CNN-sLSTM pair isolates the recurrent cell only within iMTI, not the uncertainty mechanism. Without such ablations, the central attribution to 'uncertainty-aware features' is unsupported. Please add at least: baseline + mean/std, baseline + mean/std + entropy, and corresponding iMTI variants with and without entropy, with and without sLSTM.
- [§III-A/III-B, Tables I–IV] All reported LCC/SRCC/MSE values are single runs with no error bars, confidence intervals, or significance tests. The conclusion states 'consistent and significant improvements,' but, for example, intelligibility LCC 0.7791 vs 0.7817 between the two iMTI variants is not shown to be statistically distinguishable. Please provide multi-seed means and standard deviations and appropriate significance tests (e.g., paired bootstrap or t-test) for the key comparisons.
minor comments (4)
- [Eq. (2)] The forget-gate expression f_t = ( exp(˜f_t) / σ(˜f_t) is missing its case condition. Please complete or correct the piecewise definition.
- [Eqs. (1) and (2)] The symbol h_t is used for entropy in Eq. (1) and for hidden state in Eq. (2). Rename one to avoid confusion.
- [§III-A] The dataset description mentions a quality score, but the experiments do not use it. Please clarify whether quality prediction is included or omitted.
- [§II] The choice of Whisper layer for the embeddings is not specified. Please state which layer/block is used and whether it was selected on the development set.
Circularity Check
No circular derivation found: iMTI-Net is trained and evaluated on external intelligibility/CER/STOI targets, and the reported gains are empirical rather than forced by construction.
full rationale
The paper's derivation chain is empirical. Eq. (1) defines Whisper embeddings E, per-frame mean, standard deviation, and entropy, then concatenates them into a feature vector x_t; Eq. (2) defines the sLSTM update; Eq. (3) defines the multitask loss. None of these equations uses the evaluation targets as inputs. The targets — subjective intelligibility, Google CER, Whisper CER, and STOI — are external measurements on speech utterances, and the model is trained to predict them and evaluated with LCC/SRCC/MSE on held-out data. Thus the reported correlation values are not equivalent to the model's inputs by construction. The paper does use several self-citations, notably to MTI-Net [15] and the Whisper-based study [20], but these are used as a prior architecture, dataset, and baseline source, not as proof of the proposed model's improvement. There is a real experimental attribution problem: the 'Baseline' in Tables I–IV is a modified MTI-Net (Whisper replaces HuBERT, and Whisper CER is added as a target), and the iMTI-Net variants change multiple components at once — entropy statistics, feature-concatenation scheme, and BLSTM versus sLSTM — with no ablation isolating the entropy term. That makes the causal claim about uncertainty-aware features under-supported, and it also means the numbers do not strictly compare against the original MTI-Net. However, this is a confound/ablation gap, not circularity: the predictions still come from held-out external targets, and no equation or fitted parameter is renamed as a prediction. Similarly, using Whisper for both embeddings and one CER target may make that branch easier, but it is a leakage-like concern rather than a logical loop. I therefore find no significant circularity and score 0.
Axiom & Free-Parameter Ledger
free parameters (3)
- Loss weights gamma1..gamma4 =
1, 1, 1, 5
- Architecture hyperparameters (CNN channels, LSTM units, FC neurons) =
16/32/64/128 channels, 128 units, 128 neurons
- Whisper model variant and layer selection =
Not specified
axioms (4)
- domain assumption Entropy of softmax over Whisper embedding dimensions is a proxy for uncertainty and improves intelligibility prediction.
- domain assumption Whisper embeddings generalize to unseen noise types and enhancement systems.
- domain assumption TMHINT-QI(S) subjective labels are reliable and sufficient ground truth.
- domain assumption Multi-task learning with CER and STOI targets helps predict subjective intelligibility.
Cite this review
Pith. "Pith review of Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM." pith.science (2026). https://pith.science/paper/M5FBHO2N
@misc{pith2026250903013,
author = {Pith},
title = {Pith review of: Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM},
year = {2026},
howpublished = {\url{https://pith.science/paper/M5FBHO2N}},
note = {Machine review of arXiv:2509.03013}
}
read the original abstract
Non-intrusive speech intelligibility prediction remains challenging due to variability in speakers, noise conditions, and subjective perception. We propose an uncertainty-aware approach that leverages Whisper embeddings in combination with statistical features, specifically the mean, standard deviation, and entropy computed across the embedding dimensions. The entropy, computed via a softmax over the feature dimension, serves as a proxy for uncertainty, complementing global information captured by the mean and standard deviation. To model the sequential structure of speech, we adopt a scalar long short-term memory (sLSTM) network, which efficiently captures long-range dependencies. Building on this foundation, we propose iMTI-Net, an improved multi-target intelligibility prediction network that integrates convolutional neural network (CNN) and sLSTM components within a multitask learning framework. It jointly predicts human intelligibility scores and machine-based word error rates (WER) from Google ASR and Whisper. Experimental results show that iMTI-Net outperforms the original MTI-Net across multiple evaluation metrics, demonstrating the effectiveness of incorporating uncertainty-aware features and the CNN-sLSTM architecture.
Figures
Reference graph
Works this paper leans on
-
[1]
P. C. Loizou, Speech enhancement: Theory and practice . CRC press, 2007
work page 2007
-
[2]
A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Per- ceptual evaluation of speech quality (PESQ), an objec- tive method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” in ITU-T Recommendation , 2001, p. 862
work page 2001
-
[3]
The hearing-aid speech perception index (HASPI) version 2,
J. M. Kates and K. H. Arehart, “The hearing-aid speech perception index (HASPI) version 2,” Speech Commu- nication, vol. 131, pp. 35–46, 2021
work page 2021
-
[4]
The hearing-aid speech quality index (HASQI) version 2,
J. M. Kates and K. H. Arehart, “The hearing-aid speech quality index (HASQI) version 2,” Journal of the Audio Engineering Society , vol. 62, no. 3, pp. 99–117, 2014
work page 2014
-
[5]
A physical method for measuring speech-transmission quality,
H. J. M. Steeneken and T. Houtgast, “A physical method for measuring speech-transmission quality,” Journal of the Acoustical Society of America , vol. 67, no. 1, pp. 318–326, 1980
work page 1980
-
[6]
Factors governing the intelligibility of speech sounds,
N. R. French and J. C. Steinberg, “Factors governing the intelligibility of speech sounds,” Journal of the Acous- tical Society of America , vol. 19, no. 1, pp. 90–119, 1947
work page 1947
-
[7]
Methods for calculation of the speech intelligibility index,
ANSI Std. S3.5 1997, “Methods for calculation of the speech intelligibility index,” in Acoustical Society of America, 1997. 4
work page 1997
-
[8]
Evaluation of speech transmission channels by using artificial signals,
T. Houtgast and H. 1. M. Steeneken, “Evaluation of speech transmission channels by using artificial signals,” Acustica, vol. 25, no. 6, pp. 355–367, 1971
work page 1971
-
[9]
An algorithm for intelligibility prediction of time- frequency weighted noisy speech,
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time- frequency weighted noisy speech,” IEEE/ACM Trans- actions on Audio, Speech and Language Processing , vol. 19, no. 7, pp. 2125–2136, 2011
work page 2011
-
[10]
A neural network for monaural intrusive speech intelligibility prediction,
M. B. Pedersen, A. H. Andersen, S. H. Jensen, and J. Jensen, “A neural network for monaural intrusive speech intelligibility prediction,” in Proc. ICASSP , 2020, pp. 336–340
work page 2020
-
[11]
Nonintrusive speech intelligibility predic- tion using convolutional neural networks,
A. H. Andersen, J. M. D. Haan, Z.-H. Tan, and J. Jensen, “Nonintrusive speech intelligibility predic- tion using convolutional neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, vol. 26, no. 10, pp. 1925–1939, 2018
work page 1925
-
[12]
STOI-Net: A deep learning-based non- intrusive speech intelligibility assessment model,
R. E. Zezario, S.-W. Fu, C.-S. Fuh, Y . Tsao, and H.-M. Wang, “STOI-Net: A deep learning-based non- intrusive speech intelligibility assessment model,” in Proc. APSIPA ASC , 2020, pp. 482–486
work page 2020
-
[13]
Multi-objective non-intrusive hearing- aid speech assessment model,
H.-T. Chiang, S.-W. Fu, H.-M. Wang, Y . Tsao, and J. H. L. Hansen, “Multi-objective non-intrusive hearing- aid speech assessment model,” J. Acoust. Soc. Am. , vol. 195, pp. 3574–3587, 2024
work page 2024
-
[14]
Z. Tu, N. Ma, and J. Barker, “Exploiting hidden rep- resentations from a DNN-based speech recogniser for speech intelligibility prediction in hearing-impaired lis- teners,” in Proc. INTERSPEECH, 2022, pp. 3488–3492
work page 2022
-
[15]
MTI-Net: A multi-target speech in- telligibility prediction model,
R. E. Zezario, S.-W. Fu, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “MTI-Net: A multi-target speech in- telligibility prediction model,” in Proc. INTERSPEECH, 2022, pp. 5463–5467
work page 2022
-
[16]
R. E. Zezario, S.-W. Fu, F. Chen, C.-S. Fuh, H.-M. Wan, and Y . Tsao, “Deep learning-based non-intrusive multi- objective speech assessment model with cross-domain features,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 54–70, 2023
work page 2023
-
[17]
MBI-Net: A non-intrusive multi-branched speech intelligibility prediction model for hearing aids,
R. E. Zezario, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “MBI-Net: A non-intrusive multi-branched speech intelligibility prediction model for hearing aids,” in Proc. INTERSPEECH, 2022, pp. 3944–3948
work page 2022
-
[18]
Speech foundation models on intelligibility prediction for hearing-impaired listen- ers,
S. Cuervo and R. Marxer, “Speech foundation models on intelligibility prediction for hearing-impaired listen- ers,” in Proc. ICASSP, 2024, pp. 1421–1425
work page 2024
-
[19]
R. Mogridge, G. Close, R. Sutherland, et al. , “Non- intrusive speech intelligibility prediction for hearing- impaired users using intermediate ASR features and hu- man memory models,” in Proc. ICASSP, 2024, pp. 306– 310
work page 2024
-
[20]
A study on incorporating Whis- per for robust speech assessment,
R. E. Zezario, Y .-W. Chen, S.-W. Fu, Y . Tsao, H.-M. Wang, and C.-S. Fuh, “A study on incorporating Whis- per for robust speech assessment,” in Proc. ICME, 2024, pp. 1–6
work page 2024
-
[21]
Non-intrusive speech intelligibility prediction using an auditory periphery model with hearing loss,
C. O. Mawalim, B. A. Titalim, S. Okada, and M. Unoki, “Non-intrusive speech intelligibility prediction using an auditory periphery model with hearing loss,” Applied Acoustics, vol. 214, p. 109 663, 2023
work page 2023
-
[22]
Wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. NIPS, 2020, pp. 1– 12
work page 2020
-
[23]
HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,
W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , pp. 3451– 3460, 2021
work page 2021
-
[24]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023, pp. 28 492–28 518
work page 2023
-
[25]
XLSTM: Extended long short-term memory,
M. Beck, K. P ¨oppel, M. Spanring, et al. , “XLSTM: Extended long short-term memory,” in Proc. NeurIPS , 2024, pp. 1–57
work page 2024
-
[26]
Speech recognition (version 3.6) [software],
A. Zhang, “Speech recognition (version 3.6) [software],” in Proc. ICCC, 2017
work page 2017
-
[27]
Speaker recognition from raw waveform with Sincnet,
M. Ravanelli and Y . Bengio, “Speaker recognition from raw waveform with Sincnet,” in Proc. SLT , 2018, pp. 1021–1028
work page 2018
-
[28]
InQSS: A speech intelli- gibility and quality assessment model using a multi- task learning network,
Y .-W. Chen and Y . Tsao, “InQSS: A speech intelli- gibility and quality assessment model using a multi- task learning network,” in Proc. INTERSPEECH, 2022, pp. 3088–3092
work page 2022
-
[29]
The V oiceMOS challenge 2023: Zero-shot subjective speech quality prediction for multiple domains,
E. Cooper, W.-C. Huang, Y . Tsao, H.-M. Wang, T. Toda, and J. Yamagishi, “The V oiceMOS challenge 2023: Zero-shot subjective speech quality prediction for multiple domains,” in Proc. ASRU, 2023, pp. 1–7
work page 2023
-
[30]
Speech enhancement using a minimum mean-square error log-spectral amplitude estimator,
Y . Ephraim and D. Malah, “Speech enhancement using a minimum mean-square error log-spectral amplitude estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing , vol. 33, no. 2, pp. 443–445, 1985
work page 1985
-
[31]
Raw waveform-based speech enhancement by fully convolu- tional networks,
S.-W. Fu, Y . Tsao, X. Lu, and H. Kawai, “Raw waveform-based speech enhancement by fully convolu- tional networks,” in Proc. APSIPA ASC, 2017, pp. 6–12
work page 2017
-
[32]
T-GSA: Trans- former with gaussian-weighted self-attention for speech enhancement,
J. Kim, M. El-Khamy, and J. Lee, “T-GSA: Trans- former with gaussian-weighted self-attention for speech enhancement,” in Proc. ICASSP, 2020, pp. 6649–6653
work page 2020
-
[33]
CMGAN: Conformer-based MetricGAN for speech enhancement,
R. Cao, S. Abdulatif, and B. Yang, “CMGAN: Conformer-based MetricGAN for speech enhancement,” in Proc. INTERSPEECH, 2022, pp. 936–940
work page 2022
-
[34]
Music source separation in the waveform domain,
A. D ´efossez, N. Usunier, L. Bottou, and F. Bach, “Music source separation in the waveform domain,” arXiv 1911.13254 , 2021
Pith/arXiv arXiv 1911
-
[35]
The proof and measurement of associ- ation between two things,
C. Spearman, “The proof and measurement of associ- ation between two things,” The American Journal of Psychology, vol. 15, no. 1, pp. 72–101, 1904. 5
work page 1904
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.