REVIEW 3 major objections 5 minor 33 references
No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A listener's own past intelligibility scores, taken from a handful of known audio clips, predict their scores on new audio better than their audiogram does, and the paper builds a network, SSIPNet, that does exactly this.
desk verdict The few-shot idea is worth a look, but the headline result rests on a synthetic score recalibration that silently feeds audiogram info back into the target. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is SSIPNet's two-stage design plus a dataset-decorrelation step. The Feature Extraction Module freezes the Whisper v3 encoder, temporally pools and transforms all 32 layer outputs, and injects the known intelligibility score as a linear-projected feature along the layer dimension, so the embedding is audio-and-score specific. The Support-Based Prediction Module treats each listener's support (audio, score) pairs as an order-independent set, applies global average pooling to their embeddings, concatenates the pooled listener embedding with the query embedding, and maps the result through a linear layer to a score. Before any of this, the audio-audiogram decorrelation step normalizes every audio to 65 dB SPL and recalibrates the ground-truth score using the level-intelligibility curves for conductive hearing loss from reference [22], S1 = S0 + CHL(L1) − CHL(L0), so that level-based audiogram leakage is removed from the training signal.
What would settle it
Re-run the entire experiment on the original CPC protocol without the calibration step, training SSIPNet on raw intelligibility scores and unnormalized audio, and compare against the audiogram baseline on the unchanged held-out test set; if the support-pair advantage shrinks or reverses, the core claim depends on the conductive-loss assumption. A second, independent probe: train a small classifier to predict a listener's audiogram or enhancement-system identity from the 65 dB SPL normalized audio features; if audiogram information is still recoverable, the decorrelation step has not removed the information the paper claims to exclude.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the listener's audiogram is not the most informative input for intelligibility prediction; their own previously measured intelligibility scores are. The authors introduce SSIPNet, where a frozen Whisper v3 encoder produces per-sample embeddings that are conditioned on the known score, and a permutation-invariant aggregator (global average pooling over the support embeddings) builds a listener-level representation that is concatenated with the query embedding and linearly read out to a predicted score. Trained in listener-structured batches on the CPC dataset, with audio normalized to 65 dB SPL and ground-truth scores calibrated through the conductive-loss level-intelligibility curves of reference [22] to strip audiogram-related level information, the model reports average RMSE 23.43 versus 26.16 for the audiogram-based baseline and NCC 0.811 versus 0.794. The paper further reports that performance improves once at least four support pairs are available, saturates near sixteen, and with a single pair is already comparable to audiogram-conditioned prediction.
Load-bearing premise
The load-bearing premise is that every listener in the CPC dataset can be treated as having conductive hearing loss, so the level-intelligibility curves of reference [22] validly rescale each audio to 65 dB SPL and transform the ground-truth intelligibility scores that the model is trained and evaluated on; if the real losses are sensorineural or mixed, the calibrated scores are not true intelligibility and the reported comparison does not test real prediction.
Editorial extensions
If this is right
- Hearing-aid and speech-enhancement evaluation could be personalized from a listener's existing listening history instead of requiring a clinical audiogram.
- Intelligibility prediction becomes a few-shot regression problem: the same framework naturally consumes any number of support pairs, and the paper shows useful gains already at four to sixteen pairs.
- A single (audio, score) support pair per listener already matches audiogram-conditioned prediction, implying minimal data collection could initialize a personalized model.
- Speech foundation model embeddings carry enough phonetic, semantic, and perceptual information to serve as the substrate for listener-specific conditioning.
- The conductive-loss calibration procedure, if valid, provides a reusable recipe for removing enhancement-system level effects from listener-adaptive datasets.
Reading between the lines
- The clear next test the paper does not run is evaluating SSIPNet on raw, uncalibrated intelligibility scores from listeners with sensorineural loss, where the conductive-loss rescaling of reference [22] does not apply; if the support-pair advantage survives there, the paradigm generalizes beyond the CPC setup.
- The saturation near sixteen support pairs suggests the bottleneck is the capacity of the average-pooled listener embedding, not the data; attention-based support aggregation or a larger listener embedding could push the plateau and is directly testable.
- Because the CPC enhancement systems are themselves tuned to each listener's audiogram, the decorrelation step removes level but not necessarily spectral audiogram traces; an ablation that randomizes enhancement-system identity across support and query would quantify any remaining leakage.
- A practical product reading implicit in the paper is that a hearing aid or companion app could accumulate (audio, intelligibility) pairs from daily use and continuously refine a listener embedding, turning the model into a lifelong personalization component.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SSIPNet, a few-shot model for personalized speech intelligibility prediction that takes a listener's existing (audio, score) pairs as support samples and predicts scores for new audio using a Whisper-based feature extractor and a support-set aggregation module. To prevent the model from exploiting a correlation between listener audiograms and audio RMS levels in the CPC dataset, Section 2.1 normalizes audio to 65 dB SPL and recalibrates the ground-truth intelligibility scores using conductive hearing loss curves from Hood and Poole (1971). Experiments on the CPC dataset with 5 test listeners per fold report lower RMSE and higher NCC than a re-implemented audiogram-based baseline (E011), and Section 3.3 studies the effect of the number of support pairs.
Significance. The idea of personalizing intelligibility prediction from a small number of listener-specific scores is interesting and could reduce reliance on audiograms. The use of speech foundation models and support-set aggregation is a reasonable technical direction, and the paper gives a clear architectural description. However, the experimental validation is compromised by the ground-truth recalibration, which injects audiogram information into the labels, and by the small, statistically unquantified test population. The claimed advantage over audiogram-based prediction is therefore not established as a genuine property of the method.
major comments (3)
- [Section 2.1, Figure 3] The calibration step is load-bearing and unjustified. The recalibrated target is S1 = S0 + CHL(L1) − CHL(L0), where HL is the listener's average hearing loss at 0.5, 1, and 2 kHz. This makes the ground-truth scores used for training and evaluation a function of the audiogram, so the 'No Audiogram' claim is not actually tested. Moreover, the assumption that all CPC listeners have conductive hearing loss is asserted without evidence; the typical hearing-impaired population is predominantly sensorineural, for which level-intelligibility curves differ qualitatively from conductive curves. If this assumption is wrong, Tables 2 and 3 measure performance on a synthetic target rather than the experimentally-acquired intelligibility scores, and the reported superiority of SSIPNet may be an artifact of the transformation.
- [Section 3.2, Tables 2 and 3] The evaluation is based on only 5 test listeners per fold and no error bars or significance tests are reported. The NCC improvement is not consistent across folds: SSIPNet is worse than the baseline in fold 2 (0.787 vs 0.809). With n=3 folds, an average NCC improvement of 0.017 and an RMSE improvement of 2.73 are not shown to be statistically reliable, especially given the very small number of listeners. This weakens the central claim that support pairs outperform audiogram-based prediction.
- [Section 2.1] Normalizing the RMS level of all audio to 65 dB SPL does not remove audiogram information encoded in the spectral shape, dynamic range compression, and other processing applied by enhancement systems that are personalized for each listener. The correlation shown in Figure 2 is only with RMS level, but the audio content itself may still carry audiogram-dependent cues. Therefore the 'decorrelation' is incomplete, and the model may still access audiogram-related information through the audio, confounding the interpretation that improvements come from the support (audio, score) pairs alone.
minor comments (5)
- [Section 3.3, Figure 5] The text states that the best performance is achieved with 16 support samples, yet Tables 2 and 3 report results with 64 support samples. Please clarify which configuration is the main result and justify the choice; otherwise the reader may suspect selective reporting.
- [Section 3.1.3, Equation (2)] The notation in Equation (2) is ambiguous: the symbol y is used for both the ground-truth value and its mean. Use \bar{y} for the mean in the Pearson correlation definition.
- [Section 3.1.2] The baseline E011 is re-implemented rather than taken from the official CPC2 results, but no evidence is provided that the re-implementation faithfully reproduces the original method's performance. Please report a comparison on the original challenge test set or provide a reference implementation check.
- [Section 3.1.1] The authors restructured the CPC data splits because the official test set had too few samples per listener. This is a significant deviation from the challenge protocol; please discuss more thoroughly how this affects comparability with prior published CPC results.
- [References] Reference [2] lists 'J. Jensen, C. H. Taal, J. Jensen, and C. H. Taal' with repeated authors; this appears to be a formatting error and should be corrected.
Circularity Check
No circular derivation found: §2.1's audiogram-based score calibration is a validity caveat, not an input-output identity, so the paper's prediction chain is not circular.
full rationale
The derivation chain is self-contained and externally grounded. The model is trained and evaluated on the CPC dataset with a re-implemented external baseline E011 [7]; no fitted parameter is renamed as a prediction, and no load-bearing self-citation appears (reference [7] is external work whose architecture is adopted transparently). The only step that warrants scrutiny is §2.1, where the raw CPC scores are recalibrated as S1 = S0 + CHL(L1) − CHL(L0), with HL derived from the audiogram. This does inject audiogram-derived information into the label and the support scores, and the assumption that all 27 listeners have conductive hearing loss is unsupported. However, the model never receives HL as input, and the prediction is not algebraically equal to the calibration formula; the recalibration is a label transformation, not an input-output identity. Hence the 'No Audiogram' claim is weakened as a validity matter, but the method's derivation does not reduce to its own inputs by construction. No circular step meets the evidentiary bar of this review.
Assumptions & free parameters
free parameters (2)
- Calibration curves CHL(L) from Hood and Poole [22] =
65 dB SPL target level
- Number of support samples (64) =
64
assumptions (5)
- domain assumption All listeners in the CPC dataset have conductive hearing loss
- domain assumption The level-intelligibility relationship from [22] applies to each listener given only average hearing loss
- domain assumption Whisper large-v3 features, pooled and transformed as in [7], are sufficient for intelligibility prediction
- standard math Global average pooling over support embeddings preserves listener-specific information
- domain assumption Audio spectral content does not encode audiogram information after RMS normalization
Cite this review
Pith. "Pith review of No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction." pith.science (2026). https://pith.science/paper/2FWMSECD
@misc{pith2026250602039,
author = {Pith},
title = {Pith review of: No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/2FWMSECD}},
note = {Machine review of arXiv:2506.02039}
}
read the original abstract
Personalized speech intelligibility prediction is challenging. Previous approaches have mainly relied on audiograms, which are inherently limited in accuracy as they only capture a listener's hearing threshold for pure tones. Rather than incorporating additional listener features, we propose a novel approach that leverages an individual's existing intelligibility data to predict their performance on new audio. We introduce the Support Sample-Based Intelligibility Prediction Network (SSIPNet), a deep learning model that leverages speech foundation models to build a high-dimensional representation of a listener's speech recognition ability from multiple support (audio, score) pairs, enabling accurate predictions for unseen audio. Results on the Clarity Prediction Challenge dataset show that, even with a small number of support (audio, score) pairs, our method outperforms audiogram-based predictions. Our work presents a new paradigm for personalized speech intelligibility prediction.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction
Introduction Metrics are crucial for algorithm optimization. For perceptual speech algorithms, such as speech enhancement and wide dy- namic range compression, speech intelligibility score is one of the most important metrics. However, obtaining accurate speech intelligibility data is non-trivial, as it requires conduct- ing numerous listening tests with ...
work page Pith review arXiv 2025
-
[2]
Method 2.1. Audio-audiogram decorrelation We observed a strong correlation between the listener’s audio- gram and the RMS level of the audios presented to that listener on the CPC dataset, as shown in Figure 2. This correlation is expected, as the audios are generated by different enhancement systems, each aimed at providing optimal compensation for in- d...
-
[3]
Experiments and results 3.1. Experimental setup 3.1.1. Data For our experiments, we used the Clarity Prediction Challenge (CPC) dataset [10, 11], which consists of 13126 distinct au- dio signals with experimentally-acquired intelligibility scores. These audio samples are generated by 18 enhancement systems and subsequently presented to 27 hearing-impaired...
-
[4]
Conclusion This paper introduces SSIP, a method for personalized speech intelligibility prediction that generalizes intelligibility scores from existing audio samples to unseen audio. Our experiments demonstrate that, with a carefully designed model architecture, methods leveraging a relatively small number of support (au- dio, score) pairs can outperform...
-
[5]
A short- time objective intelligibility measure for time-frequency weighted noisy speech,
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” in 2010 IEEE International Conference on Acous- tics, Speech and Signal Processing, 2010, pp. 4214–4217
work page 2010
-
[6]
An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,
J. Jensen, C. H. Taal, J. Jensen, and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 24, no. 11, p. 2009–2022, Nov. 2016
work page 2009
-
[7]
Mti-net: A multi-target speech intelligibility prediction model,
R. E. Zezario, S. wei Fu, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “Mti-net: A multi-target speech intelligibility prediction model,” in Interspeech 2022, 2022, pp. 5463–5467
work page 2022
-
[8]
End-to-end speech intelligibility prediction using time- domain fully convolutional neural networks,
M. B. Pedersen, M. Kolbæk, A. H. Andersen, S. H. Jensen, and J. Jensen, “End-to-end speech intelligibility prediction using time- domain fully convolutional neural networks,” inInterspeech 2020, 2020, pp. 1151–1155
work page 2020
Show all 33 references
-
[9]
Non- intrusive speech intelligibility prediction using convolutional neu- ral networks,
A. H. Andersen, J. M. de Haan, Z.-H. Tan, and J. Jensen, “Non- intrusive speech intelligibility prediction using convolutional neu- ral networks,” IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 26, no. 10, p. 1925–1939, Oct. 2018
1925
-
[10]
Mbi-net: A non-intrusive multi-branched speech intelligibility prediction model for hearing aids,
R. E. Zezario, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “Mbi-net: A non-intrusive multi-branched speech intelligibility prediction model for hearing aids,” in Interspeech, 2022
2022
-
[11]
Speech foundation models on intelligi- bility prediction for hearing-impaired listeners,
S. Cuervo and R. Marxer, “Speech foundation models on intelligi- bility prediction for hearing-impaired listeners,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 1421–1425
2024
-
[12]
Speech intelligibility pre- diction for hearing-impaired listeners with the leap model,
J. Roßbach, R. Huber, S. R ¨ottges, C. F. Hauth, T. Biberger, T. Brand, B. T. Meyer, and J. Rennies, “Speech intelligibility pre- diction for hearing-impaired listeners with the leap model,” in In- terspeech 2022, 2022, pp. 3498–3502
2022
-
[13]
Non-intrusive speech intelligibility pre- diction for hearing-impaired users using intermediate asr features and human memory models,
R. Mogridge, G. Close, R. Sutherland, T. Hain, J. Barker, S. Goetze, and A. Ragni, “Non-intrusive speech intelligibility pre- diction for hearing-impaired users using intermediate asr features and human memory models,” in ICASSP 2024 - 2024 IEEE Inter- national Conference on A...
2024
-
[14]
The 1st clarity prediction challenge: A ma- chine learning challenge for hearing aid intelligibility prediction,
J. Barker, M. Akeroyd, T. J. Cox, J. F. Culling, J. Firth, S. Graet- zer, H. Griffiths, L. Harris, G. Naylor, Z. Podwinska, E. Porter, and R. V . Munoz, “The 1st clarity prediction challenge: A ma- chine learning challenge for hearing aid intelligibility prediction,” in Inters...
2022
-
[15]
The 2nd clarity predic- tion challenge: A machine learning challenge for hearing aid in- telligibility prediction,
J. Barker, M. A. Akeroyd, W. Bailey, T. J. Cox, J. F. Culling, J. Firth, S. Graetzer, and G. Naylor, “The 2nd clarity predic- tion challenge: A machine learning challenge for hearing aid in- telligibility prediction,” in ICASSP 2024 - 2024 IEEE Interna- tional Conference on Ac...
2024
-
[16]
Factors affecting the use of speech testing in adult audiology,
B. J. Parmar, S. L. Rajasingam, J. K. Bizley, and D. A. Vickers, “Factors affecting the use of speech testing in adult audiology,” American journal of audiology, vol. 31, pp. 528 – 540, 2021
2021
-
[17]
Perception and cognition in the ageing brain: A brief review of the short- and long-term links be- tween perceptual and cognitive decline,
K. L. Roberts and H. A. Allen, “Perception and cognition in the ageing brain: A brief review of the short- and long-term links be- tween perceptual and cognitive decline,” Frontiers in Aging Neu- roscience, vol. 8, 2016
2016
-
[18]
Mathematical treatment of con- text effects in phoneme and word recognition
A. Boothroyd and S. Nittrouer, “Mathematical treatment of con- text effects in phoneme and word recognition.”The Journal of the Acoustical Society of America, vol. 84 1, pp. 101–14, 1988
1988
-
[19]
The contribution of auditory and cognitive factors to intelligibility of words and sentences in noise,
A. Heinrich and S. Knight, “The contribution of auditory and cognitive factors to intelligibility of words and sentences in noise,” in Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing , P. van Dijk, D. Bas ¸kent, E. Gaudrain, E. de Kleine, A. Wagner, an...
2016
-
[20]
Frequency selectivity and con- sonant intelligibility in sensorineural hearing loss
J. E. Preminger and T. L. Wiley, “Frequency selectivity and con- sonant intelligibility in sensorineural hearing loss.” Journal of speech and hearing research, vol. 28 2, pp. 197–206, 1985
1985
-
[21]
Recognition of spoken words by native and non-native listeners: talker-, listener-, and item-related factors
A. Bradlow and D. B. Pisoni, “Recognition of spoken words by native and non-native listeners: talker-, listener-, and item-related factors.” The Journal of the Acoustical Society of America , vol. 106 4 Pt 1, pp. 2074–85, 1999
1999
-
[22]
The relationship of speech intelligibility with hearing sensitivity, cognition, and perceived hearing difficulties varies for different speech percep- tion tests,
A. Heinrich, H. Henshaw, and M. A. Ferguson, “The relationship of speech intelligibility with hearing sensitivity, cognition, and perceived hearing difficulties varies for different speech percep- tion tests,” Frontiers in Psychology, vol. 6, 2015
2015
-
[23]
One-shot learning of object categories,
L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object categories,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 594–611, 2006
2006
-
[24]
Matching networks for one shot learning,
O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in Pro- ceedings of the 30th International Conference on Neural Infor- mation Processing Systems, ser. NIPS’16. Red Hook, NY , USA: Curran Associates Inc., 2016, ...
2016
-
[25]
Prototypical networks for few-shot learning,
J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates Inc., 2017, p. 4080–4090
2017
-
[26]
Speech audiometry in conductive and sensorineural hearing loss,
J. D. Hood and J. P. Poole, “Speech audiometry in conductive and sensorineural hearing loss,” British Journal of Audiology, vol. 5, no. 2, pp. 30–38, 1971
1971
-
[27]
Non-intrusive speech intelligibility prediction for hearing aids using whisper and metadata,
R. E. Zezario, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “Non-intrusive speech intelligibility prediction for hearing aids using whisper and metadata,” in Interspeech 2024 , 2024, pp. 3844–3848
2024
-
[28]
Transfer learning from whisper for microscopic intelligibility prediction,
P. Best, S. Cuervo, and R. Marxer, “Transfer learning from whisper for microscopic intelligibility prediction,” in Interspeech 2024, 2024, pp. 3839–3843
2024
- [29]
-
[30]
Robust Estimation of a Location Parameter,
P. J. Huber, “Robust Estimation of a Location Parameter,” The Annals of Mathematical Statistics , vol. 35, no. 1, pp. 73 – 101, 1964
1964
-
[31]
Adam: A method for stochastic op- timization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic op- timization,” in 3rd International Conference on Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, 2015
2015
-
[32]
SGDR: stochastic gradient descent with warm restarts,
I. Loshchilov and F. Hutter, “SGDR: stochastic gradient descent with warm restarts,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017
2017
-
[33]
Accurate, large minibatch sg d: training imagenet in 1 hour,
P. Goyal, “Accurate, large minibatch sg d: training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677, 2017
2017 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.