Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Whisper grows overconfident in noise; selective token-level recalibration cuts calibration error by 58%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

In noisy speech, Whisper often assigns high confidence to wrong tokens; a selective token-level temperature-scaling calibrator reduces ECE by 58% on the R-SPIN low-SNR range.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A genuinely useful empirical finding (Whisper gets sharply overconfident below -5 dB SNR) plus a sensible selective token-level temperature-scaling fix, held back by train/test sharing the same synthetic noise recipe. the 4 major comments →

arxiv 2509.07195 v1 pith:PWVWOE4W submitted 2025-09-08 eess.AS

Identifying and Calibrating Overconfidence in Noisy Speech Recognition

Mingyue Huo , Yuheng Zhang , Yan Tang This is my paper
classification eess.AS
keywords confidence calibrationoverconfidenceautomatic speech recognitionWhispertemperature scalingnoise robustnessselective calibrationexpected calibration error
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that end-to-end speech recognizers like Whisper become systematically overconfident in additive noise—wrong tokens routinely carry confidence above 0.7 at low signal-to-noise ratios—and that this miscalibration can be repaired without retraining the recognizer. It proposes a lightweight post-hoc pipeline that first flags likely-overconfident tokens with a small classifier and then applies temperature scaling to exactly those tokens. On a phonetically balanced test set, the method cuts expected calibration error by 58% (0.086 to 0.036 absolute) in the −18 to −5 dB range, triples normalized cross entropy, and halves the mass of overconfident errors, while leaving already-reliable high-SNR predictions largely unchanged. The value of the claim is that downstream uses of confidence scores—error detection, selective re-recognition, semi-supervised sample selection, and high-stakes transcription—become trustworthy in noisy acoustic conditions without changing the ASR model.

Core claim

In the paper's own terms, the central claim is that Whisper's softmax confidence is reliable on clean and mildly noisy speech but breaks down under severe noise, and that a selective token-level calibration framework fixes this breakdown. The authors first document the phenomenon: in the low-SNR range, 10–20% of tokens are incorrect yet predicted with confidence above 0.7, and ECE/NCE degrade sharply below about −5 dB. They then train a classifier and a temperature predictor on a shared per-token feature vector, calibrating only flagged tokens. The result is a 58% relative ECE reduction (0.050 absolute) over the uncalibrated baseline across the −18 to −5 dB range, with overconfident mass fal

What carries the argument

The load-bearing mechanism is the token-level selective calibrator: a two-stage pipeline built around a per-token feature vector with three feature groups—softmax uncertainty (top-1 probability, top-2 margin, entropy), token plausibility (token ID embedding, positional index, top-5 raw logits), and an utterance-level acoustic embedding derived from the Mel spectrogram via attention pooling. A two-layer MLP overconfidence classifier (weighted binary cross-entropy, positive weight 7) decides which tokens to adjust, and a second network predicts a temperature T > 1 for each flagged token via softplus, guaranteeing that scaling can only lower confidence. The full reassembled logit sequence is tr

Load-bearing premise

The method's training and evaluation both use the same kind of synthetic temporally-modulated speech-shaped noise; if real-world noise does not induce the same overconfidence pattern in Whisper, the classifier and temperatures may not transfer.

What would settle it

Run the trained calibrator on real recorded noisy speech (e.g., cafeteria babble, traffic, impulsive sounds) at the same SNR range and measure ECE and overconfident mass. If the overconfidence classifier flags tokens in mismatched noise with much lower recall, or the ECE reduction drops below roughly half of the reported 58%, the synthetic-noise transfer assumption fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, post-hoc calibration can restore reliable confidence in noisy ASR without retraining or modifying the underlying model, making error detection and selective re-recognition more trustworthy in the low-SNR regime.
  • Since the method is token-level and selective, it preserves calibration at high SNR while improving the severe-noise range, meaning one fixed post-processing stage can replace SNR-dependent tuning.
  • The reported NCE tripling implies confidence scores become substantially more informative about correctness, improving their utility as training signals in semi-supervised learning on noisy audio.
  • The modular design means the calibration stage can be retrained with any differentiable confidence-reduction strategy beyond temperature scaling, for example token-level credence or abstention rules.
  • Feature ablation shows that fine-grained uncertainty signals (top-k logits) are essential to avoid overcorrection, so future post-hoc calibrators should keep multi-logit information rather than relying on top-1 confidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The classifier's low precision (≈0.3) means most flagged tokens are actually correct; this suggests the reported calibration gains come from selective softening rather than accurate error detection, and that a more precise detector could yield even larger ECE reductions or allow a gentler temperature scale.
  • The paper's SNR-specific results show a mid-SNR range (−5 to +5 dB) where calibration slightly degrades; an SNR-aware gating rule (only calibrating when utterance-level noise is severe) could avoid that degradation, but the paper found explicit SNR features did not consistently help—so this tension remains open.
  • Because the acoustic embedding is an utterance-level summary, the method assumes noise is roughly stationary within an utterance; in real-world audio with abrupt level changes or intermittent interference, a token-aligned or sliding-window acoustic feature would be needed for the same gains—a testable extension of the paper's future-work remark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies confidence calibration of OpenAI's Whisper-medium.en ASR model under additive noise. It constructs temporally modulated speech-shaped noise and mixes it with LibriTTS for training and with the R-SPIN corpus for evaluation at SNR levels from +10 to −18 dB. The authors report that, at low SNR, 10–20% of decoded tokens are incorrect with softmax confidence above 0.7, and ECE/NCE degrade sharply. They propose a two-stage post-hoc framework: a binary MLP classifier flags overconfident tokens using softmax statistics, token embedding/position/top-k logits, and an attention-pooled Mel embedding; a small network then predicts a token-level temperature (T>1) for flagged tokens. Both modules are trained jointly with weighted BCE, CE on flagged tokens, and soft-binned ECE. On R-SPIN's low-SNR range (−18 to −5 dB), they report ECE improving from 0.086 to 0.036 (58% relative), NCE from 0.064 to 0.192, and overconfident mass from 11.1% to 6.6%, while high-SNR calibration is largely preserved. They include ablations on features and comparisons to an SNR-based baseline and utterance-level temperature scaling.

Significance. If the results hold, the paper makes a useful empirical observation—Whisper's token-level confidence is systematically overconfident in severe noise—and proposes a lightweight, post-hoc, selective temperature-scaling framework with plausible feature design. The reported metrics are internally consistent (ECE 0.086→0.036; NCE 0.064→0.192; overconfident mass 11.1%→6.6%), and the ablations (Table III) and comparisons to SNR-based and utterance-level alternatives are informative. However, the external validity and baseline comparisons need substantial strengthening before the central claim can be accepted.

major comments (4)
  1. [§III-A and §IV-D] Training and evaluation use the same synthetic noise recipe (speech-shaped noise with speech-envelope amplitude modulation). This matched-distribution design is the main threat to external validity: the overconfidence classifier and temperature network may learn cues specific to this noise, and §VI-B acknowledges that the synthetic noise creates relatively uniform acoustic conditions. No evaluation on babble, traffic, stationary, or real noise is provided, so the headline 0.050 ECE reduction and NCE tripling are not established for real-world noisy conditions. Add cross-noise evaluations or restrict the claims.
  2. [§V-A, Table I] The only non-selective baseline is 'SNR-based calibration', which requires oracle SNR and grid search. A standard global temperature scaling baseline (Guo et al. [19]) is missing, as is the training-free entropy-based method [28] cited as robust under 0–30 dB. Without these comparisons, the improvement attributable to the selective token-level mechanism versus simple temperature scaling is unclear. Please add these baselines.
  3. [§V-C] The final model configuration appears to be selected using the evaluation corpus: the ablation text says 'further analysis on the phonetically balanced evaluation set reveals overcorrection,' and the final model's ECE improvement (0.0502) is reported in the same table. This test-set-informed selection can inflate the reported 58% reduction in Table I. Specify which ablations used a validation split and confirm the final model was not chosen using R-SPIN.
  4. [§III-B and §V-A] Obvious hallucination outputs are excluded before computing all confidence statistics and calibration metrics. Since hallucinations are a major source of high-confidence errors in Whisper under noise, excluding them may overstate the overconfidence diagnosis and the post-calibration ECE. Please report metrics with hallucinations included or justify the exclusion. Also, all results are single-run point estimates with no error bars; with 400 R-SPIN sentences, compute variability across seeds or SNR sub-samples.
minor comments (4)
  1. [Abstract / Table II] The abstract's 'triples NCE' refers to the low-SNR aggregate; Table II shows NCE remains negative in the [-18,-15] dB bin after calibration (-0.0511). Please contextualize the aggregate claim.
  2. [§IV-B] The overconfidence threshold (0.7), positive class weight (w=7), and loss weights are set empirically. Report sensitivity to these hyperparameters.
  3. [Eq. (10)] The calibrator is trained with a soft-binned ECE loss but evaluated with hard-binned ECE; a brief justification of this mismatch would improve reproducibility.
  4. [Fig. 1] Figure 1: ensure the right-panel confidence-bin annotations and axis labels are legible in the final version; the reader cannot easily extract the 0.7–1.0 bin behavior.

Circularity Check

0 steps flagged

No circularity: the reported ECE reduction is a held-out result on R-SPIN, trained on LibriTTS; no load-bearing self-citations or definitional equivalences.

full rationale

The paper's central claim is that a token-level selective temperature-scaling framework improves calibration under low-SNR conditions. The calibration modules are trained on LibriTTS utterances corrupted with the paper's own synthetic temporally-modulated speech-shaped noise (Section IV-D), and evaluated on the independent R-SPIN corpus (Section III-A) with the same noise construction. This is a train/evaluation corpus split, so the headline ECE reduction of 0.050 (58%) is not a fitted value on the evaluation set. The calibrator is optimized with a soft-binned ECE loss [38] and evaluated with hard-binned ECE (Equation 1); the two formulations are not equivalent by construction. The overconfidence indicator is consistently defined via a fixed 0.7 confidence threshold and incorrectness (Section III-B), used both for classifier labels and for the reported overconfident mass metric, but this is a consistent operational definition rather than a prediction of itself. Feature ablations (Table III) are reported and show robustness, and while some feature choices were informed by validation/test performance, this is model selection, not circular derivation. The single self-citation ([4], Huo et al.) appears only as motivational context for studying acoustic cue degradation and is not load-bearing. There is no invocation of a uniqueness theorem from the authors' prior work, no ansatz smuggled in via citation, and no renaming of a known result. The acknowledged limitation in Section VI-B that synthetic noise creates uniform acoustic conditions is a generalization concern, not an internal circularity. The derivation chain is self-contained against an external benchmark, so the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are introduced; the overconfidence classifier and temperature predictor are algorithmic components described by equations (5)-(11). The free parameters are the threshold, class weight, loss weights, and feature engineering choices, all selected by validation or ablation.

free parameters (4)
  • Overconfidence threshold = 0.7
    Empirically chosen in Section III-B to define the overconfidence label; all downstream metrics and the classifier depend on it.
  • Positive class weight w = 7
    Selected empirically in Section IV-B to handle class imbalance in the overconfidence classifier.
  • Loss weights (lambda_BCE, lambda_CE, lambda_ECE) = 0.5, 1.0, 10.0
    Grid-searched on the validation set (Section IV-D).
  • Top-k logits count k and audio truncation length = k=5, 3s
    Selected through ablations in Section V-C; the 3s truncation 'consistently outperforms the full-length input'.
axioms (4)
  • domain assumption Whisper's softmax probabilities are a valid token-level confidence measure
    The entire analysis and calibration framework operates on decoder softmax max posteriors (Section III-B), taking them as the confidence scores to be calibrated.
  • domain assumption Temporally modulated speech-shaped noise is representative of realistic acoustic degradation
    Training and evaluation both add the authors' synthetic noise; Section III-A motivates it as a controlled, speech-like approximation, but the method is not tested on other noise types.
  • domain assumption R-SPIN is an appropriate evaluation corpus for noisy ASR calibration
    R-SPIN is a phonetic/psychoacoustic benchmark with 400 sentences evaluated at all SNR levels (Section III-A); it is small and not designed for ASR calibration, so generalizability is limited.
  • standard math ECE and NCE are valid calibration metrics
    Definitions (1)-(4) in Section III-C follow standard practice from the cited literature.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Identifying and Calibrating Overconfidence in Noisy Speech Recognition." pith.science (2026). https://pith.science/paper/PWVWOE4W

@misc{pith2026250907195,
  author       = {Pith},
  title        = {Pith review of: Identifying and Calibrating Overconfidence in Noisy Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PWVWOE4W}},
  note         = {Machine review of arXiv:2509.07195}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern end-to-end automatic speech recognition (ASR) models like Whisper not only suffer from reduced recognition accuracy in noise, but also exhibit overconfidence - assigning high confidence to wrong predictions. We conduct a systematic analysis of Whisper's behavior in additive noise conditions and find that overconfident errors increase dramatically at low signal-to-noise ratios, with 10-20% of tokens incorrectly predicted with confidence above 0.7. To mitigate this, we propose a lightweight, post-hoc calibration framework that detects potential overconfidence and applies temperature scaling selectively to those tokens, without altering the underlying ASR model. Evaluations on the R-SPIN dataset demonstrate that, in the low signal-to-noise ratio range (-18 to -5 dB), our method reduces the expected calibration error (ECE) by 58% and triples the normalized cross entropy (NCE), yielding more reliable confidence estimates under severe noise conditions.

Figures

Figures reproduced from arXiv: 2509.07195 by Mingyue Huo, Yan Tang, Yuheng Zhang.

Figure 1
Figure 1. Figure 1: Impact of additive noise level on ASR recognition and confidence performance. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed token-level selective calibration framework. A shared feature vector is computed for each [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reliability diagrams before and after selective calibra [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RAS: a Reliability Oriented Metric for Automatic Speech Recognition

    cs.SD 2026-04 unverdicted novelty 5.0

    RAS is a reliability metric for ASR that trades off informativeness against error aversion via human-calibrated abstention, trained with supervised bootstrapping plus reinforcement learning to improve reliability whil...

  2. RAS: a Reliability Oriented Metric for Automatic Speech Recognition

    cs.SD 2026-04 unverdicted novelty 5.0

    RAS is a reliability-oriented metric for ASR that balances informativeness and error aversion via human-calibrated abstention, paired with a training method using supervised bootstrapping and reinforcement learning.

Reference graph

Works this paper leans on

39 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    Confidence measures for speech recognition: A survey,

    H. Jiang, “Confidence measures for speech recognition: A survey,” Speech communication, vol. 45, no. 4, pp. 455–470, 2005

    work page 2005

  2. [2]

    Evaluating OpenAI’s Whisper ASR: Perfor- mance analysis across diverse accents and speaker traits,

    C. Graham and N. Roll, “Evaluating OpenAI’s Whisper ASR: Perfor- mance analysis across diverse accents and speaker traits,”JASA Express Letters, vol. 4, no. 2, 2024

    work page 2024

  3. [3]

    Speech recognition in adverse conditions by humans and machines,

    C. Patman and E. Chodroff, “Speech recognition in adverse conditions by humans and machines,”JASA Express Letters, vol. 4, no. 11, 2024

    work page 2024

  4. [4]

    Release from same-talker speech-in-speech masking: Effects of masker intelligibility and other contributing factors,

    M. Huo, Y . Sun, D. Fogerty, and Y . Tang, “Release from same-talker speech-in-speech masking: Effects of masker intelligibility and other contributing factors,”The Journal of the Acoustical Society of America, vol. 156, no. 5, pp. 2960–2973, 2024

    work page 2024

  5. [5]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  6. [6]

    Evaluation of a noise-robust dsr front-end on aurora databases

    D. Macho, L. Mauuary, B. No ´e, Y . M. Cheng, D. Ealey, D. Jouvet, H. Kelleher, D. Pearce, and F. Saadoun, “Evaluation of a noise-robust dsr front-end on aurora databases.” inInterspeech, 2002, pp. 17–20

    work page 2002

  7. [7]

    A minimum- mean-square-error noise reduction algorithm on mel-frequency cepstra for robust speech recognition,

    D. Yu, L. Deng, J. Droppo, J. Wu, Y . Gong, and A. Acero, “A minimum- mean-square-error noise reduction algorithm on mel-frequency cepstra for robust speech recognition,” in2008 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2008, pp. 4041–4044

    work page 2008

  8. [8]

    An investigation of deep neural net- works for noise robust speech recognition,

    M. L. Seltzer, D. Yu, and Y . Wang, “An investigation of deep neural net- works for noise robust speech recognition,” in2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2013, pp. 7398–7402

    work page 2013

  9. [9]

    Audio augmentation for speech recognition

    T. Ko, V . Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition.” inInterspeech, vol. 2015, 2015, p. 3586

    work page 2015

  10. [10]

    SpecAugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” inProc. Interspeech 2019, 2019, pp. 2613–2617

    work page 2019

  11. [11]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449– 12 460, 2020

    2020

  12. [12]

    Speech en- hancement and recognition using multi-task learning of long short-term memory recurrent neural networks

    Z. Chen, S. Watanabe, H. Erdogan, and J. R. Hershey, “Speech en- hancement and recognition using multi-task learning of long short-term memory recurrent neural networks.” inInterspeech, 2015, pp. 3274– 3278

    work page 2015

  13. [13]

    Improved estimation, evaluation and applications of confidence measures for speech recognition

    M.-h. Siu, H. Gish, and F. Richardson, “Improved estimation, evaluation and applications of confidence measures for speech recognition.” in Eurospeech, 1997, pp. 831–834

    work page 1997

  14. [14]

    Posterior probability decoding, confi- dence estimation and system combination,

    G. Evermann and P. Woodland, “Posterior probability decoding, confi- dence estimation and system combination,” inProc. Speech Transcrip- tion Workshop, vol. 27. Baltimore, 2000, pp. 78–81

    work page 2000

  15. [15]

    Confidence measures for large vocabulary continuous speech recognition,

    F. Wessel, R. Schluter, K. Macherey, and H. Ney, “Confidence measures for large vocabulary continuous speech recognition,”IEEE Transactions on speech and audio processing, vol. 9, no. 3, pp. 288–298, 2002

    work page 2002

  16. [16]

    High-level approaches to confidence estimation in speech recognition,

    S. Cox and S. Dasmahapatra, “High-level approaches to confidence estimation in speech recognition,”IEEE Transactions on Speech and Audio processing, vol. 10, no. 7, pp. 460–471, 2002

    work page 2002

  17. [17]

    Combining information sources for confidence estimation with crf models

    M. S. Seigel, P. C. Woodlandet al., “Combining information sources for confidence estimation with crf models.” inInterspeech, 2011, pp. 905–908

    work page 2011

  18. [18]

    Calibration of confidence measures in speech recognition,

    D. Yu, J. Li, and L. Deng, “Calibration of confidence measures in speech recognition,”IEEE Transactions on Audio, Speech, and Language Pro- cessing, vol. 19, no. 8, pp. 2461–2473, 2011

    work page 2011

  19. [19]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inInternational conference on machine learning. PMLR, 2017, pp. 1321–1330

    2017

  20. [20]

    Improving ASR confidence scores for Alexa using acoustic and hy- pothesis embeddings,

    P. Swarup, R. Maas, S. Garimella, S. H. Mallidi, and B. Hoffmeister, “Improving ASR confidence scores for Alexa using acoustic and hy- pothesis embeddings,” inProc. Interspeech 2019, 2019, pp. 2175–2179

    work page 2019

  21. [21]

    Confidence measures in encoder-decoder models for speech recognition

    A. Woodward, C. Bonn ´ın, I. Masuda, D. Varas, E. Bou-Balust, and J. C. Riveiro, “Confidence measures in encoder-decoder models for speech recognition.” inInterspeech, 2020, pp. 611–615

    work page 2020

  22. [22]

    Confidence estimation for attention-based sequence-to- sequence models for speech recognition,

    Q. Li, D. Qiu, Y . Zhang, B. Li, Y . He, P. C. Woodland, L. Cao, and T. Strohman, “Confidence estimation for attention-based sequence-to- sequence models for speech recognition,” inICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6388–6392

    work page 2021

  23. [23]

    Learning word-level confidence for subword end- to-end asr,

    D. Qiu, Q. Li, Y . He, Y . Zhang, B. Li, L. Cao, R. Prabhavalkar, D. Bhatia, W. Li, K. Huet al., “Learning word-level confidence for subword end- to-end asr,” inICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6393–6397

    work page 2021

  24. [24]

    Blstm-based confi- dence estimation for end-to-end speech recognition,

    A. Ogawa, N. Tawara, T. Kano, and M. Delcroix, “Blstm-based confi- dence estimation for end-to-end speech recognition,” inICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6383–6387

    work page 2021

  25. [25]

    Multi-task learning for end-to-end asr word and utterance confidence with deletion prediction,

    D. Qiu, Y . He, Q. Li, Y . Zhang, L. Cao, and I. McGraw, “Multi-task learning for end-to-end asr word and utterance confidence with deletion prediction,” inInterspeech 2021, 2021, pp. 4074–4078

    2021

  26. [26]

    Adopting Whisper for confidence estimation,

    V . Aggarwal, S. S. Nair, Y . Verma, and Y . Jogi, “Adopting Whisper for confidence estimation,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    work page 2025

  27. [27]

    Improving confidence estimation on out-of-domain data for end-to-end speech recognition,

    Q. Li, Y . Zhang, D. Qiu, Y . He, L. Cao, and P. C. Woodland, “Improving confidence estimation on out-of-domain data for end-to-end speech recognition,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6537–6541

    work page 2022

  28. [28]

    Fast entropy-based methods of word-level confidence estimation for end-to-end automatic speech recognition,

    A. Laptev and B. Ginsburg, “Fast entropy-based methods of word-level confidence estimation for end-to-end automatic speech recognition,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 152–159

    work page 2022

  29. [29]

    Utterance-level neural confidence measure for end- to-end children speech recognition,

    W. Liu and T. Lee, “Utterance-level neural confidence measure for end- to-end children speech recognition,” in2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 449–456

    work page 2021

  30. [30]

    Proper error estimation and calibration for attention-based encoder-decoder models,

    M.-H. Lee and J.-H. Chang, “Proper error estimation and calibration for attention-based encoder-decoder models,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    work page 2024

  31. [31]

    Calibration of pre-trained transformers,

    S. Desai and G. Durrett, “Calibration of pre-trained transformers,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 295–302

    2020

  32. [32]

    An evaluation of word- level confidence estimation for end-to-end automatic speech recogni- tion,

    D. Oneat ¸˘a, A. Caranica, A. Stan, and H. Cucu, “An evaluation of word- level confidence estimation for end-to-end automatic speech recogni- tion,” in2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 258–265

    work page 2021

  33. [33]

    Calibrated selective classifica- tion,

    A. Fisch, T. Jaakkola, and R. Barzilay, “Calibrated selective classifica- tion,”arXiv preprint arXiv:2208.12084, 2022

    Pith/arXiv arXiv 2022

  34. [34]

    Im- proving predictor reliability with selective recalibration,

    T. P. Zollo, Z. Deng, J. C. Snell, T. Pitassi, and R. Zemel, “Im- proving predictor reliability with selective recalibration,”arXiv preprint arXiv:2410.05407, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    LibriTTS: A corpus derived from librispeech for text-to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from librispeech for text-to-speech,” pp. 1526–1530, 2019

    work page 2019

  36. [36]

    The revised speech perception in noise test (R-SPIN) in a multiple signal-to-noise ratio paradigm,

    R. H. Wilson, R. McArdle, K. L. Watts, and S. L. Smith, “The revised speech perception in noise test (R-SPIN) in a multiple signal-to-noise ratio paradigm,”Journal of the American Academy of Audiology, vol. 23, no. 08, pp. 590–605, 2012

    work page 2012

  37. [37]

    From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition

    A. C. Morris, V . Maier, and P. D. Green, “From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.” inInterspeech, 2004, pp. 2765–2768

    work page 2004

  38. [38]

    Soft calibration objectives for neural networks,

    A. Karandikar, N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs, “Soft calibration objectives for neural networks,”Advances in Neural Information Processing Systems, vol. 34, pp. 29 768–29 779, 2021

    work page 2021

  39. [39]

    Access: Advancing innovation: Nsf’s advanced cyberinfrastructure co- ordination ecosystem: Services & support,

    T. J. Boerner, S. Deems, T. R. Furlani, S. L. Knuth, and J. Towns, “Access: Advancing innovation: Nsf’s advanced cyberinfrastructure co- ordination ecosystem: Services & support,” inPractice and experience in advanced research computing 2023: Computing for the common good, 2023, pp. 173–176

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.