Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Analyzing the Importance of Blank for CTC-Based Knowledge Distillation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Nearby blanks can replace the CTC loss in speech distillation.

desk verdict Symmetric blank selection is a simple, citable idea and the blank-elimination negative result is useful; the label-free claim rests on oracle-n selection and no actual unsupervised run. read the letter →

arxiv 2506.01503 v2 pith:DHPHXC6K submitted 2025-06-02 cs.LG cs.SDeess.AS

Benedikt Hilmes , Nick Rossenbach , Ralf Schlüter This is my paper
classification cs.LGcs.SDeess.AS
keywords knowledgedistillationconnectionisttemporalclassificationblanktokensymmetricselectioneliminationspeechrecognitionself-supervisedteacherunlabeledaudio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a CTC speech-recognition student can be distilled from a large teacher without any transcriptions if the blank frames fed into the distillation loss are chosen symmetrically around the teacher's non-blank predictions. On TED-LIUMv2 this 'symmetric blank selection' at full distillation strength beats both plain knowledge distillation and blank elimination; on LibriSpeech it closes most of the gap to the best label-dependent setup. The practical stakes are that a medium-sized student could inherit a foundation model's knowledge from unlabeled audio alone, cutting the dependence on labeled data during distillation. The authors also show that the common blank-elimination trick is not reliable across corpora.

What carries the argument

The load-bearing object is symmetric blank selection, defined as follows: for every frame in which the teacher's argmax is a non-blank token, the KLD distillation loss is also evaluated on the n frames immediately to its left and right; all other blank positions are masked out. This turns the distillation objective into a position-dependent subset of the teacher output. It is paired with the overall training loss $L = \lambda L_{KD} + (1-\lambda) L_{CTC}$, so $\lambda = 1.0$ means no CTC loss at all. The mechanism works because CTC posteriors are 'peaky': most blank mass sits in long runs, and only boundary blanks need to be transferred to teach the student the teacher's alignment.

What would settle it

Run the same scale-1.0 symmetric-selection distillation on a corpus whose blank-run statistics differ substantially from TEDv2 and LibriSpeech, using the $n$ values the paper reports; if the WER gap to the scale-0.25 CTC-assisted baseline grows beyond what the paper observed, the claim that CTC can be dropped without degradation fails. A targeted calculation would measure the average blank-run length of each corpus and check whether the optimal $n$ tracks it.

Watch

Extended reading notes

Core claim

The central discovery is that the blank positions in a CTC teacher's posterior are not all alike: blanks immediately adjacent to non-blank frames carry the alignment information, while long runs of blanks can be ignored. By keeping only the n frames around each non-blank frame in the KLD term, the authors are able to set the distillation scale to 1.0, meaning the CTC loss is removed entirely. With this setting they report 5.6% WER on TEDv2 dev, better than plain KD (6.3%) and blank elimination (6.4%), and on LibriSpeech dev-other they reach 5.8%, against 6.9% for blank elimination and a 5.9% plain-KD baseline at scale 1.0. The conclusion the authors draw is that label-free distillation is possible if a limited, positional subset of blanks is kept.

Load-bearing premise

The whole claim rests on having the right symmetric-selection radius $n$; the paper tunes $n$ separately per corpus and scale and offers no automatic way to choose it, so if $n$ does not transfer to new or truly unlabeled data, dropping the CTC loss stops working.

Editorial extensions

If this is right

  • With symmetric selection and $\lambda = 1.0$, distillation no longer needs target labels, so unlabeled audio can be included in training.
  • Blank elimination is not a safe default: it helps TEDv2 but hurts LibriSpeech at scale 1.0, so dropping the CTC loss under blank elimination is not generally valid.
  • Distilling a few neighboring blanks is enough to recover the teacher's alignment information; full blank distributions are unnecessary.
  • The optimal radius $n$ is corpus-dependent, so the method currently requires tuning $n$ for each data condition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the optimal radius $n$ likely tracks the corpus's silence structure, since CTC's blank covers both 'wait' and 'silence'; a rule that estimates blank-run lengths from unlabeled teacher outputs could set $n$ automatically, removing the remaining hyperparameter.
  • We infer the positional-selection mechanism should transfer to any CTC-style teacher with peaky posteriors, because it exploits spikiness rather than the teacher's specific pretraining objective.
  • We infer that the differing outcomes of threshold and random selection across corpora suggest blank location, not blank quantity, is what matters; this could be tested by ablating blanks at fixed distances from non-blank frames.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies blank-token handling in CTC-based knowledge distillation for automatic speech recognition. It proposes a symmetric selection method that, for each frame where the teacher's argmax is non-blank, also includes the n neighboring frames in the KLD distillation loss. The authors experiment with different distillation scales (0.25, 0.9, 1.0) on TED-LIUMv2 and LibriSpeech, using a fine-tuned HuBERT teacher and a 42M-parameter Conformer student. Their central claim is that with scale 1.0 (i.e., no CTC loss) and symmetric blank selection, the student matches or nearly matches the performance of CTC-assisted knowledge distillation, thereby removing the dependence on labeled target labels and potentially enabling distillation on untranscribed audio.

Significance. If the result holds, the paper offers a simple and practical modification to CTC-based knowledge distillation: by keeping only blanks adjacent to non-blank positions, the CTC loss can be dropped without performance loss, which would allow training on unlabeled audio. The paper is strong on reproducibility: it uses a public teacher, a detailed training recipe, and provides open-source code. The experimental design covers two corpora, multiple distillation scales, and several blank-selection baselines. However, the evidence for the headline claim is weakened by the absence of variance estimates, by explicit best-of-n hyperparameter selection on the dev set, and by the lack of any actual experiment on untranscribed audio. The paper is honest about the hyperparameter-sensitivity limitation, but that limitation is load-bearing for the label-free claim.

major comments (4)
  1. [Section 5.2 / Table 2] The headline result is based on selecting, for each corpus and each distillation scale, the best symmetric-selection radius n from n=1..5 and then reporting that best value. The paper states this explicitly: "we present the best value for the corresponding set in the table." With single-run experiments and no variance or confidence intervals, the reported advantages over baseline KD on LibriSpeech are small (e.g., symmetric scale 1.0: 5.8/6.2 vs. KD scale 0.25: 5.6/6.1; the gap is 0.2/0.1 WER). Best-of-five selection over one seed can plausibly account for differences of this size. Because the scale-1.0 setting is the one intended for label-free distillation, the method as specified is an oracle-n report rather than a transferable recipe. I request either multiple seeds and variance reporting, or a sensitivity analysis showing that the result is stable across a range of n without dev-set selection.
  2. [Section 4.2 / Section 5] The paper claims that symmetric selection at scale 1.0 removes the dependence on target labels and "potentially" allows distillation on untranscribed audio. However, no experiment actually trains a student without target labels: all reported runs use the labeled corpora, the teacher is fine-tuned on labeled transcriptions (Section 4.2), and even the scale-1.0 runs use the same labeled audio, only omitting the CTC loss. To support the label-free claim, the authors should provide at least a proof-of-concept on genuinely unlabeled audio, or clarify how the teacher is obtained without labels and how the selection hyperparameters would be chosen in that setting.
  3. [Equations (2) and (3)] The knowledge-distillation loss is written as p_tea(c) * p_tea(c) / p_stu(c), which is not the Kullback-Leibler divergence; the standard KLD includes a logarithm, p_tea(c) * log(p_tea(c)/p_stu(c)). If this is a typographical error in the printed equations, it should be corrected because the loss definition is central to the paper. If the actual implementation uses the printed form, then the paper does not evaluate KLD-based distillation as claimed and the results need to be reinterpreted. Please clarify the exact objective used in the experiments.
  4. [Section 6] The paper acknowledges that the optimal symmetric-selection radius n differs across corpora and distillation scales and states that no automatic mechanism is provided. This is a crucial limitation for the central claim. For scale 1.0, which is the configuration proposed for unsupervised data, there is no described way to select n without a labeled development set. The paper should either propose a principled automatic selection rule (e.g., based on teacher blank statistics) or demonstrate that performance is robust to a reasonable range of n on both corpora, so that the method can be applied without per-corpus tuning.
minor comments (4)
  1. [Table 2 caption] The caption uses dagger and asterisk symbols in the text but they are not typeset in the table body in the version I read; please ensure the symbols are visible and explained consistently.
  2. [Section 5.2] The sentence "even though in all cases the ranges as stated in Section 3.2 where tested" contains a typo; "where" should be "were".
  3. [Figure 1] The x-axis label "Symmetric Selection Keep Value" is ambiguous; please clarify that this is the radius n, not the proportion of kept positions.
  4. [Section 1] The reference to "KD was initially proposed in [4]" is fine, but the sentence flow could be improved by separating the general KD introduction from CTC-specific prior work.

Circularity Check

0 steps flagged · score 2.0 of 10

No definitional circularity: symmetric-blank KD losses are explicitly defined and evaluated on external held-out test sets; the only self-citations (RETURNN, the peaky-behavior analysis) are background or software references, not load-bearing.

full rationale

The paper's derivation chain is self-contained. The central claim—that symmetric blank selection allows removing the CTC term (Eq. 4: L = λLKD + (1−λ)LCTC, scale 1.0) without degrading WER—is an empirical claim checked against the held-out TED-LIUMv2 test and LibriSpeech test-other sets (Table 2), which are external benchmarks. The symmetric mask (Section 3.2.1) is a novel, explicit function of the teacher's frame-wise argmax and radius n; it is not defined in terms of the student's loss, the CTC objective, or the test WER, so no quantity reduces to its input by construction. The reported numbers are best-of-n over dev-chosen hyperparameters (the paper discloses: 'we present the best value for the corresponding set in the table'), and optimal n differs across corpora and scales (Section 6); but test WERs are genuinely held out, so this is a multiple-comparison/robustness concern—a correctness-risk issue, not a definitional equivalence. Self-citations are minor and non-load-bearing: [14] is a software framework citation, and [16] (Zeyer, Schlüter, Ney) supports only the background 'peaky behavior' observation, which is independently corroborated by external refs [17–19] and by the paper's own distribution statistics (54% non-blank; 66–76% coverage under symmetric selection). The 'untranscribed audio' capability is an extrapolation—the teacher is fine-tuned on labeled data and no unlabeled-data run is performed—but that is a scope/evidence gap, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's results rest on standard CTC and KLD mathematics, a fine-tuned HuBERT teacher, and several corpus-specific hyperparameters selected on dev sets. No new entities are introduced.

free parameters (4)
  • Distillation scale lambda = 0.25, 0.9, 1.0 tested; 0.9 or 1.0 best for TEDv2, 0.25 best for LBS
    Interpolation weight between KD and CTC losses in Eq. 4; chosen from initial experiments.
  • Symmetric selection radius n = 1 to 5 tested; n=2 best for TEDv2 scale 1.0, n=4/3/2 for LBS at different scales
    Number of neighboring blank frames kept around non-blank frames; tuned per corpus and scale.
  • Probability threshold alpha = 0.95, 0.9, 0.8 tested; best alpha=0.9 for TEDv2 scale 1.0
    Keep blank frames with blank probability below alpha; tuned per corpus.
  • Random selection scale beta = 0.5, 1.0, 2.0 tested; best beta=0.5 for TEDv2 scale 1.0
    Proportion of randomly selected blank frames relative to non-blank count; control experiment.
assumptions (5)
  • standard math CTC marginalizes over alignments with the blank label; standard definition from Graves et al. 2006.
    Used throughout, Eq. (1).
  • standard math Knowledge distillation minimizes KLD between teacher and student posterior distributions.
    Eq. (2) and (3), though Eq. (2) has an apparent typo (missing log).
  • domain assumption The fine-tuned HuBERT teacher provides a good distillation target for the student.
    Section 4.2: teacher fine-tuned on each corpus for 10 epochs; no alternative teacher compared.
  • domain assumption Downsampling teacher outputs by factor 2 to match student frame rate does not lose useful information.
    Section 4.2: authors report no WER degradation, but this is only checked on final WER, not on frame-level accuracy.
  • domain assumption WER with a 4-gram language model is the appropriate evaluation metric.
    Section 4.1; all comparisons use this metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing the Importance of Blank for CTC-Based Knowledge Distillation." pith.science (2026). https://pith.science/paper/DHPHXC6K

@misc{pith2026250601503,
  author       = {Pith},
  title        = {Pith review of: Analyzing the Importance of Blank for CTC-Based Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHPHXC6K}},
  note         = {Machine review of arXiv:2506.01503}
}
read the original abstract

With the rise of large pre-trained foundation models for automatic speech recognition new challenges appear. While the performance of these models is good, runtime and cost of inference increases. One approach to make use of their strength while retaining efficiency is to distill their knowledge to smaller models during training. In this work, we explore different CTC-based distillation variants, focusing on blank token handling. We show that common approaches like blank elimination do not always work off the shelf. We explore new blank selection patterns as a potential sweet spot between standard knowledge distillation and blank elimination mechanisms. Through the introduction of a symmetric selection method, we are able to remove the CTC loss during knowledge distillation with minimal to no performance degradation. With this, we make the training independent from target labels, potentially allowing for distillation on untranscribed audio data.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 40 canonical work pages

  1. [1]

    These models are trained on large amounts of data either in a supervised or unsu- pervised fashion

    Introduction Recently, foundation models emerged as the new state-of-the- art in automatic speech recognition (ASR). These models are trained on large amounts of data either in a supervised or unsu- pervised fashion. Examples for these large pre-trained models are Whisper [1], Hubert [2] or Wav2Vec [3]. While the training methods differ, they perform well...

    work page

  2. [2]

    Analyzing the Importance of Blank for CTC-Based Knowledge Distillation

    Balanced distillation on blank and non-blank; 3. Learning a posterior matrix over a factorized KD loss. For architectures like RNN-T, distilling the model on the teachers outputs only is more common [11, 12]. Most CTC- based approaches combine the KD objective with the CTC ob- jective [9, 10]. This is why as a baseline for this work we use a setup that in...

    work page Pith review arXiv 2025

  3. [3]

    peaky behavior

    Connectionist Temporal Classification In this work we consider ASR models using connectionist tem- poral classification (CTC) [15]. CTC is used to align an acoustic signal xT 1 and a label sequence aS 1 through the introduction of a blank label, which extends aS 1 to length T . An alignment yT 1 is a valid alignment for xT 1 and aS 1 iff B(yT 1 ) = aS 1 ,...

    work page

  4. [4]

    Knowledge Distillation Knowledge distillation (KD) describes the process of transfer- ring the information embedded in a so called teacher model into a (usually smaller) student model [4]. There are multiple vari- ations of KD, which range from pseudo-labeling unsupervised data via the teacher, to training the student to have a similar out- put distributi...

    work page

  5. [5]

    Since we hypothesize that within-sequence blanks might be more important than blank predictions outside of the sequence, we add a chain of experiments where we only trim the blanks before the first and after the last non blank prediction of the teacher from the KLD calculation. 3.2.2. Probability-based selection Instead of relying on our non-blank positio...

    work page

  6. [6]

    Data In this work we make use of the two English datasets TED- LIUMv2 (TEDv2) [22] and LibriSpeech (LBS) [23]

    Experimental Setup 4.1. Data In this work we make use of the two English datasets TED- LIUMv2 (TEDv2) [22] and LibriSpeech (LBS) [23]. TEDv2 consists of 207 hours of TED talks, while LBS offers 960 hours of audio book recordings for training. For LibriSpeech we report results on dev-other and test-other. We split the data into sub-epochs of 5 for TEDv2 an...

    work page

  7. [7]

    Baselines The baseline performance of our models can be seen in Table 1

    Experiments 5.1. Baselines The baseline performance of our models can be seen in Table 1. Comparing the teacher with our baseline trained without any KD, we can see that the teacher outperforms the baseline by around 30% WER relative across multiple corpora and test sets. This is to be expected, as the teacher not only was trained on a lot more data, but ...

    work page

  8. [8]

    This might be due to the fact that blank distributions induced by CTC training might depend a lot on the data, as blank serves both as a wait and a silence token

    Limitation and Future Work One of the caveats of most of our blank aware approaches to im- proving CTC-based KD is the introduction of an additional hy- perparameter to tune, as our experiments show different settings are optimal for different corpora and distillation scales. This might be due to the fact that blank distributions induced by CTC training m...

    work page

Show all 42 references
  1. [9]

    For this we used a large pre-trained teacher to distill knowledge to a medium sized student for two different English corpora

    Conclusion In this work we analyzed the role of blank symbols during knowledge distillation for CTC-based ASR. For this we used a large pre-trained teacher to distill knowledge to a medium sized student for two different English corpora. We show that baseline extensions as bla...

  2. [10]

    Clusters4Future

    Acknowledgments This work was partially supported by NeuroSys, which as part of the initiative “Clusters4Future” is funded by the Fed- eral Ministry of Education and Research BMBF (funding IDs 03ZU2106DA and 03ZU2106DD), and by the project RESCALE within the program AI Lightho...

  3. [11]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in International Conference on Machine Learning , 2022

    2022

  4. [12]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. rahman Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  5. [13]

    wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,” in Proceedings of the 34th International Conference on Neural Information Processing Systems , ser. NIPS ’20. Red Hook, NY , USA: Curran Associate...

    2020

  6. [14]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS Deep Learning and Representation Learning Workshop 2014, 2014

    2014

  7. [15]

    An investigation of a knowl- edge distillation method for ctc acoustic models,

    R. Takashima, S. Li, and H. Kawai, “An investigation of a knowl- edge distillation method for ctc acoustic models,” ICASSP 2018, pp. 5809–5813, 2018

    2018

  8. [16]

    Knowledge distillation for sequence model,

    M. Huang, Y . You, Z. Chen, Y . Qian, and K. Yu, “Knowledge distillation for sequence model,” in Interspeech 2018, 2018, pp. 3703–3707

    2018

  9. [17]

    Guiding ctc posterior spike tim- ings for improved posterior fusion and knowledge distillation,

    G. Kurata and K. Audhkhasi, “Guiding ctc posterior spike tim- ings for improved posterior fusion and knowledge distillation,” in Interspeech, 2019

    2019

  10. [18]

    Improved knowledge distillation from bi-directional to uni-directional lstm ctc for end-to-end speech recognition,

    ——, “Improved knowledge distillation from bi-directional to uni-directional lstm ctc for end-to-end speech recognition,” in 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 411–417

    2018

  11. [19]

    Knowledge distillation for ctc-based speech recognition via con- sistent acoustic representation learning,

    S. Tian, K. Deng, Z. Li, L. Ye, G. Cheng, T. Li, and Y . Yan, “Knowledge distillation for ctc-based speech recognition via con- sistent acoustic representation learning,” in Interspeech, 2022

    2022

  12. [20]

    Factorized and progressive knowledge distillation for ctc-based asr models,

    S. Tian, Z. Li, Z. Lyv, G. Cheng, Q. Xiao, T. Li, and Q. Zhao, “Factorized and progressive knowledge distillation for ctc-based asr models,” Speech Communication, vol. 160, p. 103071, 2024

    2024

  13. [21]

    Efficient knowledge distillation for rnn-transducer models,

    S. Panchapagesan, D. S. Park, C.-C. Chiu, Y . Shangguan, Q. Liang, and A. Gruenstein, “Efficient knowledge distillation for rnn-transducer models,” ICASSP 2021, pp. 5639–5643, 2021

    2021

  14. [22]

    Robust knowledge distillation from rnn-t models with noisy training labels using full-sum loss,

    M. Zeineldeen, K. Audhkhasi, M. K. Baskar, and B. Ramabhad- ran, “Robust knowledge distillation from rnn-t models with noisy training labels using full-sum loss,” in IEEE International Con- ference on Acoustics, Speech, and Signal Processing , Rhodes, Greece, Jun. 2023

    2023

  15. [23]

    Sisyphus, a workflow manager de- signed for machine translation and automatic speech recognition,

    J. Peter, E. Beck, and H. Ney, “Sisyphus, a workflow manager de- signed for machine translation and automatic speech recognition,” in EMNLP 2018: System Demonstrations, Brussels, Belgium, Oc- tober 31 - November 4, 2018, pp. 84–89

    2018

  16. [24]

    Returnn: The RWTH extensible training framework for universal recurrent neural networks,

    P. Doetsch, A. Zeyer, P. V oigtlaender, I. Kulikov, R. Schlüter, and H. Ney, “Returnn: The RWTH extensible training framework for universal recurrent neural networks,” in ICASSP 2017, New Or- leans, LA, USA, March 5-9, 2017, pp. 5345–5349

    2017

  17. [25]

    Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,

    A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” inProceedings of the 23rd International Conference on Machine Learning , ser. ICML ’06. New York, NY , USA: A...

    2006

  18. [26]

    Why does ctc result in peaky behavior?

    A. Zeyer, R. Schlüter, and H. Ney, “Why does ctc result in peaky behavior?” Preprint arXiv:2105.14849, May 2021

    2021 arXiv

  19. [27]

    Connectionist temporal classifica- tion with maximum entropy regularization,

    H. Liu, S. Jin, and C. Zhang, “Connectionist temporal classifica- tion with maximum entropy regularization,” in Neural Informa- tion Processing Systems, 2018

    2018

  20. [28]

    Reinterpreting ctc training as iterative fit- ting,

    H. Li and W. Wang, “Reinterpreting ctc training as iterative fit- ting,” Pattern Recognition, vol. 105, p. 107392, 2020

    2020

  21. [29]

    Less peaky and more accurate ctc forced align- ment by label priors,

    R. Huang, X. Zhang, Z. Ni, L. Sun, M. Hira, J. Hwang, V . Manohar, V . Pratap, M. Wiesner, S. Watanabe, D. Povey, and S. Khudanpur, “Less peaky and more accurate ctc forced align- ment by label priors,” ICASSP 2024, pp. 11 831–11 835, 2024

    2024

  22. [30]

    Investigation of sequence- level knowledge distillation methods for ctc acoustic models,

    R. Takashima, S. Li, and H. Kawai, “Investigation of sequence- level knowledge distillation methods for ctc acoustic models,” ICASSP 2019 - 2019 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , pp. 6156–6160, 2019

    2019

  23. [31]

    Acoustic modelling with cd-ctc-smbr lstm rnns,

    A. W. Senior, H. Sak, F. de Chaumont Quitry, T. N. Sainath, and K. Rao, “Acoustic modelling with cd-ctc-smbr lstm rnns,” 2015 IEEE Workshop on Automatic Speech Recognition and Under- standing (ASRU), pp. 604–609, 2015

    2015

  24. [32]

    Enhancing the ted- lium corpus with selected data for language modeling and more ted talks,

    A. Rousseau, P. Deléglise, and Y . Estève, “Enhancing the ted- lium corpus with selected data for language modeling and more ted talks,” in International Conference on Language Resources and Evaluation, 2014

    2014

  25. [33]

    Lib- rispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in ICASSP 2015, 2015, pp. 5206–5210

    2015

  26. [34]

    Joint-sequence models for grapheme-to- phoneme conversion,

    M. Bisani and H. Ney, “Joint-sequence models for grapheme-to- phoneme conversion,” Speech Communication, vol. 50, no. 5, pp. 434–451, May 2008

    2008

  27. [35]

    Libri-light: A benchmark for asr with limited or no supervision,

    J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar’e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. rahman Mohamed, and E. Dupoux, “Libri-light: A benchmark for asr with limited or no supervision,” ICASSP 2020,...

    2020

  28. [36]

    Compari- son of soft and hard target rnn-t distillation for large-scale asr,

    D. Hwang, K. Chai Sim, Y . Zhang, and T. Strohman, “Compari- son of soft and hard target rnn-t distillation for large-scale asr,” in ICASSP 2023, 2023, pp. 1–5

    2023

  29. [37]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech 2020, Virtual Event, Shanghai, China, October 25- 29, 2020. ISCA, 2020, pp. 5036–5040

    2020

  30. [38]

    Self-attention with rela- tive position representations,

    P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with rela- tive position representations,” in North American Chapter of the Association for Computational Linguistics, 2018

    2018

  31. [39]

    Specaugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in Interspeech, 2019

    2019

  32. [40]

    Dropout: A simple way to prevent neural net- works from overfitting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural net- works from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014

    1929

  33. [41]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in International Conference on Learning Representations , 2019

    2019

  34. [42]

    Flashlight: Enabling inno- vation in tools for machine learning,

    J. Kahn, V . Pratap, T. Likhomanenko, Q. Xu, A. Hannun, J. Cai, P. Tomasello, A. Lee, E. Grave, G. Avidov, B. Steiner, V . Liptchin- sky, G. Synnaeve, and R. Collobert, “Flashlight: Enabling inno- vation in tools for machine learning,” 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.