Label-Context-Dependent Internal Language Model Estimation for CTC

Hermann Ney; Minh-Nghia Phan; Ralf Schl\"uter; Zijian Yang

REVIEW 3 major objections 4 minor 35 references

Label-Context-Dependent Internal Language Model Estimation for CTC

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small LSTM trained by knowledge distillation can extract the implicit, label-context-dependent language model that a CTC speech recognizer learns through its encoder, and using that estimate for Bayes-corrected language model fusion…

desk verdict Useful CTC ILM estimation method with clean unregularized theory; the regularized variants that give the best WERs lack the same proof and may be better decoding priors rather than true ILM estimates. read the letter →

arxiv 2506.06096 v1 pith:BLWQ6IXI submitted 2025-06-06 cs.SD cs.CLcs.LGeess.AS

Zijian Yang , Minh-Nghia Phan , Ralf Schlüter , Hermann Ney This is my paper

classification cs.SDcs.CLcs.LGeess.AS

keywords CTCinternallanguagemodelknowledgedistillationspeechrecognitionfusionshallowcross-domainASRprefixprobability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Connectionist temporal classification (CTC) models assume each output label is independent of previous labels given the encoder state, yet with a powerful encoder they can still learn an implicit, label-context-dependent internal language model (ILM). The paper proposes estimating that ILM by knowledge distillation: the CTC model acts as teacher, and a small autoregressive LSTM is trained on CTC's label posteriors, computed from CTC prefix probabilities, using label-level and sequence-level criteria. Two regularizers, smoothing and acoustic masking, stop the student from simply memorizing the training transcriptions. In cross-domain evaluation on TED-LIUM2, the best estimator (label-level distillation with smoothing) gives more than 13% relative word-error-rate improvement over shallow fusion, and context-dependent ILMs beat context-independent priors, supporting the claim that CTC's ILM is context-dependent.

What carries the argument

The machinery is label-level knowledge distillation from CTC prefix probabilities. For a prefix $a_1^{s-1}$, the teacher's label posterior is $P(a|a_1^{s-1},X) = P(a_1^{s}\cdots|X)/P(a_1^{s-1}\cdots|X)$, where the prefix probability sums over all continuations; an <EOS> posterior is computed similarly. The student is a one-layer LSTM LM trained with a KL-divergence criterion, and two regularizers are added: smoothing interpolates the empirical joint distribution with the product of its marginals, and masking zeros out the acoustic frames of randomly chosen labels. This student is then used as $P_{\mathrm{ILM}}$ in Bayes-corrected shallow fusion decoding.

What would settle it

Train the same distillation pipeline on a CTC model whose encoder has no context modeling ability (for example, a frame-independent classifier) or on data where transcripts are randomly shuffled against acoustics. If the smoothed label-level KD ILM still produces a large cross-domain gain over shallow fusion, the improvement cannot be attributed to a genuinely learned context-dependent ILM; the expected result under the paper's claim is that the gain would shrink toward the unigram-prior baseline.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that CTC's implicit internal language model is label-context-dependent and can be extracted by student-teacher distillation. Defining the ILM as $P_{\mathrm{ILM}}(a_1^S) = \sum_X \Pr(X) P(a_1^S|X)$, the authors show that training a small LM $q_\theta$ to minimize KL divergence against CTC's label posteriors $P(a|a_1^{s-1}, X)$ — computed as ratios of CTC prefix probabilities — has a global optimum equal to the true ILM when the empirical distribution matches the true one. With finite data, the posterior is overconfident, so they smooth the empirical joint distribution or mask acoustic evidence at label boundaries. The resulting context-dependent ILM, especially the smoothed label-level variant, yields the best cross-domain WER, indicating the encoder has implicitly encoded language context that a unigram prior misses.

Load-bearing premise

The load-bearing premise is that the distilled LSTM's optimum equals the true ILM; that proof assumes unlimited training data, and the paper concedes that with finite data the student can learn an acoustic-language mixture rather than the pure context-dependent prior, with smoothing and masking as approximate fixes.

Editorial extensions

If this is right

In cross-domain recognition, context-dependent ILM estimates consistently beat context-independent priors, so a CTC's implicit ILM should be treated as a conditioned language model, not a unigram.
Using the distilled, smoothed label-level ILM for Bayes correction yields over 13% relative WER improvement over shallow fusion on TED-LIUM2, without retraining the acoustic model.
Adding a frame-level prior on top of the distilled label-level ILM gives little extra gain, indicating the two corrections carry overlapping information.
ILM perplexity does not predict recognition quality, so ILM estimators should be selected by downstream WER, not by perplexity.
Context length experiments suggest that a feedforward ILM with context 6 comes close to a full-context LSTM, but the right context length for CTC ILMs is still unresolved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A next step would be a targeted metric for label-distribution rebalancing, since ILM perplexity is uncorrelated with WER and existing metrics miss what the ILM correction is doing.
The same teacher-student recipe could be applied to RNN-T or attention-based models that lack an explicit LM module, testing whether the context-dependence found here is a general encoder phenomenon.
The masking regularizer is effectively a causal intervention — removing the acoustic evidence for a label — so it could be reused as a probe to quantify how much each label's prediction relies on its own acoustics versus language context.
The smoothing interpolation of the joint with independent marginals could be extended to stronger factorizations, such as dropping dependence on distant history, connecting the method to n-gram-style ILM estimation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know about arXiv:2506.06096. The first is good: it proposes a new way to estimate a context-dependent internal language model (ILM) for CTC by using the CTC model's own label-level posterior probabilities as teacher targets in knowledge distillation. The cross-domain result on TED-LIUM2 is strong: label-level KD with smoothing gives 13.8% WER vs 15.9% with shallow fusion, a 13% relative gain. The second is a catch: the proof that the student LM converges to the true ILM applies only to the unregularized criterion. The smoothing and masking variants that produce the headline gains change the target even with infinite data.

The derivation in Appendix A is clean. Under an unlimited-data assumption, the global optimum of Eq. (8) equals P_ILM(a|prefix), exactly as defined in Eq. (5). That is a real contribution. The use of CTC prefix probabilities as teacher probabilities is also a good trick, and the authors are honest about the limited-data problem and the risk of overconfident teachers.

The soft spots are real but addressable. As the stress-test note correctly observes, for the smoothed objective the infinite-data optimum becomes alpha * E_{X|A} P + (1-alpha) * E_X P. With alpha=0.5 that is a 50/50 mixture, not the ILM. So the theoretical justification does not cover the regularized methods that actually win. The empirical gains might come from a better decoding prior, not from a more accurate P_ILM. The paper also does not compare against the heuristic ILM from Das et al. [8], which is the closest existing method. Results are single-run, and the ILM checkpoints, smoothing factor, and scales are all chosen on dev WER, so the headline margin may be optimistic. The conclusion says 'sampling' where it means 'smoothing'; that is minor but sloppy.

Who should read this: anyone doing CTC decoding with an external LM. The method is practical and the unregularized theory is worth having. It deserves a serious referee. I would ask the authors for two things before accepting: a direct comparison with Das et al., and a small-scale Monte Carlo estimate of P_ILM to check whether the regularized student is actually closer to the ILM than the baselines are. That would resolve the main concern.

Referee Report

3 major / 4 minor

Summary. The paper studies whether a CTC acoustic model implicitly learns a label-context-dependent internal language model (ILM), despite CTC's conditional-independence assumption, and proposes knowledge-distillation (KD) methods to estimate that ILM. A small LSTM language model is trained as a student on CTC teacher posteriors, using either label-level or sequence-level distillation, with two regularizations (smoothing and masking). The authors prove in Appendix A that the unregularized label-level criterion has as its global optimum the conditional ILM defined in Eq. (5), under the assumption of unlimited data. Experiments on Librispeech (in-domain) and TED-LIUM2 (cross-domain) show that context-dependent ILM estimators outperform context-independent frame-prior and unigram-prior corrections, with the best method (label-level KD with smoothing) yielding a relative WER reduction of over 13% compared to shallow fusion on TED-LIUM2. The paper also reports that ILM perplexity is not correlated with WER and examines the effect of context length and joint use of the frame-level prior.

Significance. If the central claim holds, the paper makes a useful contribution to external language model integration for CTC: it provides a principled way to estimate a context-dependent ILM from CTC outputs, backed by a clean theoretical derivation for the unregularized objective. The cross-domain evaluation is appropriate for testing ILM estimation, and the released code is a practical strength. The finding that ILM perplexity does not correlate with WER is also worth reporting. However, the significance is conditional because the regularized objectives that deliver the best results are not covered by the theoretical derivation, and the empirical evidence does not directly measure whether the distilled model approximates the defined ILM. The paper is not circular; the derivation identifies an optimum rather than fitting the target through the model parameters.

major comments (3)

[Section 3.2.1, Eqs. (9)-(10)] The smoothing regularization changes the training objective so that the infinite-data optimum is no longer the ILM defined in Eq. (5). Under the smoothed distribution Pr_smooth(X, a) = alpha Pr(X, a) + (1 - alpha) Pr(X) Pr(a), the conditional distribution of X given a prefix is alpha Pr(X | a) + (1 - alpha) Pr(X), so the global optimum of the smoothed criterion is q*_alpha(a | prefix) = alpha * E_{X | prefix}[P(a | prefix, X)] + (1 - alpha) * E_X[P(a | prefix, X)]. With the chosen alpha = 0.5, this is a 50/50 mixture of the true conditional ILM and a marginal-acoustic average, not P_ILM. The proof in Appendix A applies only to the unregularized criterion (8) under the assumption tilde-Pr = Pr. The masking regularization has the same issue: it replaces X with a masked version in the teacher, so its infinite-data optimum is also not the conditional ILM. Consequently, the WER improvements in Table 1 (e.g., smoothing 13.8% vs. SF 15.9% on TED-LIUM2 test) do not by themselves validate the claim that q_theta estimates P_ILM; they could arise from a better decoding prior or from the dev-set tuning of scales and hyperparameters. I would like to see a direct fidelity check, such as comparing q_theta to a Monte Carlo estimate of E_{X | prefix}[P(a | prefix, X)] on held-out data, or a theoretical argument that the regularized optimum remains the ILM under some stated condition.
[Section 4.2, Table 1; Section 4.1] The empirical core is weakened by the absence of any comparison with the closest existing CTC-specific ILM estimation method, the acoustic-masking heuristic of Das et al. [8], which is cited but not evaluated. Since the paper claims the proposed methods provide 'better ILM estimation' than previous approaches, a direct comparison with that heuristic is needed. In addition, all scales, the smoothing factor alpha, the masking rate p_mask, and the ILM estimator checkpoints are selected on the dev sets (Section 4.1), and the reported WER differences come from single runs without confidence intervals or multi-seed variability. The conclusion that context-dependent ILMs outperform context-independent priors rests on small gaps in cross-domain results (e.g., unigram 14.9% vs. seq KD 14.3% and label KD smoothing 13.8% on TED-LIUM2 test, Table 1); reporting run-to-run variability or significance would make this claim more robust.
[Section 4.3.2, Table 3; Section 5] The conclusion states that 'label-level KD with sampling' achieved the best performance, and Table 3 is captioned 'with sampling', but the method introduced in Section 3.2.1 is 'label-level KD with smoothing'. This is more than a typo: it is unclear which objective was actually evaluated in the context-length experiment. If masking or sampling was used, the caption and method description must be aligned; if smoothing was used, the conclusion and Table 3 heading need correction.

minor comments (4)

[Abstract and Section 2] The phrase 'repectively' in Section 2 is a typo for 'respectively'.
[Section 3.1] The sentence 'when integrating with an during in decoding' is garbled; it should be rewritten for clarity.
[Eq. (3) vs. Eq. (5)] The ILM is first defined as a marginal distribution over sequences in Eq. (3) and then as a conditional distribution in Eq. (5). The notational relationship between P_ILM(a) and P_ILM(as | prefix) should be made explicit to avoid confusion.
[Section 3.2.1] The sentence 'the latter smoothing term can be regarded as dropping the interdependencies between X and a_S^1' is vague; a concrete statement about the resulting target distribution would help the reader understand the bias introduced by smoothing.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Appendix A optimum equals the independently defined ILM (Eqs. 3/5), and evaluations are held-out; the smoothing/masking regularizers fall outside the proof's scope, a validity gap rather than a circular step.

full rationale

The central derivation is self-contained. The ILM is defined independently in Eq. (3) (P_ILM(a^S_1) := Σ_X Pr(X) P(a^S_1|X)) and in conditional form in Eq. (5); Appendix A proves by Gibbs' inequality that, under the stated infinite-data assumption (Pr̃ = Pr), the global optimum of the unregularized KD criterion (8) equals this ILM. That is a valid reduction of an objective to an independently defined target, not a target defined through fitted parameters, so there is no self-definitional loop. The paper's own limitation statement ('In practice, the available training data is limited, and achieving a global optimum is not guaranteed') and the fact that the smoothing/masking criteria (9)-(10) are shown equivalent to each other (Appendix B) but are not covered by the Appendix A optimum result mean the best WER method (smoothing, 13.8% vs SF 15.9% on Tedlium2 test) may improve decoding through a better prior rather than through a provably more accurate ILM estimate; this is a validity gap, not circularity, because the ILM target is not defined by the fitted model. Empirically, results are held-out: ILM estimators are trained on Librispeech and scored on Librispeech dev/test and TED-LIUM2 dev/test, with scales, α, p_mask and checkpoints chosen on dev sets only, so the reported test improvements are genuine predictions rather than fitted identities. Self-citations ([7], [25], same RWTH group) appear only as supporting discussion ('consistent with discussions on ILM in [7, 25]'); they are not load-bearing for the derivation or the empirical claims. No equation in the paper equals its input by construction, and the central claim survives without any cited 'uniqueness theorem.'.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

Central method rests on two kinds of inputs: data-derived target probabilities from the CTC teacher (via prefix probabilities) and hyperparameters (alpha, p_mask, scales) tuned on dev. No new physical entities.

free parameters (4)

smoothing factor alpha = 0.5
Interpolates empirical joint distribution with product of marginals in Eq. (10); tuned on dev WER (Sec. 4.1).
masking rate p_mask = 0.4
Probability of masking each label position's acoustic input in masking regularization; tuned on dev WER.
ELM scale lambda1 / ILM scale lambda2 / FP scale lambda3 = not reported
All decoding scales optimized on dev sets; concrete values not given.
ILM checkpoint selection = best dev WER checkpoint
They explicitly select ILM estimator checkpoints based on dev-set evaluation, not PPL.

assumptions (4)

domain assumption Empirical training distribution equals the true data distribution in the limit of unlimited data.
Used in Appendix A to prove global optimum of KD criterion equals ILM definition in Eq. (5); the paper concedes it does not hold in practice.
standard math CTC prefix probabilities computed by the forward algorithm give the exact posterior P(a_s | prefix, X).
Invoked in Eq. (6) and Sec. 3.2.1 to define teacher labels.
standard math Gibbs' inequality gives a unique global optimum of the KL training criterion.
Used in Appendix A for the finite-sum exchange and KL minimization.
domain assumption Viterbi (best-path) approximation is sufficient for CTC decoding during ILM integration.
Used throughout decoding; standard but an approximation to full summation in Eq. (2).

invented entities (1)

none
purpose: No new particles, forces, or physical entities are introduced.
The ILM estimator is a standard LSTM trained on CTC posteriors, not a new postulated entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Label-Context-Dependent Internal Language Model Estimation for CTC." pith.science (2026). https://pith.science/paper/BLWQ6IXI

@misc{pith2026250606096,
  author       = {Pith},
  title        = {Pith review of: Label-Context-Dependent Internal Language Model Estimation for CTC},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLWQ6IXI}},
  note         = {Machine review of arXiv:2506.06096}
}

read the original abstract

Although connectionist temporal classification (CTC) has the label context independence assumption, it can still implicitly learn a context-dependent internal language model (ILM) due to modern powerful encoders. In this work, we investigate the implicit context dependency modeled in the ILM of CTC. To this end, we propose novel context-dependent ILM estimation methods for CTC based on knowledge distillation (KD) with theoretical justifications. Furthermore, we introduce two regularization methods for KD. We conduct experiments on Librispeech and TED-LIUM Release 2 datasets for in-domain and cross-domain evaluation, respectively. Experimental results show that context-dependent ILMs outperform the context-independent priors in cross-domain evaluation, indicating that CTC learns a context-dependent ILM. The proposed label-level KD with smoothing method surpasses other ILM estimation approaches, with more than 13% relative improvement in word error rate compared to shallow fusion.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 30 canonical work pages

[8]

Single Headed Attention Based Sequence-to-Sequence Model for State- of-the-Art Results on Switchboard-300,

Z. T ¨uske, G. Saon, K. Audhkhasi, and B. Kingsbury, “Single Headed Attention Based Sequence-to-Sequence Model for State- of-the-Art Results on Switchboard-300,” inINTERSPEECH, 2020

work page 2020
[1]

Introduction & Related Work In automatic speech recognition, sequence-to-sequence (seq2seq) models have drawn more and more attention in re- cent years, due to their simplified training/decoding pipeline and strong performance. The most famous architectures include attention-based encoder-decoder models (AED) [1], recurrent neural network transducers (RNN...

work page
[2]

CTC Given an input sequenceX=x 1, x2, . . . , xT ′ and a label vocabularyV, the probability of the label sequencea S 1 with as ∈ Vis modeled by CTC as follows: P(aS 1 |X) = X yT 1 :B(yT 1 )=aS 1 P(y T 1 |hT 1 ) = X yT 1 :B(yT 1 )=aS 1 TY t=1 P(yt|ht),(1) wherey T 1 is the blank-augmented alignment sequence,Bis the collapse function andh T 1 is the sequenc...

work page Pith review arXiv 2025
[3]

Frame-Level Unigram Prior Estimation Label prior is a simple estimation of the ILM

Internal Language Model Estimation for CTC 3.1. Frame-Level Unigram Prior Estimation Label prior is a simple estimation of the ILM. Manohar et al. proposed to compute the label prior by marginalizing the model posterior over all acoustic inputs [11]. For CTC, this approach provides a context-independent frame-level prior (FP) estimation. PFP(y) = 1PN n=1 ...

work page
[4]

Experiments 4.1. Setup Experiments are done on the 960 hours Librispeech (LBS) dataset [13] for in-domain evaluation and TED-LIUM Release 2 (Tedlium2) dataset [14] for cross-domain evaluation using the RETURNN framework [17] based on Pytorch [18]. We use byte-pair-encoding (BPE) [19] as output labels with a vocab- ulary size of 10k. We use a 12-layer conf...

work page
[5]

To this end, we proposed novel ILM estimation methods for CTC based on knowledge distillation (KD)

Conclusion In this work, we investigated the implicit context dependence of the internal language model (ILM) of connectionist tempo- ral classification (CTC). To this end, we proposed novel ILM estimation methods for CTC based on knowledge distillation (KD). We provided a solid theoretical background for label- and sequence-level KD in the probability sp...

work page
[6]

Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models,

M. Zeineldeen, A. Glushko, W. Michel, A. Zeyer, R. Schl ¨uter, and H. Ney, “Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models,” inINTERSPEECH, 2021, pp. 2856–2860

work page 2021
[7]

On Language Model Integration for RNN Transducer based Speech Recogni- tion,

W. Zhou, Z. Zheng, R. Schl ¨uter, and H. Ney, “On Language Model Integration for RNN Transducer based Speech Recogni- tion,” inICASSP. IEEE, 2022, pp. 8407–8411

work page 2022

Show all 35 references

[9]

Sequence Transduction with Recurrent Neural Net- works,

A. Graves, “Sequence Transduction with Recurrent Neural Net- works,”arXiv preprint arXiv:1211.3711, 2012

2012 arXiv
[10]

Con- nectionist Temporal Classification: Labelling Unsegmented Se- quence Data with Recurrent Neural Networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist Temporal Classification: Labelling Unsegmented Se- quence Data with Recurrent Neural Networks,” inProceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376

2006
[11]

Internal Language Model Train- ing for Domain-adaptive End-to-End Speech Recognition,

Z. Meng, N. Kanda, Y . Gaur, S. Parthasarathy, E. Sun, L. Lu, X. Chen, J. Li, and Y . Gong, “Internal Language Model Train- ing for Domain-adaptive End-to-End Speech Recognition,” in ICASSP. IEEE, 2021, pp. 7338–7342

2021
[12]

Lib- rispeech transducer model with internal language model prior cor- rection,

A. Zeyer, A. Merboldt, W. Michel, R. Schl ¨uter, and H. Ney, “Lib- rispeech transducer model with internal language model prior cor- rection,”arXiv preprint arXiv:2104.03006, 2021

2021 arXiv
[13]

and TEDLIUM Release 2 (Tedlium2) [14] datasets, re- spectively. Experimental results on Tedlium2 demonstrate that ILMs with context dependence work better than the context- independent priors in cross-domain evaluation, indicating a context-dependent ILM of CTC. Furthermore, i...
[14]

Enhancing the ted- lium corpus with selected data for language modeling and more ted talks

A. Rousseau, P. Del ´eglise, Y . Esteveet al., “Enhancing the ted- lium corpus with selected data for language modeling and more ted talks.” inLREC, 2014, pp. 3935–3939

2014
[15]

Mask the bias: Improving domain-adaptive generalization of ctc-based asr with internal language model esti- mation,

N. Das, M. Sunkara, S. Bodapati, J. Cai, D. Kulshreshtha, J. Far- ris, and K. Kirchhoff, “Mask the bias: Improving domain-adaptive generalization of ctc-based asr with internal language model esti- mation,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech ...

2023
[16]

Regarding the existence of the internal language model in ctc-based e2e asr

Z. Zhao and P. Bell, “Regarding the existence of the internal language model in ctc-based e2e asr.” [Online]. Available: https://zhaozeyu1995.github.io/pdf/icassp2025.pdf
[17]

Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,

Y . Miao, M. Gowayyed, and F. Metze, “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,” in 2015 IEEE workshop on automatic speech recognition and under- standing (ASRU). IEEE, 2015, pp. 167–174

2015
[18]

Semi-supervised max- imum mutual information training of deep neural network acous- tic models

V . Manohar, D. Povey, and S. Khudanpur, “Semi-supervised max- imum mutual information training of deep neural network acous- tic models.” inInterspeech, 2015, pp. 2630–2634

2015
[19]

Maximum a posteriori based de- coding for ctc acoustic models

N. Kanda, X. Lu, and H. Kawai, “Maximum a posteriori based de- coding for ctc acoustic models.” inInterspeech, 2016, pp. 1868– 1872

2016
[20]

Lib- rispeech: an ASR Corpus based on public Domain Audio Books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an ASR Corpus based on public Domain Audio Books,” inICASSP. IEEE, 2015, pp. 5206–5210

2015
[21]

Decoupled weight decay regularization,

I. Loshchilov, “Decoupled weight decay regularization,”arXiv preprint arXiv:1711.05101, 2017

2017 arXiv
[22]

Supervised sequence labelling with recurrent neu- ral networks,

A. Graves, “Supervised sequence labelling with recurrent neu- ral networks,” Ph.D. dissertation, Technical University Munich, 2008

2008
[23]

Hy- brid ctc/attention architecture for end-to-end speech recognition,

S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hy- brid ctc/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017

2017
[24]

We apply the time- synchronous Viterbi search implemented in RETURNN

as the ELM for in-domain evaluation and a 4-layer LSTM LM as the ELM for cross-domain evaluation, with perplexity (PPL) 37 and 48 on dev sets, respectively. We apply the time- synchronous Viterbi search implemented in RETURNN. All the scales applied in search are optimized on ...

2025
[25]

RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition,

A. Zeyer, T. Alkhouli, and H. Ney, “RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition,” 2018, pp. 128–133

2018
[26]

Pytorch: An imperative style, high-performance deep learning library,

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Ad- vances in neural information processing systems, vol. 32, 2019

2019
[27]

Neural machine translation of rare words with sub- word units,

R. Sennrich, “Neural machine translation of rare words with sub- word units,”arXiv preprint arXiv:1508.07909, 2015

2015 arXiv
[28]

Conformer: Convolution-augmented transformer for speech recognition,

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech 2020, 2020, pp. 5036–5040

2020
[29]

Advancing rnn transducer technology for speech recognition,

G. Saon, Z. T ¨uske, D. Bolanos, and B. Kingsbury, “Advancing rnn transducer technology for speech recognition,” inICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5654–5658

2021
[30]

Dynamic encoder size based on data-driven layer-wise pruning for speech recognition,

J. Xu, W. Zhou, Z. Yang, E. Beck, and R. Schl ¨uter, “Dynamic encoder size based on data-driven layer-wise pruning for speech recognition,” inInterspeech 2024, 2024, pp. 4563–4567

2024
[31]

Language Modeling with Deep Transformers,

K. Irie, A. Zeyer, R. Schl ¨uter, and H. Ney, “Language Modeling with Deep Transformers,” inINTERSPEECH, 2019

2019
[32]

On the relation be- tween internal language model and sequence discriminative train- ing for neural transducers,

Z. Yang, W. Zhou, R. Schl ¨uter, and H. Ney, “On the relation be- tween internal language model and sequence discriminative train- ing for neural transducers,” inICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, p...

2024
[33]

Testing the correlation of word error rate and perplexity,

D. Klakow and J. Peters, “Testing the correlation of word error rate and perplexity,”Speech Communication, vol. 38, no. 1-2, pp. 19–28, 2002

2002
[34]

Advancing neural language modeling in automatic speech recognition

K. Irie, “Advancing neural language modeling in automatic speech recognition.” Ph.D. dissertation, RWTH Aachen Univer- sity, Germany, 2020. Appendix A. The Global Optimum of Criterion(8) In this section, we derive the global optimum of the training criterion (8), and show that...

2020
[35]

:= S+1X s=1 X a∈V+ P(a|as−1 1 , X) logP(a|as−1 1 , X) qθ(a|as−1 1 ) Therefore, Eq. (9) can be rewritten as: F(θ) = X X,S,aS1 P r(X, aS 1)G(P, q|X, aS 1) = X S,X,aS1 α N NX n=1 δ(aS 1, aSn n,1)δ(X, Xn)G(P, q|X, aS 1) + X S,X,aS1 1−α N NX n=1 δ(aS 1, aSn n,1) 1 N NX n′=1 δ(X, Xn...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [8]

Single Headed Attention Based Sequence-to-Sequence Model for State- of-the-Art Results on Switchboard-300,

Z. T ¨uske, G. Saon, K. Audhkhasi, and B. Kingsbury, “Single Headed Attention Based Sequence-to-Sequence Model for State- of-the-Art Results on Switchboard-300,” inINTERSPEECH, 2020

work page 2020

[2] [1]

Introduction & Related Work In automatic speech recognition, sequence-to-sequence (seq2seq) models have drawn more and more attention in re- cent years, due to their simplified training/decoding pipeline and strong performance. The most famous architectures include attention-based encoder-decoder models (AED) [1], recurrent neural network transducers (RNN...

work page

[3] [2]

CTC Given an input sequenceX=x 1, x2, . . . , xT ′ and a label vocabularyV, the probability of the label sequencea S 1 with as ∈ Vis modeled by CTC as follows: P(aS 1 |X) = X yT 1 :B(yT 1 )=aS 1 P(y T 1 |hT 1 ) = X yT 1 :B(yT 1 )=aS 1 TY t=1 P(yt|ht),(1) wherey T 1 is the blank-augmented alignment sequence,Bis the collapse function andh T 1 is the sequenc...

work page Pith review arXiv 2025

[4] [3]

Frame-Level Unigram Prior Estimation Label prior is a simple estimation of the ILM

Internal Language Model Estimation for CTC 3.1. Frame-Level Unigram Prior Estimation Label prior is a simple estimation of the ILM. Manohar et al. proposed to compute the label prior by marginalizing the model posterior over all acoustic inputs [11]. For CTC, this approach provides a context-independent frame-level prior (FP) estimation. PFP(y) = 1PN n=1 ...

work page

[5] [4]

Experiments 4.1. Setup Experiments are done on the 960 hours Librispeech (LBS) dataset [13] for in-domain evaluation and TED-LIUM Release 2 (Tedlium2) dataset [14] for cross-domain evaluation using the RETURNN framework [17] based on Pytorch [18]. We use byte-pair-encoding (BPE) [19] as output labels with a vocab- ulary size of 10k. We use a 12-layer conf...

work page

[6] [5]

To this end, we proposed novel ILM estimation methods for CTC based on knowledge distillation (KD)

Conclusion In this work, we investigated the implicit context dependence of the internal language model (ILM) of connectionist tempo- ral classification (CTC). To this end, we proposed novel ILM estimation methods for CTC based on knowledge distillation (KD). We provided a solid theoretical background for label- and sequence-level KD in the probability sp...

work page

[7] [6]

Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models,

M. Zeineldeen, A. Glushko, W. Michel, A. Zeyer, R. Schl ¨uter, and H. Ney, “Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models,” inINTERSPEECH, 2021, pp. 2856–2860

work page 2021

[8] [7]

On Language Model Integration for RNN Transducer based Speech Recogni- tion,

W. Zhou, Z. Zheng, R. Schl ¨uter, and H. Ney, “On Language Model Integration for RNN Transducer based Speech Recogni- tion,” inICASSP. IEEE, 2022, pp. 8407–8411

work page 2022