pith. sign in

arxiv: 2604.26514 · v1 · submitted 2026-04-29 · 💻 cs.CL · cs.AI· cs.NE

Text-Utilization for Encoder-dominated Speech Recognition Models

Albert Zeyer , Tim Posielek , Ralf Schl\"uter , Hermann Ney This is my paper

Pith reviewed 2026-05-07 13:33 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.NE
keywords speech recognitiontext-only dataencoder-decoder modelsmodality matchingdynamic downsamplingLibriSpeechautomatic speech recognitionneural networks
0
0 comments X p. Extension

The pith

Text-only integration into the encoder lets larger-encoder smaller-decoder speech models equal or surpass larger-decoder performance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests ways to bring text-only data into the encoder of speech recognition models so that the decoder can stay small. This setup aims at faster recognition while still reaching strong accuracy. On LibriSpeech, the authors find that modality matching combined with dynamic downsampling allows the encoder to produce text-level representations directly. As a result, models with a big encoder and small decoder match or beat the accuracy of balanced architectures that use bigger decoders. Simple random-duration models often work better than elaborate alternatives, which cuts the complexity of training.

Core claim

By using modality matching and dynamic downsampling to align text-only data inside the encoder, encoder-dominated models reach or exceed the word error rates of architectures that rely on larger decoders, while simple random duration modeling proves sufficient for effective training on LibriSpeech.

What carries the argument

Modality matching and dynamic downsampling methods that convert text-only inputs into text-level representations directly within the encoder.

If this is right

  • Encoder-dominated models become practical for real-time speech recognition because the smaller decoder reduces computation at inference time.
  • Text-only data can be used more directly without needing complex decoder-side fusion techniques.
  • Training pipelines simplify when random duration models replace detailed duration predictors and still deliver competitive results.
  • The same integration approach may allow scaling the encoder further while keeping the decoder minimal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the alignment holds, similar text-to-encoder techniques could apply to other encoder-decoder tasks such as machine translation or summarization.
  • Reducing decoder size may lower memory use during deployment without accuracy loss, opening possibilities for edge-device ASR.
  • The finding that simple random durations suffice invites tests on whether duration modeling itself can be removed in many sequence tasks.

Load-bearing premise

The integration steps succeed in aligning the two modalities inside the encoder without adding hidden biases that would hurt performance on new data.

What would settle it

An experiment on a held-out corpus or different language where the larger-encoder smaller-decoder model, after identical text integration, shows clearly higher error rates than the larger-decoder baseline.

Figures

Figures reproduced from arXiv: 2604.26514 by Albert Zeyer, Hermann Ney, Ralf Schl\"uter, Tim Posielek.

Figure 1
Figure 1. Figure 1: Speech recognition using denoising LM [1, 2]. can be very small in relation to the encoder, which is what we also argue here. Note that the TTS data utilized the text data on first part of the encoder and also the DLM. These results show that a larger encoder with a smaller de￾coder is just as good or even better than a smaller encoder with a larger decoder. With a larger encoder, it is crucial that the te… view at source ↗
Figure 2
Figure 2. Figure 2: Model architecture including the training part to uti￾lize the text-only data, which is not used during recognition. the text-only data. While the CTC + AED training on TTS data is simple (despite the TTS), it is very expensive to train. • We want to have models that are efficient in recognition, which means encoder-dominated models with a small de￾coder. Our contributions are as follows: • We point out th… view at source ↗
read the original abstract

This paper investigates efficient methods for utilizing text-only data to improve speech recognition, focusing on encoder-dominated models that facilitate faster recognition. We provide a comprehensive comparison of techniques to integrate text-only data, including modality matching and dynamic downsampling to reach text-level representations within the encoder. Our experiments on the LibriSpeech corpus show that a larger encoder with a smaller decoder can equal or surpass the performance of architectures with larger decoders. We demonstrate that simple configurations, such as random duration models, are often more effective than complex alternatives, significantly simplifying the training pipeline. All code and recipes are made publicly available.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript investigates methods for integrating text-only data into encoder-dominated ASR models to enable more efficient architectures. It compares integration techniques including modality matching and dynamic downsampling to align text representations at the encoder level. Experiments on LibriSpeech are reported to show that larger-encoder + smaller-decoder models can equal or surpass the performance of larger-decoder architectures when using these techniques. The work also finds that simple random duration models often outperform more complex alternatives, and releases all code and recipes publicly.

Significance. If the empirical claims hold under scrutiny, the results could support more efficient ASR inference by favoring smaller decoders while still leveraging abundant text data for performance gains. The public release of code and recipes is a clear strength for reproducibility. However, the absence of quantitative details in the abstract limits immediate assessment of practical impact relative to existing encoder-decoder baselines.

major comments (2)
  1. [Abstract] Abstract: The central claim that 'a larger encoder with a smaller decoder can equal or surpass the performance of architectures with larger decoders' is presented without any numerical results, WER values, baselines, error bars, or ablation statistics. This makes it impossible to evaluate the magnitude or statistical reliability of the reported gains from the provided summary alone.
  2. [Experiments] Experiments section: The headline result attributes performance parity or improvement to the text-integration methods (modality matching, dynamic downsampling). However, without explicit ablations isolating these techniques from the effects of simply enlarging the encoder or altering training regimes, it is unclear whether the architectural claim follows from the data or from unequal capacity/training conditions.
minor comments (2)
  1. [Abstract] The abstract states that 'simple configurations, such as random duration models, are often more effective than complex alternatives' but does not specify the exact metrics or number of alternatives compared.
  2. [Methods] Consider clarifying in the methods whether the dynamic downsampling introduces any duration-model artifacts or information loss that could bias the encoder representations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment below and will revise the paper to improve clarity and provide additional quantitative details and ablations as suggested.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that 'a larger encoder with a smaller decoder can equal or surpass the performance of architectures with larger decoders' is presented without any numerical results, WER values, baselines, error bars, or ablation statistics. This makes it impossible to evaluate the magnitude or statistical reliability of the reported gains from the provided summary alone.

    Authors: We agree that the abstract would benefit from including specific numerical results to allow immediate evaluation of the claims. In the revised version, we will update the abstract to report key WER values on LibriSpeech for the encoder-dominated models using our text-utilization methods, the corresponding larger-decoder baselines, and the observed performance parity or improvements, including any available statistics. revision: yes

  2. Referee: [Experiments] Experiments section: The headline result attributes performance parity or improvement to the text-integration methods (modality matching, dynamic downsampling). However, without explicit ablations isolating these techniques from the effects of simply enlarging the encoder or altering training regimes, it is unclear whether the architectural claim follows from the data or from unequal capacity/training conditions.

    Authors: Our experiments section includes comparisons of multiple model sizes and integration techniques under consistent training conditions on LibriSpeech. To directly address the concern about isolating the contribution of the text methods, we will add explicit ablation tables in the revision that hold encoder size fixed while varying the use of modality matching and downsampling (including random duration baselines), as well as controls for training regime differences. This will demonstrate that the gains are attributable to the proposed techniques rather than capacity or training variations alone. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical comparison without derivations or self-referential reductions

full rationale

The paper is an empirical study comparing text-only integration techniques (modality matching, dynamic downsampling) for encoder-dominated ASR models on LibriSpeech. It reports experimental results showing larger encoders with smaller decoders can match or exceed larger-decoder baselines, and notes that simple random duration models often suffice. No equations, derivations, or predictions are present that reduce to fitted parameters by construction, self-definitions, or load-bearing self-citations. The central claims rest on direct experimental outcomes rather than any of the enumerated circularity patterns. This is a self-contained empirical comparison with independent content from the reported runs and public code.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no mathematical derivations, so no free parameters, axioms, or invented entities can be identified.

pith-pipeline@v0.9.0 · 5397 in / 1108 out tokens · 40759 ms · 2026-05-07T13:33:39.681078+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    Text-Utilization for Encoder-dominated Speech Recognition Models

    Introduction It remains an open question how to best utilize the large amount of text-only data for improving ASR, in addition to the paired audio-text data. A standard approach is to train a separate lan- guage model (LM) on the text-only data, and then use shal- low fusion to utilize the LM during decoding. Using text-to- speech (TTS) to convert the tex...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Related Work A lot of work is on utilizing a pre-trained language model (LM) in the decoder, or even a large LM (LLM). The (L)LM is ex- tended to integrate the encoder, usually by prepending the en- coder output adapted into the LM dimension, known as prefix- LM [9], and then finetuned on paired audio-text data. Text-utilization in encoder has also been e...

  3. [3]

    Model The model architecture is shown in Figure 2. Speech-only encoder .We perform log-mel feature extraction on the input audio, then a convolutional front-end to downsam- ple the features by a factor of 6 in time, and then our speech encoder which is a stack of Conformer blocks. Downsampling to text-level.After the speech encoder, we optionally perform ...

  4. [4]

    Paired audio-text data We use the typical supervised training with cross-entropy loss on the decoder output, and CTC loss on the encoder output

    Training 4.1. Paired audio-text data We use the typical supervised training with cross-entropy loss on the decoder output, and CTC loss on the encoder output. Trainable duration models in the pseudo-speech-encoder are trained with the paired audio-text data, by using the CTC forced alignment between the speech encoder output and the text encoder output. 4...

  5. [5]

    Setup and Baselines 5.1. Data We use the LibriSpeech dataset [23] for our experiments: We use the standard paired audio-text data for training consisting of around 960 hours of speech and the corresponding text tran- scriptions of around 10M words, as well as the text-only data from the LibriSpeech LM corpus consisting of around 800M words, or around 75k ...

  6. [6]

    Experiments All our models are CTC + AED models like described in Sec- tion 3, Figure 2, if not otherwise specified. 6.1. Downsampling & Text-level Encoder Variants Our dynamic downsampling approach is based on the CTC out- put probabilities, and the intention is that the upper text-level encoder operates on more text-like representations. We can en- cour...

  7. [7]

    Our results indicate that a larger encoder paired with a small decoder provides performance that is equal to or better than models with larger decoders (Table 1)

    Conclusions & Future Work We have shown that encoder-dominated models are highly ef- fective for speech recognition when text-only data is utilized within the encoder. Our results indicate that a larger encoder paired with a small decoder provides performance that is equal to or better than models with larger decoders (Table 1). In encoder-dominated model...

  8. [8]

    Clusters4Future

    Acknowledgements This work was partially supported by NeuroSys, which as part of the initiative “Clusters4Future” is funded by the Fed- eral Ministry of Education and Research BMBF (funding IDs 03ZU2106DA and 03ZU2106DD), and by the project RESCALE within the programAI Lighthouse Projects for the Environment, Climate, Nature and Resourcesfunded by the Fed...

  9. [9]

    Generative AI Use Disclosure We use LLMs to improve the formulations and grammar of the paper

  10. [10]

    Denoising LM: Pushing the limits of error correction models for speech recognition,

    Z. Gu, T. Likhomanenko, H. Bai, E. McDermott, R. Collobert, and N. Jaitly, “Denoising LM: Pushing the limits of error correction models for speech recognition,” arXiv:2405.15216, 2024

    work page arXiv 2024

  11. [11]

    Reproducing and dissecting denoising language models for speech recognition,

    D. Koch, A. Zeyer, N. Rossenbach, R. Schl ¨uter, and H. Ney, “Reproducing and dissecting denoising language models for speech recognition,” 2025. [Online]. Available: https://arxiv.org/abs/2512.13576

    work page arXiv 2025

  12. [12]

    Generating synthetic audio data for attention-based speech recognition sys- tems,

    N. Rossenbach, A. Zeyer, R. Schl ¨uter, and H. Ney, “Generating synthetic audio data for attention-based speech recognition sys- tems,” inIEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, May 2020, pp. 7069– 7073

    2020

  13. [13]

    MAESTRO: Matched speech text rep- resentations through modality matching,

    Z. Chen, Y . Zhang, A. Rosenberg, B. Ramabhadran, P. J. Moreno, A. Bapna, and H. Zen, “MAESTRO: Matched speech text rep- resentations through modality matching,” inInterspeech 2022, 2022, pp. 4093–4097

    2022

  14. [14]

    CJST: CTC compressor based joint speech and text training for decoder- only ASR,

    W. Zhou, J. Jia, L. Sari, J. Mahadeokar, and O. Kalinli, “CJST: CTC compressor based joint speech and text training for decoder- only ASR,” inICASSP 2025 - 2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  15. [15]

    JOIST: A joint speech and text streaming model for ASR,

    T. N. Sainath, R. Prabhavalkar, A. Bapna, Y . Zhang, Z. Huo, Z. Chen, B. Li, W. Wang, and T. Strohman, “JOIST: A joint speech and text streaming model for ASR,” in2022 IEEE spoken language technology workshop (SLT). IEEE, 2023, pp. 52–59

    2023

  16. [16]

    SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training,

    Z. Zhang, L. Zhou, J. Ao, S. Liu, L. Dai, J. Li, and F. Wei, “SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training,” inProceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, 2022, pp. 1663–1676

    2022

  17. [17]

    SpeechT5: Unified-modal encoder-decoder pre- training for spoken language processing,

    J. Ao, R. Wang, L. Zhou, C. Wang, S. Ren, Y . Wu, S. Liu, T. Ko, Q. Li, Y . Zhang, Z. Wei, Y . Qian, J. Li, and F. Wei, “SpeechT5: Unified-modal encoder-decoder pre- training for spoken language processing,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), S. Muresan, P. Nakov, and A. Vill...

    2022

  18. [18]

    What language model architecture and pretraining objective works best for zero-shot generalization?

    T. Wang, A. Roberts, D. Hesslow, T. L. Scao, H. W. Chung, I. Beltagy, J. Launay, and C. Raffel, “What language model architecture and pretraining objective works best for zero-shot generalization?” inProceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Sz...

    2022

  19. [19]

    SPLAT: Speech-language joint pre-training for spoken language understanding,

    Y .-A. Chung, C. Zhu, and M. Zeng, “SPLAT: Speech-language joint pre-training for spoken language understanding,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell...

    2021

  20. [20]

    Understanding shared speech-text rep- resentations,

    G. Wang, K. Kastner, A. Bapna, Z. Chen, A. Rosenberg, B. Ram- abhadran, and Y . Zhang, “Understanding shared speech-text rep- resentations,” inICASSP 2023-2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  21. [21]

    ASTRA: Aligning speech and text representa- tions for ASR without sampling,

    N. Gaur, R. Agrawal, G. Wang, P. Haghani, A. Rosenberg, and B. Ramabhadran, “ASTRA: Aligning speech and text representa- tions for ASR without sampling,” inInterspeech 2024, 2024, pp. 3904–3908

    2024

  22. [22]

    SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,

    A. Bapna, Y . an Chung, N. Wu, A. Gulati, Y . Jia, J. H. Clark, M. Johnson, J. Riesa, A. Conneau, and Y . Zhang, “SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,” 2021. [Online]. Available: https://arxiv.org/abs/2110.10329

    work page arXiv 2021

  23. [23]

    Bapna, C

    A. Bapna, C. Cherry, Y . Zhang, Y . Jia, M. Johnson, Y . Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mSLAM: Massively multilingual joint pre-training for speech and text,” 2022. [Online]. Available: https://arxiv.org/abs/2202.01374

    work page arXiv 2022

  24. [24]

    Unified speech-text pre-training for speech translation and recognition,

    Y . Tang, H. Gong, N. Dong, C. Wang, W.-N. Hsu, J. Gu, A. Baevski, X. Li, A. Mohamed, M. Auli, and J. Pino, “Unified speech-text pre-training for speech translation and recognition,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, I...

    2022

  25. [25]

    Towards reducing the need for speech training data to build spoken lan- guage understanding systems,

    S. Thomas, H.-K. J. Kuo, B. Kingsbury, and G. Saon, “Towards reducing the need for speech training data to build spoken lan- guage understanding systems,” inICASSP 2022 - 2022 IEEE In- ternational Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 2022, pp. 7932–7936

    2022

  26. [26]

    A non- autoregressive model for joint STT and TTS,

    V . Sunder, B. Kingsbury, G. Saon, S. Thomas, S. Shecht- man, H. Aronowitz, E. Fosler-Lussier, and L. Lastras, “A non- autoregressive model for joint STT and TTS,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  27. [27]

    Can unpaired textual data replace synthetic speech in ASR model adaptation?

    P. D’Alterio, C. Hensel, and B. A. S. Hasan, “Can unpaired textual data replace synthetic speech in ASR model adaptation?” in2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU), 2023, pp. 1–8

    2023

  28. [28]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to- end speech recognition,

    Z. Gao, S. Zhang, I. McLoughlin, and Z. Yan, “Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to- end speech recognition,” inInterspeech 2022, 2022, pp. 2063– 2067

    2022

  29. [29]

    Paraformer-v2: An improved non-autoregressive transformer for noise-robust speech recognition,

    K. An, Z. Li, Z. Gao, and S. Zhang, “Paraformer-v2: An improved non-autoregressive transformer for noise-robust speech recognition,” 2024. [Online]. Available: https://arxiv.org/abs/ 2409.17746

    work page arXiv 2024

  30. [30]

    E-Paraformer: A faster and better parallel transformer for non-autoregressive end-to-end Mandarin speech recognition,

    K. Zou, F. Tan, Z. Zhuang, C. Miao, T. Wei, S. Zhai, Z. Li, W. Hu, S. Wang, and J. Xiao, “E-Paraformer: A faster and better parallel transformer for non-autoregressive end-to-end Mandarin speech recognition,” inInterspeech 2024, 2024, pp. 267–271

    2024

  31. [31]

    CTC-based compression for direct speech translation,

    M. Gaido, M. Cettolo, M. Negri, and M. Turchi, “CTC-based compression for direct speech translation,” inProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V olume, P. Merlo, J. Tiedemann, and R. Tsarfaty, Eds. Online: Association for Computational Linguistics, Apr. 2021, pp. 690–696. [Online]....

    2021

  32. [32]

    Lib- rispeech: an ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an ASR corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  33. [33]

    Relaxing the conditional indepen- dence assumption of CTC-based ASR by conditioning on inter- mediate predictions,

    J. Nozaki and T. Komatsu, “Relaxing the conditional indepen- dence assumption of CTC-based ASR by conditioning on inter- mediate predictions,” inInterspeech 2021, 2021, pp. 3735–3739

    2021