Pith. sign in

REVIEW 3 major objections 5 minor 50 references

On-device Streaming Discrete Speech Units

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Discrete speech units can be extracted by a streaming, half-size model that stays close to the full extractor's ASR accuracy.

desk verdict The paper's headline efficiency claim is contradicted by its own appendix tables; the actual 0.96-TFLOP configuration gives roughly 11.5 WER, not 6.3, so the Pareto story as written doesn't hold. read the letter →

arxiv 2506.01845 v1 pith:QV3AAKRW submitted 2025-06-02 eess.AS cs.LGcs.SD

Kwanghee Choi , Masao Someki , Emma Strubell , Shinji Watanabe This is my paper
classification eess.AScs.LGcs.SD
keywords discretespeechunitsstreamingrecognitionon-deviceASRWavLMself-supervisedmodelsattentionwindowFLOPsreductionML-SUPERB
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that discrete speech units (DSUs), the cluster tokens produced from self-supervised speech models, can be computed on the fly and on a resource-limited device at a fraction of the cost of the original full-context model. The authors build a lightweight speech-to-unit predictor that is trained to reproduce the DSUs of a frozen WavLM-large extractor, then reduce both the attention window and the number of layers. They report cutting floating-point operations by about 50% while keeping character error rate on the ML-SUPERB 1-hour benchmark within about 6.5% relative of the full model's CER. The payoff, if true, is that speech can be tokenized on the client and shipped to a server-side language model at a bitrate three or four orders of magnitude lower than raw audio.

What carries the argument

The load-bearing object is the lightweight DSU predictor, a fine-tuned self-supervised model with a trainable fully-connected layer in place of k-means, trained to output the original model's DSU sequence as ground truth. Two mechanisms make it streaming: a time-restricted self-attention mask that limits how many past and future frames each position can see, and a reduced layer count. A third mechanism, a learnable weighted sum across self-supervised layers, recovers information that individual late layers lose, and fine-tuning the unit-to-text module adapts it to the noisier DSU predictions of small-window models.

What would settle it

Retrain or fully fine-tune the unit-to-text module on the DSU stream produced by the 12-layer, 64-frame-window model and measure WER and CER on the same three test sets; if the gap to the 5.0/9.2/23.0 baseline does not shrink, or instead widens beyond the reported 6.3/14.0/24.4, the near-parity claim fails. A complementary check is to measure real end-to-end latency on the target device to confirm that the TFLOPs reduction translates into streaming feasibility.

Watch

Extended reading notes

Core claim

The paper's central claim is that the expensive parts of discrete speech unit extraction, full-length bidirectional attention and hundreds of millions of parameters, are not necessary. A predictor fine-tuned from WavLM-large, restricted to a symmetric time window of past and future frames and using fewer transformer layers, can mimic the original DSU stream closely enough that a frozen unit-to-text ASR module still transcribes well. With a 12-layer model and a 64-frame symmetric window (about 0.96 TFLOPs per minute of audio), the authors report 6.3 WER on LibriSpeech test-clean, 14.0 WER on test-other, and 24.4 CER on ML-SUPERB 1h, compared with the full-model baselines of 5.0 WER, 9.2 WER, and 23.0 CER. They further show that a learnable weighted sum of layer features and fine-tuning the unit-to-text module each close part of the remaining gap.

Load-bearing premise

The result depends on the assumption that a unit-to-text module trained on the original clean discrete units is a fair judge of the streaming model's predicted units; if those predicted units differ in ways that module cannot absorb, the reported error rates would not reflect performance after adapting the recognizer to the new unit distribution.

Editorial extensions

If this is right

  • DSUs can be produced with a theoretical latency proportional to a small future window rather than full utterance length, enabling streaming ASR and speech-to-LLM pipelines.
  • Halving FLOPs from 1.94 to 0.96 TFLOPs per minute with WER and CER within a few points of baseline suggests on-device DSU extraction is feasible on edge hardware.
  • Symmetric windows outperform past-only or future-only windows at the same size, while full past with limited future matches symmetric performance beyond about 0.5 seconds, guiding latency and accuracy design.
  • Fine-tuning the unit-to-text module on predicted DSUs improves robustness to noisy predictions, particularly for small windows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported numbers likely understate the achievable accuracy: the frozen unit-to-text module was trained on the original DSU distribution, and retraining it on the streaming model's predicted DSUs, as the paper's own fine-tuning experiments suggest, could close much of the remaining WER and CER gap.
  • The Pareto curve implies a deployment choice: for a given device budget, an operator could pick the window size and layer count that meet a target error rate rather than treating the full self-supervised model as a fixed cost.
  • Because DSUs compress one second of audio from roughly 512 kbps raw to about 0.6 kbps, a streaming extractor that runs on-device would let a phone or IoT device send token streams to a server-side LLM, a natural extension the architecture enables.
  • A testable extension is varying the k-means vocabulary size for the lightweight predictor; the current experiments fix the original challenge's clusters, and the tradeoff may shift if a smaller vocabulary suffices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether discrete speech units (DSUs) can be extracted in a streaming, on-device setting by reducing both the attention window and the number of layers of a WavLM-large based speech-to-unit (S2U) module. The lightweight S2U is trained to reproduce the DSUs of the full frozen model, and the resulting DSUs are evaluated with a discrete ASR system (unit-to-text, U2T) on LibriSpeech (WER) and ML-SUPERB 1h (CER). The authors systematically vary symmetric, past-only, and future-only attention windows; the number of layers; learnable weighted feature summation (WF); and U2T fine-tuning. They report a Pareto-optimal tradeoff curve and claim a 50% FLOPs reduction with only a 6.5% relative CER increase.

Significance. If the empirical results were consistent, the paper would provide a useful contribution to on-device and streaming DSU extraction, an area with practical relevance for speech transmission and LLM-based speech interfaces. The appendix tables give exact values for every plotted operating point, and the code is released, which aids reproducibility. The inclusion of a U2T-fine-tuned condition partly addresses the concern that the frozen baseline U2T may be unfair to the new DSU distribution. However, the central efficiency claim in Section 5 is contradicted by the paper's own appendix, which is a load-bearing issue: if the appendix numbers are correct, the 0.96 TFLOPs operating point has much larger WER degradation than the text states. The paper's contribution is therefore significantly weakened and must be reassessed after correcting the inconsistency.

major comments (3)
  1. [Section 5, "Pareto optimal curve"] The sentence "Especially with 0.96 TFLOPs, we achieve 6.3 WER (test-clean), 14.0 WER (test-other), and 24.4 CER (test-1h)" is not supported by the appendix. The only 0.96 TFLOPs configurations in the paper are Layer 12, l=r=64 in Table 4 (Sym.+WF: 11.92/26.21/24.52) and Table 5 (Sym.+WF+U2T FT: 11.50/25.63/24.50). The cited numbers 6.3/14.0/24.4 actually match Table 4, Layer 21, l=r=32, whose TFLOPs is 1.412, not 0.96. This is an internal inconsistency in the headline result. If the appendix is correct, the 0.96 TFLOPs operating point degrades test-clean WER from 5.0 to about 11.5 (a roughly 130% relative increase) and test-other WER from 9.2 to about 25.6 (a roughly 178% relative increase), while CER remains roughly flat at 24.5 versus 23.0. The claim "close to the baseline performance" is therefore false for WER, and the paper's central Pareto-optimal efficiency/accuracy claim must be corrected and re-evaluated.
  2. [Abstract and Section 5] The abstract highlights only the CER result ("relative increase of 6.5% in CER"), which is consistent with the 0.96 TFLOPs row (24.4/24.5 vs. 23.0 CER). However, the Section 5 sentence extends the near-baseline claim to WER as well. After the correction, the paper will report a 50% FLOPs reduction with near-flat CER but roughly 2.3–2.8× worse WER. This is a substantially different message and should be reflected in the abstract and conclusion. The authors should clearly state the WER degradation at the reduced-FLOP operating point rather than implying it is close to baseline on all metrics.
  3. [Section 5, "Pareto optimal curve" and Figure 1] The term "Pareto optimal" is used without a formal definition or a demonstration that the plotted points form the true Pareto frontier. A set of configurations with varying layers and window sizes does not automatically constitute a Pareto-optimal curve unless all dominated points have been removed. Since the paper explicitly lists "Pareto optimal trade-off curve" as a contribution, the authors should either define the dominance criterion used, show that every plotted point is non-dominated, or use a more modest term such as "trade-off curve." This is a presentation issue for the central claim, but the numeric inconsistency above is the more pressing problem.
minor comments (5)
  1. [Section 2.4] The learning rate schedule is described as "step learning rate decay with a rate of 0.9 every 1K steps," but it is unclear whether "steps" refers to optimizer steps or training steps within an epoch; please clarify.
  2. [References] Reference [24] lists the author as "A. Waswani" but the correct name is "A. Vaswani" (Vaswani et al., "Attention is all you need").
  3. [Figure 1 and Tables 1–5] The axis label in Figure 1 reads "TFLOPs per 1m" whereas Figures 2 and 4 use "TFLOPs per 60s"; please use consistent units (e.g., "TFLOPs per 60s" or "TFLOPs per minute").
  4. [Section 1] The paper claims to be the first to explore DSUs in the on-device streaming setting, but related work [38] already applies DSUs to streaming voice conversion; please qualify the novelty claim to avoid overstatement.
  5. [Section 3, Table 1] In Table 1, the "Baseline [6]" row reports test-other WER of 9.1 while "Baseline (Ours)" reports 9.2 with the same configuration except beam size. This small difference is fine, but the caption could explicitly note that the beam size reduction explains it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the DSU predictor is trained against original DSUs and evaluated on downstream ASR, so the derivation is not self-referential. The Section 5 '0.96 TFLOPs' operating point is internally inconsistent with the appendix, but that is a reporting error, not a circular step.

full rationale

The paper is an empirical engineering study with no derived equation that reduces to its own inputs. The pipeline is explicit: (i) the original S2U module fS2U = fkmeans(fS3M(x)) produces DSUs D (Eq. 2); (ii) a lightweight predictor fbarS2U = argmax fFC(fS3M(x)) is trained with the original DSUs D as ground truth (Eq. 7, Section 2.2); (iii) downstream ASR performance is measured by cascading fbarS2U with the frozen U2T module from the challenge baseline (Eq. 5). The training target D and the evaluation metric WER/CER are different objects; reproducing D is the learned task, not the evaluated claim. The frozen U2T is an external benchmark module from the Interspeech 2024 challenge, and its weights are not fitted to the lightweight predictor in any way that would force the reported WER/CER values. Self-citations ([13], [15], [16]) support background claims about phonetic information in S3M layers, but they are not load-bearing: the central results are the data in Tables 1-5, generated in this paper. No uniqueness theorem or prior ansatz is invoked to forbid alternatives. The only notable textual problem is an internal inconsistency, not a circularity: Section 5 states 'Especially with 0.96 TFLOPs, we achieve 6.3 WER (test-clean), 14.0 WER (test-other), and 24.4 CER (test-1h)', but Appendix Table 4's 0.960-TFLOP row (Layer 12, l=r=64, Sym.+WF) gives 11.92/26.21/24.52, and Table 5's same row gives 11.50/25.63/24.50; the cited 6.3/14.0/24.4 numbers instead match the Layer-21, l=r=32 row at 1.412 TFLOPs. This inconsistency affects the reliability of the flagship efficiency claim, but it does not make any derivation circular. The reader-flagged frozen-U2T concern is also not circular: the authors include a U2T-fine-tuned condition (Sym.+WF+U2T FT), whose 0.96-TFLOP row still shows roughly 11.5 WER, so the evaluation is not forced by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new theoretical constructs or entities. The set of assumptions reflect the empirical methodology: using a frozen downstream discrete ASR system to evaluate DSU predictions and using FLOPs as the efficiency metric. The primary free parameter is the post hoc choice of the Pareto-optimal headline configuration, which is a form of test-set selection.

free parameters (1)
  • Headline configuration selection (12 layers, l=r=64, U2T FT) = TFLOPs 0.960
    The Pareto-optimal point highlighted in the abstract and text was selected post hoc from the test-set results, making the choice of layer count and window size a fitted selection rather than a fixed experimental design. The paper does not describe a validation split for this selection.
assumptions (3)
  • domain assumption The frozen unit-to-text (U2T) module trained on original DSUs is a valid evaluation proxy for DSU quality.
    Section 2.3 defines the evaluation as using the challenge's discrete ASR system with fixed U2T. The validity of the reported WER/CER as a proxy for real system performance depends on this module being robust to DSU distribution shift.
  • domain assumption FLOPs computed by calflops accurately reflect computational cost across different attention masks and layer counts.
    Section 2.3 uses calflops to measure TFLOPs per minute. This assumes the tool correctly handles sparse attention masks and that FLOPs are a meaningful proxy for on-device latency and energy. The near-constant TFLOPs for different symmetric window sizes in Table 1 suggest attention FLOPs are a small fraction, which should be verified.
  • domain assumption The original DSUs from the full model are treated as ground truth for training the lightweight predictor.
    Section 2.2: the lightweight DSU predictor is trained to match the original model's DSU labels. This assumes those DSUs are a good target for the downstream task, which is supported by the challenge baseline but not independently re-verified here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-device Streaming Discrete Speech Units." pith.science (2026). https://pith.science/paper/QV3AAKRW

@misc{pith2026250601845,
  author       = {Pith},
  title        = {Pith review of: On-device Streaming Discrete Speech Units},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QV3AAKRW}},
  note         = {Machine review of arXiv:2506.01845}
}
read the original abstract

Discrete speech units (DSUs) are derived from clustering the features of self-supervised speech models (S3Ms). DSUs offer significant advantages for on-device streaming speech applications due to their rich phonetic information, high transmission efficiency, and seamless integration with large language models. However, conventional DSU-based approaches are impractical as they require full-length speech input and computationally expensive S3Ms. In this work, we reduce both the attention window and the model size while preserving the effectiveness of DSUs. Our results demonstrate that we can reduce floating-point operations (FLOPs) by 50% with only a relative increase of 6.5% in character error rate (CER) on the ML-SUPERB 1h dataset. These findings highlight the potential of DSUs for real-time speech processing in resource-constrained environments.

Figures

Figures reproduced from arXiv: 2506.01845 by the authors.

Figure 1
Figure 1. Pareto tradeoff between ASR performance (WER on Librispeech, CER on ML-SUPERB 1h) and computational effi￾ciency (TFLOPs per 1 minute input) for various modifications on WavLM-large. Results show that we can extract DSUs in a lightweight way with limited impact on ASR performance. However, DSUs have yet to be leveraged for streaming on edge devices due to two major challenges. First, conventional S3Ms rely on bidirec… view at source ↗
Figure 3
Figure 3. (1) the original challenge baseline (eq. (2), DSU Base [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Results using features from various layers when ex￾tracting DSUs. Each methods are summarized in [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Results with varying symmetric attention window sizes (Sym.), with applying additional methods, i.e., learnable weights (WF) and fine-tuning U2T module (U2T FT). Baseline uses the full speech input, while Full Past uses full past, but lim￾ited future attention window. …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 45 canonical work pages

  1. [1]

    universal audio feature

    Introduction Self-supervised speech models (S3Ms) have demonstrated re- markable efficacy in various speech-related tasks, including automatic speech recognition (ASR), text-to-speech synthesis, speech translation, speaker identification, and emotion recogni- tion [1–3]. With the advent of S3Ms, many have started to lever- age their features for tokenizin...

    arXiv 2025

  2. [2]

    To evaluate the predicted DSUs, we focus on the discrete ASR system

    Experimental Settings Our work investigates whether DSUs can be accurately pre- dicted using a smaller model. To evaluate the predicted DSUs, we focus on the discrete ASR system. We use the same experi- mental settings across all experiments. 2.1. Discrete ASR System Discrete ASR system contains two main modules: speech-to- unit (S2U) module and unit-to-t...

    work page

  3. [3]

    However, this issue can be mitigated by limiting the future window size of the S2U module

    Reducing the Attention Window Size Why does window size matter? S3Ms leverage Transformers with the full attention window, which requires the full audio to produce discrete units, making the streaming scenario impossi- ble. However, this issue can be mitigated by limiting the future window size of the S2U module. The smaller the required fu- ture frames, ...

    work page

  4. [4]

    Since the number of layers has a linear relationship with computational cost, reduc- ing them benefits resource-constrained on-device applications

    Reducing the Number of Layers Why does the number of layers matter? Existing S3Ms typ- ically consist of a large number of layers. Since the number of layers has a linear relationship with computational cost, reduc- ing them benefits resource-constrained on-device applications. Also, through this experiment, we aim to estimate the amount of compute requir...

    work page

  5. [5]

    Also, by applying such, we produce a Pareto optimal curve that represents the trade-off between the computational over- head and the downstream performance

    Towards the Pareto Optimal We test various methods to improve downstream performance. Also, by applying such, we produce a Pareto optimal curve that represents the trade-off between the computational over- head and the downstream performance. Learnable weights for multi-layer features (WF). As demonstrated in [13,14], different layers encode different typ...

    work page

  6. [6]

    Knowledge distillation (KD) is of- ten used to reduce the size of S3Ms, such as DistilHuBERT

    Related Works Making S3Ms lightweight. Knowledge distillation (KD) is of- ten used to reduce the size of S3Ms, such as DistilHuBERT

    work page

  7. [7]

    How- ever, current methods for generating DSUs rely on full speech input and computationally heavy S3Ms

    Conclusion DSUs offer advantages for on-device streaming due to their transmission efficiency and compatibility with LLMs. How- ever, current methods for generating DSUs rely on full speech input and computationally heavy S3Ms. Therefore, we investi- gated both the attention window size and the number of layers in S3Ms. Our experiments demonstrate the fea...

    work page

  8. [8]

    Acknowledgements This study was supported by the BRIDGE program of the Cab- inet Office, Government of Japan. We also used the Bridges2 system at PSC and Delta system at NCSA through allocations CIS210014 and IRI120008P from the Advanced Cyberinfras- tructure Coordination Ecosystem: Services & Support (AC- CESS) program, through National Science Foundatio...

    work page

Show all 50 references
  1. [9]

    Au- dioPaLM: A large language model that can speak and listen,

    P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen et al., “Au- dioPaLM: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  2. [10]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt et al. , “Self-supervised speech representation learning: A review,”IEEE J-STSP, 2022

    2022

  3. [11]

    wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” in Proc. NeurIPS, 2020

    2020

  4. [12]

    WavLM: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen et al. , “WavLM: Large-scale self- supervised pre-training for full stack speech processing,” IEEE J-STSP, 2022

    2022

  5. [13]

    Exploration of efficient end- to-end asr using discretized input from self-supervised learning,

    X. Chang, B. Yan, Y . Fujita et al., “Exploration of efficient end- to-end asr using discretized input from self-supervised learning,” in Proc. Interspeech, 2023

    2023

  6. [14]

    Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,

    X. Chang, B. Yan, K. Choi et al., “Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,” inProc. ICASSP, 2024

    2024

  7. [15]

    The Interspeech 2024 Challenge on Speech Processing Using Discrete Units,

    X. Chang, J. Shi, J. Tian et al., “The Interspeech 2024 Challenge on Speech Processing Using Discrete Units,” inProc. Interspeech, 2024

    2024

  8. [16]

    AudioLM: a language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent et al., “AudioLM: a language modeling approach to audio generation,” IEEE/ACM TASLP , 2023

    2023

  9. [17]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abili- ties,

    D. Zhang, S. Li, X. Zhang et al., “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abili- ties,” in Proc. EMNLP, 2023

    2023

  10. [18]

    PARP: Prune, adjust and re-prune for self-supervised speech recognition,

    C.-I. J. Lai, Y . Zhang, A. H. Liu et al., “PARP: Prune, adjust and re-prune for self-supervised speech recognition,” Proc. NeurIPS, 2021

    2021

  11. [19]

    Discrete speech unit ex- traction via independent component analysis,

    T. Nakamura, K. Choi, K. Hojo et al., “Discrete speech unit ex- traction via independent component analysis,” in Proc. ICASSP Workshop on Speech and Audio Language Models (SALMA) , 2024

    2024

  12. [20]

    AnyGPT: Unified multimodal LLM with discrete sequence modeling,

    J. Zhan, J. Dai, J. Ye et al., “AnyGPT: Unified multimodal LLM with discrete sequence modeling,” in Proc. ACL, 2024

    2024

  13. [21]

    Anonymizing dysarthric speech: Investigating the effects of voice conversion on pathological information preservation,

    A. Hernandez, P. A. Perez-Toro, T. Arias-Vergara et al. , “Anonymizing dysarthric speech: Investigating the effects of voice conversion on pathological information preservation,” in Intl. Conf. on Text, Speech, and Dialogue. Springer, 2024

    2024

  14. [22]

    Self-supervised speech representations are more phonetic than semantic,

    K. Choi, A. Pasad, T. Nakamura et al., “Self-supervised speech representations are more phonetic than semantic,” in Proc. Inter- speech, 2024

    2024

  15. [23]

    Comparative layer-wise analy- sis of self-supervised speech models,

    A. Pasad, B. Shi, and K. Livescu, “Comparative layer-wise analy- sis of self-supervised speech models,” in Proc. ICASSP, 2023

    2023

  16. [24]

    Leveraging Allophony in Self- Supervised Speech Models for Atypical Pronunciation Assess- ment,

    K. Choi, E. Yeo, K. Chang et al., “Leveraging Allophony in Self- Supervised Speech Models for Atypical Pronunciation Assess- ment,” in Proc. NAACL, 2025

    2025

  17. [25]

    Understanding probe be- haviors through variational bounds of mutual information,

    K. Choi, J.-w. Jung, and S. Watanabe, “Understanding probe be- haviors through variational bounds of mutual information,” in Proc. ICASSP, 2024

    2024

  18. [26]

    HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai et al. , “HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM TASLP, 2021

    2021

  19. [27]

    Streaming automatic speech recog- nition with the transformer model,

    N. Moritz, T. Hori, and J. Le, “Streaming automatic speech recog- nition with the transformer model,” in Proc. ICASSP, 2020

    2020

  20. [28]

    Structured pruning of self- supervised pre-trained models for speech recognition and under- standing,

    Y . Peng, K. Kim, F. Wu et al. , “Structured pruning of self- supervised pre-trained models for speech recognition and under- standing,” in Proc. ICASSP, 2023

    2023

  21. [29]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in Proc. ICASSP, 2015

    2015

  22. [30]

    Techniques like pruning [18, 19] and quantization [32] is also used

    or FitHuBERT [31]. Techniques like pruning [18, 19] and quantization [32] is also used. These approaches, evaluated on general benchmarks [21], primarily aim to create generally us- able S3Ms with lower computational cost, often maintaining the non-streamable Transformer archi...

  23. [31]

    ML-SUPERB: Multilin- gual Speech Universal PERformance Benchmark,

    J. Shi, D. Berrebbi, W. Chen et al. , “ML-SUPERB: Multilin- gual Speech Universal PERformance Benchmark,” in Proc. In- terspeech, 2023

    2023

  24. [32]

    calflops: a FLOPs and params calculate tool for neu- ral networks,

    X. Ye, “calflops: a FLOPs and params calculate tool for neu- ral networks,” https://github.com/MrYxJ/calculate-flops.pytorch, 2023, accessed: 2025-05-20

    2023

  25. [33]

    E-branchformer: Branchformer with enhanced merging for speech recognition,

    K. Kim, F. Wu, Y . Peng et al., “E-branchformer: Branchformer with enhanced merging for speech recognition,” in Proc. SLT, 2022

    2022

  26. [34]

    Attention is all you need,

    A. Waswani, N. Shazeer, N. Parmar et al., “Attention is all you need,” in Proc. NeurIPS, 2017

    2017

  27. [35]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in Proc. ICLR, 2019

    2019

  28. [36]

    A time-restricted self- attention layer for asr,

    D. Povey, H. Hadian, P. Ghahremani et al., “A time-restricted self- attention layer for asr,” in Proc. ICASSP, 2018

    2018

  29. [37]

    SUPERB: Speech Processing Universal PERformance Benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuang et al., “SUPERB: Speech Processing Universal PERformance Benchmark,” in Proc. Inter- speech, 2021

    2021

  30. [38]

    wav2vec-S: Adapting pre-trained speech models for streaming,

    B. Fu, K. Fan, M. Liao et al., “wav2vec-S: Adapting pre-trained speech models for streaming,” in Proc. ACL, 2024

    2024

  31. [39]

    DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden- unit BERT,

    H.-J. Chang, S.-w. Yang, and H.-y. Lee, “DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden- unit BERT,” inProc. ICASSP, 2022

    2022

  32. [40]

    FitHuBERT: Going thinner and deeper for knowledge distillation of speech self-supervised learn- ing,

    Y . Lee, K. Jang, J. Goo et al., “FitHuBERT: Going thinner and deeper for knowledge distillation of speech self-supervised learn- ing,” in Proc. Interspeech, 2022

    2022

  33. [41]

    Adaptive compression of supervised and self-supervised models for green speech recognition,

    M. Oujabour, L. B. Letaifa, J.-F. Dollinger, and J.-L. Rouas, “Adaptive compression of supervised and self-supervised models for green speech recognition,” in Proc. ICASSP, 2025

    2025

  34. [42]

    Soundstream: An end- to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran et al., “Soundstream: An end- to-end neural audio codec,” IEEE/ACM TASLP, 2021

    2021

  35. [43]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” TMLR, 2023

    2023

  36. [44]

    ESPnet-Codec: Comprehensive train- ing and evaluation of neural codecs for audio, music, and speech,

    J. Shi, J. Tian, Y . Wuet al., “ESPnet-Codec: Comprehensive train- ing and evaluation of neural codecs for audio, music, and speech,” in Proc. SLT, 2024

    2024

  37. [45]

    Neural discrete representa- tion learning,

    A. Van Den Oord, O. Vinyals et al., “Neural discrete representa- tion learning,” Proc. NeurIPS, vol. 30, 2017

    2017

  38. [46]

    Distilling hubert with lstms via decoupled knowledge distillation,

    D. de Oliveira and T. Gerkmann, “Distilling hubert with lstms via decoupled knowledge distillation,” in Proc. ICASSP, 2024

    2024

  39. [47]

    Knowledge distillation from self- supervised representation learning model with discrete speech units for any-to-any streaming voice conversion,

    H. Kanagawa and Y . Ijima, “Knowledge distillation from self- supervised representation learning model with discrete speech units for any-to-any streaming voice conversion,” in Proc. Inter- speech, 2024

    2024

  40. [48]

    Speechtokenizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li et al. , “Speechtokenizer: Unified speech tokenizer for speech language models,” in Proc. ICLR, 2024

    2024

  41. [49]

    Moshi: a speech- text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar ´e, M. Orsini et al. , “Moshi: a speech- text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  42. [50]

    We denote various attention window configurations as the number of left, center, and right frames, i.e., [l, c = 1 , r]

    Appendix Table 1: We provide the exact values of Figure 2. We denote various attention window configurations as the number of left, center, and right frames, i.e., [l, c = 1 , r]. As mentioned in Section 2.4, our baseline differs from [6] only by the number of beam size, where...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.