Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

DuRep: Dual-Mode Speech Representation Learning via ASR-Aware Distillation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DuRep trains a single speech encoder to serve both streaming and non-streaming ASR, with reported double-digit WER gains over single-mode baselines.

desk verdict A genuinely novel dual-mode distillation recipe, but the headline gains are measured only on an internal SUPERB variant, so the SOTA claims are not yet verifiable. read the letter →

arxiv 2505.19774 v1 pith:K7AQJVC4 submitted 2025-05-26 eess.AS

Prabash Reddy Male , Swayambhu Nath Ray , Harish Arsikere , Akshat Jaiswal , Prakhar Swarup , Prantik Sen , Debmalya Chakrabarty , K V Vijay Girish
show 4 more authors
Nikhil Bhave Frederick Weber Sambuddha Bhattacharya Sri Garimella
This is my paper
classification eess.AS
keywords dual-modespeechencoderstreamingASRnon-streamingknowledgedistillationvariableattentionmaskingself-supervisedrepresentationlearningtransducerfine-tuningmultilingual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to remove the usual either-or choice in speech encoding: either a full-context encoder for accurate offline recognition or a causal encoder for low-latency streaming. It introduces DuRep, a four-stage training recipe that produces one encoder able to operate in both modes at inference time by varying only the attention mask. The recipe distills a full-context, ASR-finetuned teacher into a student encoder using variable look-back and look-ahead contexts, then fine-tunes the student with a transducer objective. If the results hold, a single deployed model can replace two mode-specific encoders, cutting training and serving cost while matching or improving accuracy in both regimes.

What carries the argument

The mechanism that carries the argument is variable attention masking combined with pseudo-label distillation. During the distillation stage, the output of an intermediate block of the ASR-finetuned full-context teacher is quantized with k-means into cluster IDs, and the dual-mode student is trained to predict those IDs from the same speech under attention masks whose look-back and look-ahead are sampled from two uniform distributions. Future context uses chunked attention and past context uses a fixed window, preventing look-ahead from accumulating across layers. This exposes one encoder to many latency regimes, so a single set of weights can be deployed in multiple modes; the later transducer fine-tuning with the same variable masking sharpens the representations for ASR.

What would settle it

Run the same encoder and baselines through the public benchmark's original character-tokenizer, beam-search ASR evaluation; if the streaming and full-context WER advantages over the baselines shrink substantially or reverse, the dual-mode claim fails under the standard protocol.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that mode-specific specialization is unnecessary: a single convolution-augmented transformer encoder can be made dual-mode by distilling from a supervised full-context teacher into a student that is trained to predict quantized pseudo-labels under randomly sampled past and future context lengths. At inference, the same weights run in full-context, streaming, or partial-look-ahead mode by choosing the mask. The paper reports that the 200M-parameter version improves over single-mode baselines by 9.78% and 11.6% average WER on monolingual and multilingual full-context tests, and by 15.14% and 12% in streaming mode; at 2B parameters, it reports new performance benchmarks among the compared open-source encoders on both ASR and non-ASR tasks.

Load-bearing premise

The load-bearing assumption is that the in-house evaluation setup, which uses subword tokenizers and greedy decoding instead of the benchmark's character tokenizer and beam search, preserves the relative ranking of encoders; if it does not, the reported WER gains may not transfer to the public benchmark or to real deployment conditions.

Editorial extensions

If this is right

  • A single DuRep encoder can serve streaming, full-context, and intermediate latency settings without extra parameters or mode-specific weights.
  • Distilling from a supervised full-context teacher is load-bearing: the same distillation from an unsupervised teacher yields worse WER, so the ASR-aware step is essential to the result.
  • Restricting context sampling to only causal and full-context modes improves those two modes but degrades partial-look-ahead scenarios, so varied sampling is needed for balanced deployment performance.
  • Scaling from 200M to 2B parameters preserves dual-mode behavior and improves both ASR and non-ASR tasks, including emotion recognition and language identification, without task-specific fine-tuning of the encoder.
  • In full-context mode, the 2B encoder outperforms the compared open-source encoders, and in streaming mode it still beats most of them even when those baselines run in full-context mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate test would be to evaluate DuRep with the benchmark's original character-tokenizer and beam-search ASR protocol rather than the internal subword-tokenizer, greedy-decoding setup; if the relative gains persist, the result is protocol-independent.
  • The distillation recipe appears teacher-agnostic, so a cheap check is to swap the full-context teacher for a different ASR-finetuned encoder and repeat the distillation stage; if dual-mode gains persist, the method transfers beyond this specific architecture.
  • The layer analysis suggests that downstream task heads should weight early layers for acoustic tasks and deep layers for semantic tasks differently for DuRep than for the teacher, a prediction that layer-weighted probing could verify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes DuRep, a four-stage training procedure for a Conformer-based speech encoder intended to operate in both streaming and full-context modes with a single parameter set. In S1 a full-context encoder is pretrained with BestRQ; in S2 it is fine-tuned with a transducer ASR objective; in S3 a dual-mode student is trained by cross-entropy distillation against k-means pseudo-labels extracted from an intermediate layer of the supervised teacher, while variable attention masking samples look-back and look-ahead contexts; in S4 the student is fine-tuned with the transducer objective under the same variable masking. A 200M model is evaluated on an internal SUPERB-style protocol on monolingual and multilingual ASR and compared with in-house full-context and streaming baselines; a 2B model is compared with Whisper-large-v3, Wav2Vec2, WavLM, and mHuBERT on ASR, language identification, and emotion recognition tasks. The paper also reports a layer-wise analysis of acoustic and semantic information across encoder blocks.

Significance. The core idea of using a supervised full-context teacher to distill a dual-mode student is sensible, and the controlled ablation in Table 2 (BestRQ-only training vs. distillation from an unsupervised teacher E1 vs. distillation from a supervised teacher E2) directly supports the claim that the ASR-aware teacher is the key ingredient. The scale of the experiments and the layer-wise SUPERB analysis are useful additions to the literature. However, every headline quantitative claim is measured under an internal modified SUPERB protocol rather than the public benchmark, so the reported improvements and the state-of-the-art statements are not yet anchored to externally comparable numbers. This is a correctness-risk concern about external validity, not an internal inconsistency, and it can be addressed with additional validation experiments or a rank-preservation analysis.

major comments (3)
  1. [3.4 (Tables 1 and 4)] The evaluation protocol is an internal re-implementation of SUPERB that differs from the public benchmark in three ways: wordpiece tokenizers (4k and 15k vocabularies) instead of characters, greedy instead of beam-search CTC decoding, and no external LM. The authors state that a constant protocol supports relative comparison, but this is an assumption about rank preservation, not a consequence of keeping the protocol fixed. Wordpiece targets and greedy decoding can interact with properties of individual encoders (for example, how well their representations align with subword units or how peaked their CTC posteriors are), so the ordering under this protocol need not match the public SUPERB ordering. Since the abstract's 12%/11.6% improvements and the Section 5.1 'outperforms all encoders' statement are computed exclusively under this protocol, the paper should provide evidence that ranking is preserved, for example by running the standard public SUPERB evaluation on the main models or by reporting rank correlations between the two protocols over a set of public encoders.
  2. [3.1 and 3.4 (Tables 1 and 4)] The training corpus includes CommonVoice, FLEURS, VoxPopuli, and 'others,' while the evaluation uses CommonVoice test partitions, FLEURS, and MSP-Podcast. The manuscript does not state which splits are used for training or whether evaluation utterances and speakers are excluded. If the training partition overlaps with the evaluation partitions, the MulASR, LID, and ER numbers in Tables 1 and 4 would be inflated independently of representation quality. Please state the exact training and evaluation split for each public dataset and describe the deduplication procedure, including speaker-level exclusion where applicable.
  3. [5.1 (Table 4)] The external comparisons do not control for pretraining data size or supervised data. DuRep-2B is trained on 8.3M hours, including proprietary data and machine-generated transcripts for the supervised stages, whereas the open-source baselines use substantially smaller corpora; the reported margin over Whisper-large-v3 and the SSL baselines may therefore reflect data scale rather than the proposed dual-mode distillation. The 'sets new performance benchmarks' claim should be qualified as a systems-level result, or supplemented with a comparison encoder trained on a comparable corpus, so that the effect of the proposed method is isolated.
minor comments (6)
  1. [4.1 (Table 1)] The stated streaming-mode average improvement of 15.14% on monolingual ASR does not follow from the two per-test-set reductions in Table 1, which average to roughly 13.6%; please state the averaging formula and reconcile this with the abstract's 12% and 11.6% figures.
  2. [2.2] The sentence 'Our ablation studies showed that not applying masking to the input speech frames during distillation leads to a more effective student encoder' is ambiguous: it could mean input feature masking, SpecAug-style augmentation, or attention masking. Please clarify and point to the ablation that supports this statement.
  3. [5.2] The distillation layer (block 14 for the 200M model, block 15 for the 2B model) is selected 'based on semantic richness' using the same SUPERB-style evaluation as the main results; please state whether this selection was made on a held-out development set and not on the reported test partitions.
  4. [Tables 1-4] No error bars, confidence intervals, or repeated-seed runs are reported; because some conclusions depend on small WER differences (for example, 4.61 vs. 5.07 in Table 1), a statement about run-to-run variance would help the reader assess significance.
  5. [Figure 2] The attention-mask diagram would benefit from a caption that defines the matrix axes and explains the 'Target Tokens' label, which is currently unclear.
  6. [Abstract and Section 6] The abstract and conclusion use 'state-of-the-art' and 'sets new performance benchmarks' language that is stronger than what the internal, non-public evaluation protocol can support; please calibrate the wording to the comparisons actually made.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's headline results are empirical comparisons on an external benchmark protocol, not derivations that reduce to their inputs.

full rationale

The central claims of DuRep are empirical measurements rather than derived predictions. DuRep-200M is compared against independently trained single-mode baselines in Table 1, and DuRep-2B is compared against external open-source encoders (Whisper-large-v3, Wav2Vec2, mHUBERT, WavLM) in Table 4 under the same evaluation protocol. No fitted parameter is later renamed as a prediction, no self-citation is load-bearing, and no uniqueness theorem is imported from the authors' prior work. The only protocol caveat is in Section 3.4, where the authors state that their internal SUPERB implementation uses a 4000/15000 wordpiece tokenizer, greedy decoding, and no external LM, and they justify relative comparisons by noting that the setup is kept constant across runs. That is an assumption about rank preservation across protocols, which is an external-validity and benchmarking concern, not a circularity: the reported WERs are measurements under a fixed protocol, not consequences of the encoders' definitions. The choice of distillation layer in Section 5.2 is based on development-set SUPERB analysis and is a standard hyperparameter selection step, not a fitted input called a prediction. Consequently, there is no step in which an output is equivalent to an input by construction, and the paper is not circular. Score 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The empirical claims rest on a proprietary 8.3M-hour dataset, an internal SUPERB variant, several task-specific hyperparameters (cluster count, distillation block, context ranges) tuned without public ablations, and proprietary transcription models. These are the unshared premises a reader must accept to trust the reported numbers.

free parameters (5)
  • Number of k-means clusters for distillation pseudo-labels = 2500 (DuRep-200M), 4500 (DuRep-2B)
    Chosen by design in Section 2.2; no ablation on cluster count is reported, yet it affects the granularity of the teacher signal.
  • Distillation source block (teacher intermediate layer) = 14th block (DuRep-200M), 15th block (DuRep-2B)
    Selected based on SUPERB layer analysis in Section 5.2, using the same evaluation family as the final results, which introduces selection bias.
  • Variable attention masking distributions (L_past, L_future) = L_past=[inf,5.4,4.6,3.6], L_future=[0,1,1.8,inf] (T1)
    Set from 'internal encoder usage scenarios' (Section 3.2, 4.3); the paper's own ablation shows the choice significantly affects streaming vs partial look-ahead performance.
  • BestRQ pretraining mask durations for baselines = 320ms (streaming baseline), 400ms (non-streaming baseline)
    Used only for the single-mode baselines in Section 3.3, but these choices affect the strength of the baselines being compared against.
  • Streaming evaluation look-back/look-ahead = LB=5.4s, LA=0s
    Defines what 'streaming' means in the evaluation (Section 3.4); a 5.4s look-back is large and may not represent low-latency streaming use cases.
assumptions (5)
  • domain assumption The internal SUPERB implementation faithfully preserves relative encoder quality despite tokenizer, decoding, and LM differences.
    Invoked in Section 3.4 as justification for comparing models within the modified setup.
  • domain assumption Machine-generated transcriptions from proprietary models are accurate enough for transducer fine-tuning in S2 and S4.
    Stated in Section 3.1, used for the untranscribed portion of the 8.3M-hour dataset.
  • domain assumption BestRQ pretraining on 8.3M hours of unlabeled audio provides a sound foundation for the later stages.
    The paper relies on BestRQ [9] and prior industry usage [10] rather than deriving or validating this stage independently.
  • domain assumption K-means cluster labels from a chosen intermediate Conformer block are adequate surrogate targets for distillation.
    Introduced in Section 2.2; no ablation on cluster count or quality is reported.
  • standard math Standard speech-processing mathematics: cross-entropy loss for distillation and the transducer objective behave as expected.
    Background methodology adopted without proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DuRep: Dual-Mode Speech Representation Learning via ASR-Aware Distillation." pith.science (2026). https://pith.science/paper/K7AQJVC4

@misc{pith2026250519774,
  author       = {Pith},
  title        = {Pith review of: DuRep: Dual-Mode Speech Representation Learning via ASR-Aware Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K7AQJVC4}},
  note         = {Machine review of arXiv:2505.19774}
}
read the original abstract

Recent advancements in speech encoders have drawn attention due to their integration with Large Language Models for various speech tasks. While most research has focused on either causal or full-context speech encoders, there's limited exploration to effectively handle both streaming and non-streaming applications, while achieving state-of-the-art performance. We introduce DuRep, a Dual-mode Speech Representation learning setup, which enables a single speech encoder to function efficiently in both offline and online modes without additional parameters or mode-specific adjustments, across downstream tasks. DuRep-200M, our 200M parameter dual-mode encoder, achieves 12% and 11.6% improvements in streaming and non-streaming modes, over baseline encoders on Multilingual ASR. Scaling this approach to 2B parameters, DuRep-2B sets new performance benchmarks across ASR and non-ASR tasks. Our analysis reveals interesting trade-offs between acoustic and semantic information across encoder layers.

Figures

Figures reproduced from arXiv: 2505.19774 by the authors.

Figure 1
Figure 1. Dual-Mode encoder training framework 2.2. Knowledge Distillation Following the pseudo-target prediction approach outlined in [11], during S3 (shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. presents the development evaluation results for semantic and acoustic tasks across the layers of DuRep-2B. Our findings show that embeddings from the deeper Conformer blocks yield the best performance for semantically-oriented tasks like ASR and Speaking Rate Prediction. In contrast, for tasks that capture acoustic properties like Intensity and Pitch Contour Prediction, embeddings from the initial blocks of the enco… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Group Relative Policy Optimization for Speech Recognition

    eess.AS 2025-09 conditional novelty 5.0 of 10

    Applying GRPO with rule-based rewards to LLM-based ASR improves WER by up to 18.4% relative and reduces hallucination errors on unseen acoustic conditions.

Reference graph

Works this paper leans on

39 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Full-context encoders [1, 2], leveraging entire speech utterances, offer superior accuracy but higher latency, making them ideal for offline use

    Introduction The design of speech encoders, which extract meaningful rep- resentations from raw speech data, often involves a trade-off between performance and latency. Full-context encoders [1, 2], leveraging entire speech utterances, offer superior accuracy but higher latency, making them ideal for offline use. Conversely, streaming encoders process spe...

    work page

  2. [2]

    Methodology 2.1. Training stages In this work, we propose a novel four-stage approach to develop a dual-mode encoder as demonstrated in Figure 1: •Full-context Pretraining (S1):We begin by training a non- streaming (NS) encoder using BestRQ framework [9], which is a widely accepted pretraining method in the industry for speech encoders on unlabelled data ...

    arXiv 2025

  3. [3]

    Experimental Details We conducted all initial experiments and ablation studies on DuRep-200M, subsequently scaling our best-performing train- ing strategy to DuRep-2B. 3.1. Data We utilized a comprehensive speech dataset totalling 8.3M hours, which included both hand-transcribed speech data and a significant portion from untranscribed sources. Data consis...

    work page

  4. [4]

    Results Table 1 presents a comparative analysis of the dual-mode en- coder and single-mode baseline encoders across full-context and streaming scenarios, in ASR tasks

    DuRep-200M 4.1. Results Table 1 presents a comparative analysis of the dual-mode en- coder and single-mode baseline encoders across full-context and streaming scenarios, in ASR tasks. In the full-context set- ting, Dual-mode encoder outperformed the baseline encoder, achieving an average improvement of 9.78% and 11.6% across monolingual and multilingual t...

    work page

  5. [5]

    Due to these variations, absolute WERs reported after our SUPERB bench- marking are higher than those reported in the original works

    no external LM is used in any evaluation. Due to these variations, absolute WERs reported after our SUPERB bench- marking are higher than those reported in the original works. But since the setup is kept constant across all runs, we can still use the WER numbers for relative comparisons across en- coder models. We evaluate the encoders on Monolingual ASR ...

    work page

  6. [6]

    utilised variable attention mask to balance efficient model- ing and reduced future frame dependency. Despite the focus on transducer [7] or CTC-based objectives, previous works have not explored distillation strategies for training dual-mode en- coders, leaving a gap that our research aims to address. In this paper, we present a zero-additional-parameter...

    work page

  7. [7]

    Results Table 4 compares the performance of the DuRep-2B encoder in two inference modes with open-source encoders on ASR and non-ASR tasks, evaluated using SUPERB framework (3.4)

    DuRep-2B 5.1. Results Table 4 compares the performance of the DuRep-2B encoder in two inference modes with open-source encoders on ASR and non-ASR tasks, evaluated using SUPERB framework (3.4). In full-context mode, the DuRep-2B encoder outperforms all en- coders [24–26] across all tasks. It exceeds the performance of the whisper-large-v3 encoder by 19.26...

    work page

  8. [8]

    In ASR-SUPERB evaluations, on average, the DuRep-200M encoder outperforms baselines by 13.06% in streaming and 10.42% in non-streaming mode

    Conclusions We introduce DuRep, a dual-mode speech representation learn- ing approach using variable attention based distillation, to train a general-purpose speech encoder that is optimized for multiple look-ahead inference scenarios. In ASR-SUPERB evaluations, on average, the DuRep-200M encoder outperforms baselines by 13.06% in streaming and 10.42% in ...

    work page

Show all 39 references
  1. [9]

    Acknowledgements We would additionally like to acknowledge Jahn Heymann, Ankish Bansal, Harish Mallidi, Phani Nidadavolu, Milind Rao, Bharat Padi, Venkata Kishore Nandury, Tuan Dinh, and An- dreas Schwarz for their valuable contribution to data prepara- tion efforts

  2. [10]

    Google usm: Scaling automatic speech recognition beyond 100 languages,

    Y . Zhang, W. Han, J. Qin, Y . Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V . Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. ...

  3. [11]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe, “Self-supervised speech representation learning: A review,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, ...

    2022

  4. [12]

    Robust speech recognition via large- scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

    2022 arXiv

  5. [13]

    Dual-mode asr: Unify and improve streaming asr with full-context modeling,

    J. Yu, W. Han, A. Gulati, C.-C. Chiu, B. Li, T. N. Sainath, Y . Wu, and R. Pang, “Dual-mode asr: Unify and improve streaming asr with full-context modeling,” 2021. [Online]. Available: https://arxiv.org/abs/2010.06030

    2021 arXiv

  6. [14]

    Conformer with dual-mode chunked attention for joint online and offline asr,

    F. Weninger, M. Gaudesi, M. A. Haidar, N. Ferri, J. Andr ´es- Ferrer, and P. Zhan, “Conformer with dual-mode chunked attention for joint online and offline asr,” 2022. [Online]. Available: https://arxiv.org/abs/2206.11157

    2022 arXiv

  7. [15]

    Multi- mode transformer transducer with stochastic future context,

    K. Kim, F. Wu, P. Sridhar, K. J. Han, and S. Watanabe, “Multi- mode transformer transducer with stochastic future context,”

  8. [16]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneauet al., “Fleurs: Few-shot learning evaluation of universal representations of speech,”2022 IEEE Spoken Language Technology Workshop (SLT), pp. 798–805, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID: 249062909

    2022

  9. [17]

    Variable attention masking for configurable transformer transducer speech recognition,

    P. Swietojanski, S. Braun, D. Can, T. F. da Silva, A. Ghoshal, T. Hori, R. Hsiao, H. Mason, E. McDermott, H. Silovsky, R. Travadi, and X. Zhuang, “Variable attention masking for configurable transformer transducer speech recognition,” 2023. [Online]. Available: https://arxiv.o...

    2023 arXiv

  10. [18]

    Sequence transduction with recurrent neural networks,

    A. Graves, “Sequence transduction with recurrent neural networks,” 2012. [Online]. Available: https://arxiv.org/abs/1211. 3711

    2012

  11. [19]

    Superb: Speech processing universal performance benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, “Superb: Speech processing universal per...

    2021 arXiv

  12. [20]

    Self- supervised learning with random-projection quantizer for speech recognition,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self- supervised learning with random-projection quantizer for speech recognition,” 2022. [Online]. Available: https://arxiv.org/abs/ 2202.01855

    2022 arXiv

  13. [21]

    Universal-1: Robust and accurate multi- lingual speech-to-text,

    AssemblyAI, “Universal-1: Robust and accurate multi- lingual speech-to-text,” 2024. [Online]. Available: https: //www.assemblyai.com/research/universal-1

    2024

  14. [22]

    Ml-superb: Multilingual speech universal performance benchmark,

    J. Shi, D. Berrebbi, W. Chen, H.-L. Chung, E.-P. Hu, W. P. Huang, X. Chang, S.-W. Li, A. Mohamed, H. yi Lee, and S. Watanabe, “Ml-superb: Multilingual speech universal performance benchmark,” 2023. [Online]. Available: https: //arxiv.org/abs/2305.10615

    2023 arXiv

  15. [23]

    Lib- rispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  16. [24]

    Mls: A large-scale multilingual dataset for speech research,

    V . Pratapet al., “Mls: A large-scale multilingual dataset for speech research,” inInterspeech 2020. ISCA, Oct. 2020. [Online]. Available: http://dx.doi.org/10.21437/Interspeech. 2020-2826

    2020 doi

  17. [25]

    Common voice: A massively-multilingual speech corpus,

    R. Ardilaet al., “Common voice: A massively-multilingual speech corpus,” 2020. [Online]. Available: https://arxiv.org/abs/ 1912.06670

    2020 arXiv

  18. [26]

    The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,

    D. Galvezet al., “The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage,” 2021. [Online]. Available: https://arxiv.org/abs/2111.09344

    2021 arXiv

  19. [27]

    On the utility of self-supervised mod- els for prosody-related tasks,

    G.-T. Lin, C.-L. Feng, W.-P. Huang, Y . Tseng, T.-H. Lin, C.-A. Li, H.-y. Lee, and N. G. Ward, “On the utility of self-supervised mod- els for prosody-related tasks,” in2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1104–1111

    2023

  20. [28]

    V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,

    C. Wanget al., “V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” 2021. [Online]. Available: https: //arxiv.org/abs/2101.00390

    2021 arXiv

  21. [29]

    free public domain audiobooks,

    LibriV ox, “free public domain audiobooks,” 2024. [Online]. Available: https://librivox.org/

    2024

  22. [30]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,”

  23. [32]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  24. [33]

    Specaugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” inInterspeech, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:121321299

    2019

  25. [35]

    Msp-podcast corpus: A large naturalistic speech emotional dataset,

    “Msp-podcast corpus: A large naturalistic speech emotional dataset,” https://ecs.utdallas.edu/research/researchlabs/msp-lab/ MSP-Podcast.html, 2024, accessed: 29-08-2024

    2024

  26. [36]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020. [Online]. Available: https://arxiv.org/abs/ 2006.11477

    2020 arXiv

  27. [37]

    mhubert-147: A compact multilingual hubert model,

    M. Z. Boito, V . Iyer, N. Lagos, L. Besacier, and I. Calapodescu, “mhubert-147: A compact multilingual hubert model,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06371

    2024 arXiv

  28. [38]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”IEEE Journal of Selected...

    2022

  29. [2020]

    Available: https://arxiv.org/abs/2005.08100

    [Online]. Available: https://arxiv.org/abs/2005.08100

    2005 arXiv

  30. [2021]

    Available: https://arxiv.org/abs/2106.09760

    [Online]. Available: https://arxiv.org/abs/2106.09760

    arXiv

  31. [2023]

    Available: https://arxiv.org/abs/2303.01037

    [Online]. Available: https://arxiv.org/abs/2303.01037

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.