k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning

Daniel Povey; Fangjun Kuang; Jianheng Zhuo; Liyong Guo; Long Lin; Wei Kang; Xiaoyu Yang; Xie Chen; Yifan Yang; Zengrui Jin

REVIEW 3 major objections 7 minor 1 cited by

k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The k2SSL framework claims that Zipformer SSL beats HuBERT Base by up to 34.8% WER at a 3.5x pretraining speedup, and that Zipformer Large matches HuBERT Large in 5/8 the steps.

desk verdict A credible and useful engineering result — Zipformer works well as an SSL backbone — but the headline WER claims overstate the architecture's contribution because batch size changes with the backbone. read the letter →

arxiv 2411.17100 v2 pith:OK3GQKXB submitted 2024-11-26 eess.AS

Yifan Yang , Jianheng Zhuo , Zengrui Jin , Ziyang Ma , Xiaoyu Yang , Zengwei Yao , Liyong Guo , Wei Kang

show 4 more authors

Fangjun Kuang Long Lin Daniel Povey Xie Chen

This is my paper · ORCID

classification eess.AS

keywords self-supervisedlearningspeechrepresentationHuBERTZipformerautomaticrecognitionefficientpretrainingmaskedpredictionRNN-Tloss

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that self-supervised speech representation learning can be made both faster and more accurate for downstream ASR by rebuilding the HuBERT recipe: stripping memory-heavy normalizers, replacing the Transformer encoder with the Zipformer encoder, switching to a cross-entropy pre-training loss, and fine-tuning with pruned RNN-T. On LibriSpeech-960h pretraining followed by 100h fine-tuning, Zipformer Base cuts word error rate by up to 34.8% relative to the reference HuBERT Base on dev-other and 32.4% on test-other, while using roughly 3.5x fewer total GPU hours. Scaled to 60,000 hours of LibriLight, Zipformer Large matches HuBERT Large's WER on LibriSpeech-960h in 250k pretraining steps instead of 400k. If these comparisons hold, the recipe lowers the compute barrier to SSL speech pretraining substantially.

What carries the argument

The central object is the Zipformer encoder, a U-Net-like stack of six cascaded blocks that process 50Hz speech at varying frame rates (50, 25, 12.5, 6.25, 12.5, and 25Hz) with BiasNorm and Swoosh activations, paired with the ScaledAdam optimizer. This combination makes longer contexts and larger batches memory-feasible, which is the mechanism behind both the speedup and the accuracy gain. Supporting machinery includes the simplified cross-entropy pre-training loss, which removes the codeword-embedding updates, and the pruned RNN-T loss, which makes fine-tuning memory-efficient.

What would settle it

Pre-train a Transformer HuBERT Base with the same 600-second per-GPU batch size, the same cross-entropy loss, ScaledAdam, the same number of steps, and the same pruned RNN-T fine-tuning, and compare WER with Zipformer Base; if the gap narrows to near zero, the architectural claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a HuBERT-style masked-prediction learner does not need the Transformer encoder to do well; the Zipformer encoder, originally designed for streaming ASR, transfers its efficiency to SSL when combined with ScaledAdam and a simplified cross-entropy loss. The authors report that k2SSL-trained HuBERT Base with ScaledAdam is already 1.48x faster than the reference HuBERT, the CE loss brings it to 2.12x, and the Zipformer backbone brings the total to 3.53x with consistent relative WER reductions across all LibriSpeech dev/test sets (up to 34.8% on dev-other, 32.4% on test-other). They further claim that Zipformer Large reaches HuBERT-Large-level accuracy on the 960h fine-tuning benchmark with 250k rather than 400k pretraining steps on LibriLight, and that the memory savings allow Base to pretrain on 8 V100 GPUs and Large on 32, without gradient accumulation. The authors attribute the gains to the Zipformer architecture and training recipe, while noting that total batch size is itself a critical performance factor.

Load-bearing premise

The central claim assumes the WER gains come from the Zipformer backbone and recipe, not from the roughly sevenfold larger batch size, even though the paper calls total batch size critical.

Editorial extensions

If this is right

The framework needs only 8 V100 GPUs for Base and 32 for Large, so pretraining becomes feasible for research groups that previously could only fine-tune.
Using the same k-means targets and masking as HuBERT, the Zipformer backbone is claimed to lower fine-tuned WER on every LibriSpeech split without added data.
The 3.5x GPU-hour reduction means SSL pretraining experiments can be repeated and ablated at roughly one-third the previous cost.
At the 60,000-hour scale, Zipformer Large reaches HuBERT Large-level WER in 250k steps instead of 400k, which shortens large-scale pretraining schedules by 37.5%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The architecture effect is not cleanly separated from the batch-size effect: Zipformer Base runs at 600 seconds per GPU versus 87.5 seconds for HuBERT Base, and the paper itself calls total batch size critical.
A batch-matched Transformer control (same 600-second batches, ScaledAdam, CE loss, and step count) would tell how much of the 34.8% WER gain is the encoder versus the larger effective batch.
The recipe components are separable, so practitioners skeptical of the architecture claim can still adopt the CE loss, pruned RNN-T fine-tuning, or streaming data-loading improvements independently.
Nothing in the paper tests transfer beyond LibriSpeech/LibriLight and ASR fine-tuning; whether the same recipe helps other downstream tasks or languages remains open.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The thing to know: this is a real engineering contribution. k2SSL is an open-source framework that makes HuBERT-style pretraining substantially cheaper (3.5x on GPU hours) and simultaneously gets better WER on LibriSpeech fine-tuning. The new piece is using Zipformer as the SSL backbone with CE loss and ScaledAdam, which nobody had done before. The paper also ships code and models, and uses the exact same external k-means targets as HuBERT for the pretraining comparison, which keeps the comparison honest.

The internal controls are actually useful. Row 2 vs Row 3 of Table II isolates the CE-loss change (with everything else held, including batch size and backbone), and Row 3 vs Row 4 isolates the backbone change — except it doesn't fully. The batch size changes there too: effective 350s per update for the Transformer CE run vs. 600s for the Zipformer run. That's a 1.71x larger effective batch, and the paper itself cites HuBERT for the claim that total batch size is a critical factor. So the 24-35% relative WER gains are not cleanly attributable to Zipformer. The system-level speedup claim stands, because that's about GPU-hours and memory, not about which component causes the WER gain. A matched-batch ablation (or even a run with Zipformer at 350s) would settle it.

Other soft spots: no error bars or repeated seeds reported, which matters when the headline deltas are on the order of a few percent. The comparison against WavLM and wav2vec 2.0 in Table III is taken from other papers' fine-tuning setups, so cross-framework differences are uncontrolled; the authors are transparent about that, and the within-framework comparisons carry the load.

But the load-bearing claim — that k2SSL is faster and better than fairseq HuBERT as a system — is credible and supported. The architecture-specific superiority claim is conditional.

Who should read it: anyone doing speech SSL pretraining, especially groups with limited GPUs. The framework is real; the numbers will be useful as a benchmark. I'd send it to a serious referee, with the request that they push for the batch-matched ablation and a small multi-seed check. If those come through, this becomes a solid reference point for the field.

Best.

Referee Report

3 major / 7 minor

Summary. The manuscript describes k2SSL, an open-source framework for self-supervised speech representation learning oriented toward ASR. The authors identify memory and I/O inefficiencies in the standard HuBERT/fairseq pipeline; they modify the HuBERT architecture by removing per-layer Fp32 LayerNorm, replacing the original loss with cross-entropy, and switching the Transformer backbone to the Zipformer encoder with the ScaledAdam optimizer. They evaluate on LibriSpeech-100h fine-tuning with pruned RNN-T and CTC losses. Table II reports that k2SSL-pre-trained HuBERT Base is faster than fairseq HuBERT Base, and that Zipformer Base gives relative WER reductions of up to 34.8% on dev-other and 32.4% on test-other versus HuBERT Base while using 531 GPU hours versus 1878. Table IV reports that Zipformer Large matches HuBERT Large on LibriSpeech-960h with 250k pre-training steps instead of 400k. The paper is a systems and empirical contribution; code and checkpoints are promised as public.

Significance. The practical contribution is potentially valuable: if the reported speedups and WER reductions are reproducible, k2SSL lowers the compute barrier for SSL-based ASR and provides a working open-source implementation. The paper has some internal controls: Rows 2 and 3 of Table II isolate the effect of the CE loss, and the use of identical HuBERT clustering targets across systems is a good design choice. The main gap is that the headline Zipformer-versus-Transformer comparison changes the effective batch size at the same time as the backbone, so the WER improvement is not cleanly attributable to Zipformer; the speedup claim is credible because it is a system-level measurement. With a matched-batch-size ablation or a carefully reworded architecture claim, the work would be a solid empirical systems paper.

major comments (3)

[§IV-A, Table I; §IV-B, Table II] The comparison between Row 3 (Transformer CE) and Row 4 (Zipformer Base CE) in Table II is used to isolate the effect of the backbone, but Table I shows that the effective pre-training batch size changes at the same time: Transformer CE uses BS/GPU=87.5s with GA=4 (effective 350s per update), while Zipformer CE uses BS/GPU=600s with GA=1 (effective 600s per update). Because Section III-B explicitly states that total batch size is a critical factor for SSL performance (citing HuBERT [3]), the 24.8-34.8% relative WER reductions in Table II cannot be attributed solely to the Zipformer encoder without a matched-batch-size ablation. The fine-tuning conditions also differ (BS/GPU 200s versus 600s and different learning rates), adding a further confound to the downstream results. Please provide an experiment that varies the backbone while holding effective batch size constant (for example, training Transformer CE with a 600s effective batch, or Zipformer with a 350s effective batch), or otherwise quantify the contribution of batch size.
[§IV-B, Table III] The cross-model comparison in Table III is not fully controlled: baseline numbers are either taken from the original papers or from checkpoints fine-tuned with unspecified configurations, while Zipformer Base is fine-tuned in k2SSL. The reader cannot tell whether the reported WavLM gap is due to the pre-trained representation or to differences in fine-tuning. Please either fine-tune all baselines under the same k2SSL protocol (including the same CTC/letter setup, learning rate schedule, and number of updates) or explicitly restrict the superiority claim to the configuration actually compared.
[§IV-B, Table IV] The claim that Zipformer Large matches HuBERT Large while requiring only 5/8 of the pre-training steps is based on a single operating point: Zipformer Large at 250k steps gives test-clean/test-other WERs of 1.8/4.0, while HuBERT Large at 400k steps gives 1.8/3.9. Because the two systems also differ in optimizer, loss, batch size, and step schedule, the step-count comparison alone does not establish that the Zipformer backbone is more sample-efficient; reporting WERs at several matched step counts (for example, both models at 250k and 400k steps) would make this claim directly testable.

minor comments (7)

[§IV-B, first bullet] The phrase 'provides a around 2.1x pre-training speedup' should be 'provides an around 2.1x pre-training speedup'.
[§IV-B, third bullet] The text says 'yielding a 3.53 pre-training speedup'; the unit should be written as '3.53x' to match Table II.
[§III-A, Eq. (2)] Equation (2) uses the symbol C without defining it; please define it in the text or refer the reader back to Eq. (1) for the notation.
[§III-A] The memory measurements are described only as 'Testing on an idle V100 GPU with a batch of six 60-second audio samples'; please specify which model variant (Base or Large) and which exact configuration each number corresponds to.
[§IV-B, Table III] Table III reports no dev-set results for WavLM Base; a sentence explaining why would avoid the appearance of selective reporting.
[§IV-A] The paper does not report standard deviations or multiple seeds; given the small margins in some comparisons (for example, Row 2 versus Row 1 on dev-other), at least one repeated run would increase confidence in the claimed differences.
[§III-D] The statement that pre-training on Libri-Light 'is initiated in merely dozens of seconds' would be more convincing with a concrete startup-time measurement or a comparison against fairseq.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the WER comparisons are empirical evaluations against external baselines and fixed external targets, not derivations from fitted inputs or self-referential claims.

full rationale

The paper's central claims are empirical systems results, not a derivation chain. Pretraining targets are taken unchanged from external HuBERT/WavLM checkpoints (Section IV-A: 'We directly utilize the target labels identical to those used in HuBERT [3] and WavLM [4] for fair comparisons'), and downstream performance is measured as fine-tuned WER on LibriSpeech, an external benchmark, against baselines run from public checkpoints or cited papers. No parameter is fitted to a subset and then relabeled as a prediction, no equation defines one claimed quantity in terms of another claimed quantity, and no load-bearing uniqueness assertion is imported from the authors' prior work. The use of Zipformer and ScaledAdam is an architectural reuse cited to prior work, but the paper's contribution is the empirical comparison, which is self-contained and externally checkable. The batch-size difference between HuBERT and Zipformer setups noted by a skeptical reader is a potential experimental confound and a threat to the attribution of gains to the backbone, but it is not circular reasoning; it concerns internal validity, not logical reduction of the result to its inputs. The system-level speedup and the WER numbers do not presuppose the conclusion they support.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or architectural entities and fits no parameter to the final WER. Its load-bearing inputs are empirical transfer assumptions: HuBERT cluster targets remain usable, Zipformer's ASR design transfers to SSL, HuBERT's masking works for Zipformer, and removing most Fp32 normalizations is harmless. The batch duration and step counts are hand-chosen experimental variables that the headline comparison depends on.

free parameters (2)

Per-GPU batch duration in seconds = 87.5s for HuBERT rows, 600s for Zipformer rows
Chosen to exploit GPU memory and differs across the compared models. The paper itself notes total batch size is a critical performance factor, so this is a hand-chosen variable that the central comparison depends on.
Number of pretraining steps = 400k for HuBERT Base, 225k for Zipformer Base, 400k vs 250k for Large
Not matched across models. The 3.5x speedup and the Large-model comparison both depend on these step counts, which were chosen by the authors rather than controlled.

assumptions (4)

domain assumption The public HuBERT k-means labels (iter1 l6 k500 and iter2 l9 k500) remain valid targets for Zipformer-based pretraining.
Invoked in Section IV-A, where the paper directly uses HuBERT's target labels to attribute performance differences to model and training variations.
domain assumption Zipformer, designed and tuned for supervised ASR, transfers to SSL masked-prediction pretraining without task-specific redesign.
The entire approach rests on this transfer, and no control experiment isolates architecture from batch size.
domain assumption HuBERT's masking strategy is appropriate for Zipformer's U-Net structure with varying frame rates across stacks.
Section IV-A states 'The same masking strategies as in [3] are used', without ablation on Zipformer.
domain assumption Removing most Fp32 LayerNorm operations in the convolutional feature extractor does not substantially hurt final performance.
Section III-A says experiments indicate this, but no supporting table or ablation is shown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning." pith.science (2026). https://pith.science/paper/OK3GQKXB

@misc{pith2026241117100,
  author       = {Pith},
  title        = {Pith review of: k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OK3GQKXB}},
  note         = {Machine review of arXiv:2411.17100}
}

read the original abstract

Self-supervised learning (SSL) has achieved great success in speech-related tasks. While Transformer and Conformer architectures have dominated SSL backbones, encoders like Zipformer, which excel in automatic speech recognition (ASR), remain unexplored in SSL. Concurrently, inefficiencies in data processing within existing SSL training frameworks, such as fairseq, pose challenges in managing the growing volumes of training data. To address these issues, we propose k2SSL, an open-source framework that offers faster, more memory-efficient, and better-performing self-supervised speech representation learning, focusing on downstream ASR tasks. The optimized HuBERT and proposed Zipformer-based SSL systems exhibit substantial reductions in both training time and memory usage during SSL training. Experiments on LibriSpeech demonstrate that Zipformer Base significantly outperforms HuBERT and WavLM, achieving up to a 34.8% relative WER reduction compared to HuBERT Base after fine-tuning, along with a 3.5x pre-training speedup in GPU hours. When scaled to 60k hours of LibriLight data, Zipformer Large exhibits remarkable efficiency, matching HuBERT Large's performance while requiring only 5/8 pre-training steps.

Figures

Figures reproduced from arXiv: 2411.17100 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining
eess.AS 2025-05 conditional novelty 5.0 of 10

A 68M-parameter Vietnamese ASR model, pretrained on 70,000 hours of unlabeled audio and fine-tuned on 50 hours of labels, reports average WER 8.31, beating Whisper Large-v3 and commercial systems.

Reference graph

Works this paper leans on

43 extracted references · 38 canonical work pages · cited by 1 Pith paper

[3]

HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

W. N. Hsu, B. Bolte, Y . H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, 2021

work page 2021
[1]

wav2vec: Unsu- pervised pre-training for speech recognition,

S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsu- pervised pre-training for speech recognition,” in Proc. INTERSPEECH, Graz, 2019

work page 2019
[2]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. NeurIPS, 2020

2020
[4]

WavLM: Large-scale self-supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, 2022

work page 2022
[5]

data2vec: A general framework for self-supervised learning in speech, vision and language,

A. Baevski, A. Babu, W. N. Hsu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” in Proc. ICML, Baltimore, 2022

work page 2022
[6]

Efficient self-supervised learning with contextualized target rep- resentations for vision, speech and language,

——, “Efficient self-supervised learning with contextualized target rep- resentations for vision, speech and language,” in Proc. ICML, Honolulu, 2023

work page 2023
[7]

Self-supervised speech representation learning: A review,

A. Mohamed, H. yi Lee, L. Borgholt et al. , “Self-supervised speech representation learning: A review,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, 2022

work page 2022
[8]

Recent advances in end-to-end automatic speech recog- nition,

J. Li et al. , “Recent advances in end-to-end automatic speech recog- nition,” APSIPA Transactions on Signal and Information Processing , vol. 11, 2022

work page 2022

Show all 43 references

[9]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar et al. , “Attention is all you need,” in Proc. NeurIPS, Long Beach, 2017

2017
[10]

SUPERB: Speech processing universal performance benchmark,

S. W. Yang, H. Chi, Y . S. Chuang et al., “SUPERB: Speech processing universal performance benchmark,” in Proc. INTERSPEECH , Brno, 2021

2021
[11]

BASE TTS: Lessons from building a billion-parameter text-to-speech model on 100k hours of data,

M. Łajszczak, G. C ´ambara, Y . Li et al. , “BASE TTS: Lessons from building a billion-parameter text-to-speech model on 100k hours of data,” arXiv preprint arXiv:2402.08093 , 2024

2024 arXiv
[12]

V oicebox: Text-guided multilingual universal speech generation at scale,

M. Le, A. Vyas, B. Shi et al. , “V oicebox: Text-guided multilingual universal speech generation at scale,” in Proc. NeurIPS , Vancouver, 2024

2024
[13]

Scaling speech technology to 1,000+ languages,

V . Pratap, A. Tjandra, B. Shi et al., “Scaling speech technology to 1,000+ languages,” arXiv preprint arXiv:2305.13516 , 2023

2023 arXiv
[14]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu et al. , “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, Honolulu, 2023

2023
[15]

Google USM: Scaling automatic speech recognition beyond 100 languages,

Y . Zhang, W. Han, J. Qinet al., “Google USM: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037 , 2023

2023 arXiv
[16]

Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context,

W. Kang, X. Yang, Z. Yao, F. Kuang, Y . Yang, L. Guo, L. Lin, and D. Povey, “Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context,” in Proc. ICASSP, Seoul, 2024

2024
[17]

Gigaspeech 2: An evolving, large-scale and multi-domain ASR corpus for low-resource languages with automated crawling, transcription and refinement,

Y . Yang, Z. Song, J. Zhuo et al. , “Gigaspeech 2: An evolving, large-scale and multi-domain ASR corpus for low-resource languages with automated crawling, transcription and refinement,” arXiv preprint arXiv:2406.11546, 2024

2024 arXiv
[18]

LibriheavyMix: a 20,000-hour dataset for single-channel reverberant multi-talker speech separation, ASR and speaker diarization,

Z. Jin, Y . Yang, M. Shi, W. Kang, X. Yang, Z. Yao, F. Kuang, L. Guo, L. Meng, L. Lin, Y . Xu, S.-X. Zhang, and D. Povey, “LibriheavyMix: a 20,000-hour dataset for single-channel reverberant multi-talker speech separation, ASR and speaker diarization,” in Proc. INTERSPEECH, Ko...

2024
[19]

fairseq: A fast, extensible toolkit for sequence modeling,

M. Ott, S. Edunov, A. Baevski et al., “fairseq: A fast, extensible toolkit for sequence modeling,” in Proc. NAACL-HLT, Minneapolis, 2019

2019
[20]

ESPnet: End-to-end speech processing toolkit,

S. Watanabe, T. Hori, S. Karita et al. , “ESPnet: End-to-end speech processing toolkit,” in Proc. INTERSPEECH, Hyderabad, 2018

2018
[21]

The S3PRL toolkit: Self-supervised speech pre-training and representation learning,

A. T. Liu and Y . S. wen, “The S3PRL toolkit: Self-supervised speech pre-training and representation learning,” https://github.com/s3prl/s3prl, 2020

2020
[22]

Zipformer: A faster and better encoder for automatic speech recognition,

Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y . Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” in Proc. ICLR, Vienna, 2024

2024
[23]

Librispeech: an ASR corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in Proc. ICASSP, South Brisbane, 2015

2015
[24]

Libri-Light: A benchmark for ASR with limited or no supervision,

J. Kahn, M. Rivi `ere, W. Zheng et al. , “Libri-Light: A benchmark for ASR with limited or no supervision,” in Proc. ICASSP, Barcelona, 2020

2020
[25]

Reducing barriers to self-supervised learning: HuBERT pre-training with academic compute,

W. Chen, X. Chang, Y . Peng, Z. Ni, S. Maiti, and S. Watanabe, “Reducing barriers to self-supervised learning: HuBERT pre-training with academic compute,” in Proc. INTERSPEECH, Dublin, 2023

2023
[26]

MelHuBERT: A simplified hubert on mel spectrograms,

T. Q. Lin, H. Y . Lee, and H. Tang, “MelHuBERT: A simplified hubert on mel spectrograms,” in Proc. ASRU, Taipei, 2023

2023
[27]

Fast-HuBERT: an efficient training framework for self-supervised speech representation learning,

G. Yang, Z. Ma, Z. Zheng et al. , “Fast-HuBERT: an efficient training framework for self-supervised speech representation learning,” in Proc. ASRU, Taipei, 2023

2023
[28]

Speech pre-training with acoustic piece,

S. Ren, S. Liu, Y . Wu, L. Zhou, and F. Wei, “Speech pre-training with acoustic piece,” in Proc. INTERSPEECH, Incheon, 2022

2022
[29]

Towards universal speech discrete tokens: A case study for ASR and TTS,

Y . Yang, F. Shen, C. Du, Z. Ma, K. Yu, D. Povey, and X. Chen, “Towards universal speech discrete tokens: A case study for ASR and TTS,” in Proc. ICASSP, Seoul, 2024

2024
[30]

Ctcbert: Advancing hidden-unit bert with ctc objectives,

R. Fan, Y . Wang, Y . Gaur, and J. Li, “Ctcbert: Advancing hidden-unit bert with ctc objectives,” in Proc. ICASSP, Rhodes Island, 2023

2023
[31]

Supervision-guided codebooks for masked prediction in speech pre- training,

C. Wang, Y . Wang, Y . Wu, S. Chen, J. Li, S. Liu, and F. Wei, “Supervision-guided codebooks for masked prediction in speech pre- training,” in Proc. INTERSPEECH, Incheon, 2022

2022
[32]

Pushing the limits of unsupervised unit discovery for SSL speech representation,

Z. Ma, Z. Zheng, G. Yang, Y . Wang, C. Zhang, and X. Chen, “Pushing the limits of unsupervised unit discovery for SSL speech representation,” in Proc. INTERSPEECH, Dublin, 2023

2023
[33]

Towards end-to-end unsupervised speech recognition,

A. H. Liu, W. N. Hsu, M. Auli, and A. Baevski, “Towards end-to-end unsupervised speech recognition,” in Proc. SLT, Doha, 2022

2022
[34]

Connec- tionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. J. Gomez, and J. Schmidhuber, “Connec- tionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, Pittsburgh, 2006

2006
[35]

Pruned RNN-T for fast, memory-efficient ASR training,

F. Kuang, L. Guo, W. Kang, L. Lin, M. Luo, Z. Yao, and D. Povey, “Pruned RNN-T for fast, memory-efficient ASR training,” in Proc. INTERSPEECH, Incheon, 2022

2022
[36]

Speech recognition with deep recurrent neural networks,

A. Graves, A. rahman Mohamed, and G. E. Hinton, “Speech recognition with deep recurrent neural networks,” in Proc. ICASSP , Vancouver, 2013

2013
[37]

Lhotse: a speech data representation library for the modern deep learning ecosystem,

P. ˙Zelasko, D. Povey, J. Trmal, and S. Khudanpur, “Lhotse: a speech data representation library for the modern deep learning ecosystem,” in NeurIPS Data-Centric AI Workshop , 2021

2021
[38]

Neural machine translation of rare words with subword units,

R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proc. ACL, Berlin, 2016

2016
[39]

Fast and parallel decoding for transducer,

W. Kang, L. Guo, F. Kuang, L. Lin, M. Luo, Z. Yao, X. Yang, P. Zelasko, and D. Povey, “Fast and parallel decoding for transducer,” in Proc. ICASSP, Rhodes Island, 2023

2023
[40]

Wav2letter++: A fast open-source speech recognition system,

V . Pratap, A. Y . Hannun, Q. Xuet al., “Wav2letter++: A fast open-source speech recognition system,” in Proc. ICASSP, Brighton, 2019

2019
[41]

Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,

Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,” in Proc. ICASSP, Barcelona, 2020

2020
[42]

Conformer: Convolution-augmented transformer for speech recognition,

A. Gulati, J. Qin, C. C. Chiu et al., “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. INTERSPEECH, Shanghai, 2020

2020
[43]

Pushing the limits of semi- supervised learning for automatic speech recognition,

Y . Zhang, J. Qin, D. S. Park et al. , “Pushing the limits of semi- supervised learning for automatic speech recognition,” in Proc. NeurIPS Self-Supervised Learning for Speech and Audio Processing Workshop , 2020

2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [3]

HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

W. N. Hsu, B. Bolte, Y . H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, 2021

work page 2021

[2] [1]

wav2vec: Unsu- pervised pre-training for speech recognition,

S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsu- pervised pre-training for speech recognition,” in Proc. INTERSPEECH, Graz, 2019

work page 2019

[3] [2]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. NeurIPS, 2020

2020

[4] [4]

WavLM: Large-scale self-supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, 2022

work page 2022

[5] [5]

data2vec: A general framework for self-supervised learning in speech, vision and language,

A. Baevski, A. Babu, W. N. Hsu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” in Proc. ICML, Baltimore, 2022

work page 2022

[6] [6]

Efficient self-supervised learning with contextualized target rep- resentations for vision, speech and language,

——, “Efficient self-supervised learning with contextualized target rep- resentations for vision, speech and language,” in Proc. ICML, Honolulu, 2023

work page 2023

[7] [7]

Self-supervised speech representation learning: A review,

A. Mohamed, H. yi Lee, L. Borgholt et al. , “Self-supervised speech representation learning: A review,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, 2022

work page 2022

[8] [8]

Recent advances in end-to-end automatic speech recog- nition,

J. Li et al. , “Recent advances in end-to-end automatic speech recog- nition,” APSIPA Transactions on Signal and Information Processing , vol. 11, 2022

work page 2022