Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Optimized Self-supervised Training with BEST-RQ for Speech Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that combining multiple quantization codebooks, a KL-divergence auxiliary loss, and cluster-specific codebooks derived from low-level audio features improves BEST-RQ self-supervised speech pre-training, cutting word error…

desk verdict Solid engineering result with a real attribution gap: full-scale WER gains are promising, but the paper does not isolate which new component drives them. read the letter →

arxiv 2501.16131 v1 pith:EV7NYA3R submitted 2025-01-27 cs.SD

Ilja Baumann , Dominik Wagner , Korbinian Riedhammer , Tobias Bocklet This is my paper · ORCID
classification cs.SD
keywords self-supervisedlearningspeechrecognitionBEST-RQrandom-projectionquantizermulti-codebookKL-divergenceregularizationcluster-specificcodebooksLibri
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the BEST-RQ recipe for self-supervised speech pre-training can be pushed to substantially lower word error rates without changing the downstream model size. The authors replace the single random-projection codebook with several codebooks, add a KL-divergence term that pushes the model's output distributions toward the quantizer's similarity scores, and assign each utterance a codebook chosen by clustering low-level acoustic features. In limited train-100 experiments, the multi-codebook step alone improves test-clean WER by 11.2% relative, and the KL loss adds a further gain; on the full 960-hour LibriSpeech pre-training, the combined recipe improves WER by up to 23.8% on test-clean and 30.6% on test-other. It also reports faster convergence and more stable validation loss during pre-training. If correct, this gives a simple, parameter-free quantizer path to better self-supervised speech models.

What carries the argument

The core mechanism is the random-projection quantizer (RQ), a fixed, randomly initialized projection matrix and codebook that maps each masked speech frame to the index of its nearest codebook vector, producing the training target. The paper keeps that quantizer frozen but modifies the objective and target structure in three ways: N parallel output heads and N codebooks whose losses are averaged; a KL-divergence term comparing each output distribution to the quantizer's cosine-similarity vector, scaled by a weight; and utterance-level clustering of low-level features so that each cluster gets its own codebook, with the matching codebook's loss weighted at least twice as heavily as the others. The argument is carried by the interaction of these three modifications with the fixed quantizer, which remains parameter-free and collapse-free.

What would settle it

Compute the cosine-similarity vectors d_i from the random-projection quantizer on a few LibriSpeech batches and check whether each vector's entries are nonnegative and sum to 1. If the sums are not always 1, the KL term in Eq. (2) is not a well-defined divergence, and an ablation that replaces it with a properly normalized distribution would settle whether the auxiliary-loss gains are real.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that BEST-RQ's fixed random-projection quantizer is not a bottleneck: by using six independent codebooks with per-codebook output heads, by training with a weighted sum of cross-entropy and a KL-divergence regularizer against the quantizer's cosine-similarity vectors, and by specializing codebooks to utterance clusters from MFCC, spectral-contrast, spectral roll-off, and zero-crossing-rate features, the model learns a better representation for speech recognition. The authors report that this setup beats the single-codebook BEST-RQ baseline on every LibriSpeech dev and test set, with the largest relative WER reduction appearing on test-other (30.6%). They also observe that the modified pre-training reaches the baseline's best validation loss in about a third of the steps, and that the improved encoder transfers to fine-tuning at no extra model size.

Load-bearing premise

The auxiliary KL loss assumes that the quantizer's per-frame similarity scores form a proper probability distribution (nonnegative and summing to 1). The paper does not describe a normalization step for those cosine similarities, so if they do not sum to 1 the reported gains from the auxiliary loss rest on an invalid objective.

Editorial extensions

If this is right

  • With the full 960-hour LibriSpeech pre-training, the optimized recipe improves WER by 23.8% relative on test-clean and 30.6% on test-other compared with the single-codebook baseline.
  • In the train-100 preliminary setup, using six codebooks instead of one reduces test-clean WER by 11.2% relative.
  • Combining cross-entropy with the KL-divergence auxiliary loss improves WER over cross-entropy alone (7.8% on dev-clean and 8.4% on test-clean with one codebook; 4.6% and 4.5% with six) and reaches the baseline's best validation loss in roughly a third of the training steps.
  • Cluster-specific codebooks on top of the six-codebook CE+KL setup give a further WER reduction (to 16.13 dev-clean / 16.28 test-clean in the preliminary setup) and stabilize validation loss.
  • The extra pre-training complexity disappears at fine-tuning: the downstream encoder is the same size as the baseline's, so the WER gains do not cost inference compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the KL term only compares the model's output distribution to the quantizer's similarity scores, it should transfer to other self-supervised speech objectives; testing it on those objectives would show whether the gain is tied to BEST-RQ or to the regularizer itself.
  • Explicitly normalizing the similarity vectors before computing the KL term (for example with a softmax) would make the auxiliary loss a true divergence; a controlled ablation with and without normalization would isolate whether any improvement comes from the divergence or just from the extra smoothing.
  • The utterance-level clustering could be refined: instead of hand-chosen low-level features, a jointly learned clustering or clustering on deeper embeddings might give larger gains, and the same idea could apply to noise conditions or speakers.
  • If the preliminary pattern holds, the multi-codebook component likely contributes the largest share of the full-scale gain, but the paper does not demonstrate that breakdown at 960 hours.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes three modifications to BEST-RQ self-supervised speech pre-training: (i) using multiple codebooks, (ii) adding a KL-divergence auxiliary loss to the standard cross-entropy objective, and (iii) assigning cluster-specific codebooks derived from low-level utterance features. Preliminary experiments on LibriSpeech train-100 show relative WER improvements (e.g., 11.2% from multiple codebooks, 4.5% from adding KL loss on top of the 6-codebook setting). On full 960-hour LibriSpeech pre-training and fine-tuning, the combined recipe reports relative WER improvements of up to 23.8% on test-clean and 30.6% on test-other, together with faster convergence and more stable pre-training.

Significance. If fully supported, the proposed recipe is a simple, practical improvement over BEST-RQ that does not increase fine-time model size and is reported to stabilize and accelerate pre-training. The paper is transparent about its limitations, provides concrete WER tables, and builds on an open-source implementation (SpeechBrain), which aids reproducibility. The significance is tempered, however, by the lack of full-scale ablations for the individual components and by an unnormalized KL term that may invalidate the stated objective; these issues need addressing before the headline gains can be confidently attributed.

major comments (3)
  1. [Section III-C, Eq. (2)] The KL-divergence term in Eq. (2) uses 'similarity vectors' d_i, but the random-projection quantizer with cosine similarity (as stated in Section IV) yields cosine similarities, which are not guaranteed to be nonnegative and do not necessarily sum to 1. Without a normalization step (e.g., softmax or shifting and renormalizing), L_KL is not a valid KL divergence, may be negative, and does not measure a distribution-to-distribution distance. This affects all experiments that include the CE+KL objective, including the headline results in Table IV. The authors should specify the exact normalization applied to d_i and confirm that the reported improvements are not an artifact of an invalid loss.
  2. [Section IV-F, Table IV and Section VI] The full-scale comparison in Table IV reports only the baseline versus the combined 'ours' recipe (6 codebooks + CE+KL + cluster-specific codebooks) on 960 hours, with no intermediate ablations at that scale. Section VI explicitly states that 'the effect of each modification on the full pre-training and fine-tuning still needs to be explored.' This is a load-bearing gap because Section IV-B shows that changing the codebook size/dimension from 8192/16 to 10240/32 alone yields a 4.2% relative improvement on test-clean, yet Tables II–IV do not state which codebook size/dimension is used for the baseline and for 'ours'. To support the claim that the proposed modifications cause the 23.8%/30.6% improvements, the paper should either provide full-scale ablations or explicitly restrict the claim to the combined recipe and fully specify the configuration of each baseline.
  3. [Section IV-C, Table II] The choice of N=6 codebooks is justified primarily by test-clean WER rather than a validation set, and the paper states that the difference between 4 and 6 codebooks is 'not statistically significant' without reporting any significance test, confidence interval, or variance across runs. Because the same test set is then used to report the headline improvements, this creates a selection-bias concern. The authors should select hyperparameters using a validation split (e.g., dev-clean) and report significance or variance estimates for the WER differences.
minor comments (5)
  1. [Section IV-A] The set notation for the number of codebooks, N ∈ {x | x ∈ Z, 1 ≤ x ≤ 10, x is even}, excludes the N=1 baseline that appears in Table II; the notation should include odd values or otherwise be corrected.
  2. [Section III-B] The primary and secondary loss weights w_p and w_s are only described as 'at least twice as high,' but no concrete values are reported, and the procedure for choosing the number of clusters (6) is not described. This hampers reproducibility.
  3. [Section VI] The statement 'Extreme weighting factors result in decreased WER' appears to be a typo; if the intended meaning is that extreme weights harm performance, it should read 'increased WER' or 'degraded WER.'
  4. [Section III-C] In Eq. (1), 'M represents the number of targets' is ambiguous; M appears to be the codebook size (number of classes), and this should be stated explicitly.
  5. [Throughout] Several small typos and spacing issues appear, e.g., 'wCE = 1as scaling factors' in Section III-C and 'Both, the projection matrix...' in Section II-A2. A careful proofread is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: held-out WER comparisons and external baselines; missing ablations are an attribution gap, not circularity.

full rationale

This is an empirical systems paper whose central claims are held-out word-error-rate comparisons. No equation defines a predicted quantity in terms of a fitted parameter: Equations (1)-(3) are standard loss terms with fixed weights (wCE=1, wKL=0.1), and the reported improvements are evaluated on dev/test splits that are not used to fit those weights. The multi-codebook and cluster-specific codebooks are fixed random-projection quantizers and cluster assignments derived from acoustic features, not from the ASR outputs they are used to predict, so the predictions are not equivalent to their inputs by construction. The paper invokes no self-citation chain and no uniqueness theorem: its key references [7], [10], and [13] are external prior work. The nearest concerns are non-circular. First, Section VI explicitly states, "the effect of each modification on the full pre-training and fine-tuning still needs to be explored," meaning the Table IV gain is not fully attributed to the individual novel components; that is a missing-support or attribution gap, not a circular reduction. Second, Equation (2) writes LKL in terms of "similarity vectors" d_i without stating that they are normalized to a probability distribution, so the cosine-similarity outputs may not form a valid KL-divergence target; this is a formal correctness issue, not a circularity. Finally, selecting N=6 on the basis of test-clean results is a form of test-set selection bias, but the full-scale model is then evaluated on held-out data, so the reported WERs are not statistically forced by the selection rule. Overall, the derivation chain is self-contained: the contributions are empirical configurations compared against external baselines, and no central claim reduces by the paper's own equations or citations to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several freely chosen hyperparameters and unstated modeling assumptions. The KL loss weight, the number of codebooks, the cluster weights, and the clustering configuration are all chosen empirically and not fully reported. No new physical entities are introduced; the assumptions are domain-specific choices about the training objective and feature clustering. The most fragile assumption is that the quantizer similarity vectors are valid probability distributions for the KL term, which is necessary for Eq. (2) to be a proper loss.

free parameters (5)
  • KL loss weight w_KL = 0.1
    Chosen by empirical tests on validation sets; the paper says w_KL = 0.1 and w_CE = 1 yield approximately equally weighted losses, but no search details are given. This weight directly scales the auxiliary loss and affects the reported gains from KL regularization.
  • Number of codebooks N = 6
    Selected from even values 1 through 10 based on preliminary dev-clean and test-clean WER. The authors note the difference between 4 and 6 codebooks is not statistically significant, but they choose 6 anyway, and this N is used in the full-scale experiment.
  • Cluster primary and secondary loss weights w_p, w_s = unspecified, w_p at least 2 times w_s
    The paper states only that w_p is chosen to be at least twice as high as w_s, but the actual values are not reported. This leaves the loss weighting in the cluster-specific codebook setup unknown and prevents exact replication.
  • Codebook size and dimension for final setup = unspecified
    Preliminary single-codebook experiments found 10240 length with 32 dimensions best, but the paper never states which codebook size is used for the multi-codebook or full-scale experiments. Table II appears to use the baseline 8192x16 codebook, but this is not explicit.
  • Number of acoustic clusters = 6
    Set equal to the number of codebooks. The clustering algorithm, feature normalization, and distance metric are not specified, making this a free choice that affects which utterances share a codebook.
assumptions (4)
  • domain assumption Cosine similarity, rather than L2 distance, is used to select the target index in the random-projection quantizer.
    Adopted from Google USM [10]; the paper does not evaluate this choice or compare it against the original L2 selection from BEST-RQ [7].
  • ad hoc to paper KL divergence between prediction and quantizer similarity distributions is a useful regularizer for self-supervised speech pre-training.
    Proposed without theoretical justification; the benefit is shown only empirically on LibriSpeech, and only for the particular weight w_KL = 0.1.
  • ad hoc to paper Utterance-level clustering using MFCC, spectral contrast, spectral roll-off, and zero-crossing rate yields acoustically meaningful groups that benefit from specialized codebooks.
    No evidence is provided that these features create useful clusters beyond the observed WER gain; the clustering itself is not analyzed, and the number of clusters is set equal to the number of codebooks.
  • ad hoc to paper The random-projection quantizer's similarity vectors can be treated as probability distributions for the KL term.
    Unstated and unproven in the paper. Eq. (2) requires d to be a distribution (nonnegative, summing to one), and the paper does not describe any normalization step, making this a load-bearing assumption for the auxiliary loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimized Self-supervised Training with BEST-RQ for Speech Recognition." pith.science (2026). https://pith.science/paper/EV7NYA3R

@misc{pith2026250116131,
  author       = {Pith},
  title        = {Pith review of: Optimized Self-supervised Training with BEST-RQ for Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EV7NYA3R}},
  note         = {Machine review of arXiv:2501.16131}
}
read the original abstract

Self-supervised learning has been successfully used for various speech related tasks, including automatic speech recognition. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) has achieved state-of-the-art results in speech recognition. In this work, we further optimize the BEST-RQ approach using Kullback-Leibler divergence as an additional regularizing loss and multi-codebook extension per cluster derived from low-level feature clustering. Preliminary experiments on train-100 split of LibriSpeech result in a relative improvement of 11.2% on test-clean by using multiple codebooks, utilizing a combination of cross-entropy and Kullback-Leibler divergence further reduces the word error rate by 4.5%. The proposed optimizations on full LibriSpeech pre-training and fine-tuning result in relative word error rate improvements of up to 23.8% on test-clean and 30.6% on test-other using 6 codebooks. Furthermore, the proposed setup leads to faster convergence in pre-training and fine-tuning and additionally stabilizes the pre-training.

Figures

Figures reproduced from arXiv: 2501.16131 by the authors.

Figure 1
Figure 1. BEST-RQ training setup: (left) shows the baseline setup of BEST-RQ proposed in [7], (right) shows our proposed modifications, including KL-divergence [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Validation loss in pre-training for the baseline method and our [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 12 canonical work pages

  1. [1]

    Dimensionality reduction by learning an invariant mapping,

    R. Hadsell, S. Chopra, and Y . LeCun, “Dimensionality reduction by learning an invariant mapping,” in 2006 IEEE Computer Society Con- ference on Computer Vision and Pattern Recognition (CVPR’06), vol. 2, 2006, pp. 1735–1742

    work page 2006

  2. [2]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Ed...

    work page 2019

  3. [3]

    wav2vec 2.0: a framework for self-supervised learning of speech representations,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representations,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20, 2020

    work page 2020

  4. [4]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 29, p. 3451–3460, Oct. 2021

    work page 2021

  5. [5]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  6. [6]

    w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244–250, 2021

    work page 2021

  7. [7]

    Self-supervised learning with random-projection quantizer for speech recognition,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in International Conference on Machine Learning , 2022, pp. 3915–3924

    work page 2022

  8. [8]

    Accented speech recognition with accent-specific codebooks,

    D. D. Prabhu, P. Jyothi, S. Ganapathy, and V . Unni, “Accented speech recognition with accent-specific codebooks,” in The 2023 Conference on Empirical Methods in Natural Language Processing , 2023

    work page 2023

Show all 16 references
  1. [9]

    Conformer: Convolution- augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution- augmented transformer for speech recognition,” in Interspeech 2020 , 2020, pp. 5036–5040

    2020

  2. [10]

    Google usm: Scaling automatic speech recognition beyond 100 languages,

    Y . Zhang, W. Han, J. Qin, Y . Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V . Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. N. Sainath, P. J. Moreno, C.-C. Chiu, J. Schalkwy...

    2023 arXiv

  3. [11]

    Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,

    S. Davis and P. Mermelstein, “Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 28, no. 4, pp. 357–366, 1980

    1980

  4. [12]

    Music type classification by spectral contrast feature,

    D.-N. Jiang, L. Lu, H.-J. Zhang, J.-H. Tao, and L.-H. Cai, “Music type classification by spectral contrast feature,” in Proceedings. IEEE international conference on multimedia and expo , vol. 1. IEEE, 2002, pp. 113–116

    2002

  5. [13]

    Open imple- mentation and study of best-rq for speech processing,

    R. Whetten, T. Parcollet, M. Dinarelli, and Y . Est `eve, “Open imple- mentation and study of best-rq for speech processing,” 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 460–464, 2024

    2024

  6. [14]

    SpeechBrain: A general- purpose speech toolkit,

    M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lu- gosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y . Gao, R. D. Mori, and Y . Bengio, “SpeechBrain: A general- p...

    2021 arXiv

  7. [15]

    Open-source conversational ai with speechbrain 1.0,

    M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y . Wang, P. Mousavi, L. D. Libera, A. Ploujnikov, F. Pais- san, D. Borra, S. Zaiem, Z. Zhao, S. Zhang, G. Karakasidis, S.-L. Yeh, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Zuluaga-Gomez, S. M. Mo...

    2024

  8. [16]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.