{"id":"4b24fe74-2220-4e33-8f21-b83ee7c788b6","arxiv_id":"2501.16131","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding multiple codebooks, a KL-divergence regularizer, and cluster-specific codebooks to BEST-RQ improves LibriSpeech ASR word error rates by up to 30.6% relative with faster convergence.","lead":"Speech recognition pre-training gets a boost from three tweaks to the BEST-RQ self-supervised method: several codebooks instead of one, an added KL-divergence loss, and codebooks specialized by acoustic clusters. On LibriSpeech the combined recipe cuts word errors by up to 24% on the clean test set and 31% on the other test set, with faster pre-training and fine-tuning convergence reported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-scale WER gain is not attributable to the novel components: Table IV lacks ablations at 960h, Section VI admits this, and the baseline versus ours may differ in codebook size.","rationale":"The reader's weakest_assumption was the unnormalized KL divergence in Eq. (2), which is a real mathematical gap. I agree that needs clarification, but I see an even more decisive issue for the central quantitative claim: the full-scale result is a single combined comparison without ablations, and the authors themselves flag in Section VI that the per-component effect at full scale is unexplored. Because Section IV-B identifies a codebook-size change that alone improves WER by 4.2% relative, the omission of codebook-size details in Tables II and IV creates a concrete attribution risk: part of the 23.8%/30.6% headline gain could stem from a known codebook-size effect rather than the proposed optimizations. The preliminary ablations are on train-100/25 epochs, so they do not establish transfer to the full 960h/100k-step setting. This does not mean the combined recipe is ineffective; it means the paper currently lacks the evidence needed to support the specific claim that the combination of multiple codebooks, KL loss, and cluster-specific codebooks is responsible for the reported improvements. The appropriate remedy is the same conditional requirement the reader already set: add full-scale ablations, specify codebook sizes, and clarify the loss normalization. Therefore my read does not change the reader's verdict; it sharpens the reason why the condition is necessary.","tokens_in":7694,"tokens_out":8440,"duration_ms":78018,"concrete_test":"Run full-scale ablations on 960h with matched codebook size/dimension and compute budget: (a) single-codebook CE baseline at the best codebook size (10240/32), (b) 6 codebooks CE only, (c) 6 codebooks CE+KL, and (d) 6 codebooks CE+KL+cluster-specific; report test-clean and test-other after 100k steps, with run-to-run variance. Also state explicitly the codebook size/dim used for the Table IV baseline and for 'ours'. If (c) and (d) do not improve over (b) by more than the noise floor, the attribution of the headline improvement to the novel KL and cluster-specific components fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is attribution of the full-scale improvement. Table IV reports only baseline versus the combined 'ours' recipe (6 codebooks + CE+KL + cluster-specific) on 960 hours, with no intermediate ablations at this scale. Section VI explicitly states: 'the effect of each modification on the full pre-training and fine-tuning still needs to be explored.' This is a stated missing support. It matters because Section IV-B shows that a single-codebook codebook-size change (10240/32 vs the underlined 8192/16 baseline) already yields 4.2% relative improvement on test-clean, yet neither Table II nor Table IV states which codebook size/dimension is used for the baseline and for 'ours'. If 'ours' uses the better codebook size, or the baseline uses the worse one, part of the reported 23.8%/30.6% gain may be due to codebook size rather than the proposed multi-codebook, KL, or cluster-specific components. The preliminary support in Tables II-III is on train-100 with only 25 epochs, and no evidence shows those small-scale gains transfer to 960h/100k steps. Additionally, Eq. (2) leaves the similarity vectors d_i unnormalized, so the KL term is not a proper KL divergence; this is a secondary but independent correctness concern that also needs fixing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three modifications to BEST-RQ self-supervised speech pre-training: (i) using multiple codebooks, (ii) adding a KL-divergence auxiliary loss to the standard cross-entropy objective, and (iii) assigning cluster-specific codebooks derived from low-level utterance features. Preliminary experiments on LibriSpeech train-100 show relative WER improvements (e.g., 11.2% from multiple codebooks, 4.5% from adding KL loss on top of the 6-codebook setting). On full 960-hour LibriSpeech pre-training and fine-tuning, the combined recipe reports relative WER improvements of up to 23.8% on test-clean and 30.6% on test-other, together with faster convergence and more stable pre-training.","tokens_in":8031,"tokens_out":3885,"duration_ms":35110,"significance":"If fully supported, the proposed recipe is a simple, practical improvement over BEST-RQ that does not increase fine-time model size and is reported to stabilize and accelerate pre-training. The paper is transparent about its limitations, provides concrete WER tables, and builds on an open-source implementation (SpeechBrain), which aids reproducibility. The significance is tempered, however, by the lack of full-scale ablations for the individual components and by an unnormalized KL term that may invalidate the stated objective; these issues need addressing before the headline gains can be confidently attributed.","major_comments":[{"comment":"The KL-divergence term in Eq. (2) uses 'similarity vectors' d_i, but the random-projection quantizer with cosine similarity (as stated in Section IV) yields cosine similarities, which are not guaranteed to be nonnegative and do not necessarily sum to 1. Without a normalization step (e.g., softmax or shifting and renormalizing), L_KL is not a valid KL divergence, may be negative, and does not measure a distribution-to-distribution distance. This affects all experiments that include the CE+KL objective, including the headline results in Table IV. The authors should specify the exact normalization applied to d_i and confirm that the reported improvements are not an artifact of an invalid loss.","section":"Section III-C, Eq. (2)"},{"comment":"The full-scale comparison in Table IV reports only the baseline versus the combined 'ours' recipe (6 codebooks + CE+KL + cluster-specific codebooks) on 960 hours, with no intermediate ablations at that scale. Section VI explicitly states that 'the effect of each modification on the full pre-training and fine-tuning still needs to be explored.' This is a load-bearing gap because Section IV-B shows that changing the codebook size/dimension from 8192/16 to 10240/32 alone yields a 4.2% relative improvement on test-clean, yet Tables II–IV do not state which codebook size/dimension is used for the baseline and for 'ours'. To support the claim that the proposed modifications cause the 23.8%/30.6% improvements, the paper should either provide full-scale ablations or explicitly restrict the claim to the combined recipe and fully specify the configuration of each baseline.","section":"Section IV-F, Table IV and Section VI"},{"comment":"The choice of N=6 codebooks is justified primarily by test-clean WER rather than a validation set, and the paper states that the difference between 4 and 6 codebooks is 'not statistically significant' without reporting any significance test, confidence interval, or variance across runs. Because the same test set is then used to report the headline improvements, this creates a selection-bias concern. The authors should select hyperparameters using a validation split (e.g., dev-clean) and report significance or variance estimates for the WER differences.","section":"Section IV-C, Table II"}],"minor_comments":[{"comment":"The set notation for the number of codebooks, N ∈ {x | x ∈ Z, 1 ≤ x ≤ 10, x is even}, excludes the N=1 baseline that appears in Table II; the notation should include odd values or otherwise be corrected.","section":"Section IV-A"},{"comment":"The primary and secondary loss weights w_p and w_s are only described as 'at least twice as high,' but no concrete values are reported, and the procedure for choosing the number of clusters (6) is not described. This hampers reproducibility.","section":"Section III-B"},{"comment":"The statement 'Extreme weighting factors result in decreased WER' appears to be a typo; if the intended meaning is that extreme weights harm performance, it should read 'increased WER' or 'degraded WER.'","section":"Section VI"},{"comment":"In Eq. (1), 'M represents the number of targets' is ambiguous; M appears to be the codebook size (number of classes), and this should be stated explicitly.","section":"Section III-C"},{"comment":"Several small typos and spacing issues appear, e.g., 'wCE = 1as scaling factors' in Section III-C and 'Both, the projection matrix...' in Section II-A2. A careful proofread is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical question, and the preliminary experiments are clearly presented. The strongest issue is attribution: the full-scale result is only shown for the combined recipe, and the codebook-size confound plus the unnormalized KL term undermine the internal validity of the central claim. I believe these are fixable with additional experiments and reporting, which is why I recommend major revision rather than rejection. If the venue is a workshop or short paper where full-scale ablations are infeasible, the authors should at least explicitly frame Table IV as a system comparison and remove the implication that each component contributes on its own; the KL normalization must be fixed in any case."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical paper on BEST-RQ. The authors stack a few cheap modifications — multiple codebooks (borrowed from USM), a KL-divergence auxiliary loss, and cluster-specific codebooks — and report up to 23.8%/30.6% relative WER improvement on LibriSpeech test sets after 960h pretraining, plus faster convergence. If those numbers reproduce, it's a meaningful recipe for the speech SSL community. I believe the broad result is probably true; the small-scale experiments are internally consistent, and the paper is honest about what is and isn't tested.\n\nWhat's actually new: the specific combination of cluster-specific codebooks with CE+KL loss, and the observation that this speeds up convergence. The multi-codebook piece is from USM, so that's not new, and accent-specific codebooks exist. But the acoustic clustering and the KL regularizer are legitimate extensions, and the codebook-size study is a useful sanity check.\n\nWhere it gets soft: the attribution. Table IV compares a 6-codebook 'ours' (with CE+KL and cluster-specific codebooks) to a single-codebook baseline, but no full-scale ablations isolate the contributions. Section VI explicitly says that still needs exploring, which is the honest thing to write, but it means the headline gain is a bundle, not a verdict. Worse, neither Table II nor IV states the codebook size/dimension for each condition. The preliminary table shows a simple codebook-size change (10240/32 vs 8192/16) gives 4.2% relative improvement, so without that detail part of the 23.8% could just be codebook size. That's a load-bearing omission.\n\nSecond, Eq. (2) defines a KL divergence between the prediction and the similarity vector d_i, but d_i comes from cosine similarities and is not normalized to a probability distribution. The loss is not a proper KL divergence as written. The authors need to state the normalization or drop the KL label. That's a correctness point, minor fix but important.\n\nAlso: no variance or significance tests anywhere, and N=6 is chosen partly by looking at test-clean (they claim no significant difference between 4 and 6 on dev-clean, but then pick 6). Standard practice criticism, not fatal.\n\nBottom line: this paper deserves a serious referee. The engineering is plausible, the writing is clear, the limitations section is honest, and the result is worth verifying. But it needs full-scale ablations, a spelled-out loss, and variance reporting before the claim is credible. I'd send it to review, not desk-reject.\n\nYes, I'd cite it if I worked on SSL speech, and it's a good reading-group paper for a methods comparison.","headline":"Solid engineering result with a real attribution gap: full-scale WER gains are promising, but the paper does not isolate which new component drives them.","tokens_in":8538,"tokens_out":2525,"would_cite":true,"duration_ms":22385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining multiple quantization codebooks, a KL-divergence auxiliary loss, and cluster-specific codebooks derived from low-level audio features improves BEST-RQ self-supervised speech pre-training, cutting word error…","keywords":["self-supervised learning","speech recognition","BEST-RQ","random-projection quantizer","multi-codebook","KL-divergence regularization","cluster-specific codebooks","LibriSpeech"],"falsifier":"Compute the cosine-similarity vectors d_i from the random-projection quantizer on a few LibriSpeech batches and check whether each vector's entries are nonnegative and sum to 1. If the sums are not always 1, the KL term in Eq. (2) is not a well-defined divergence, and an ablation that replaces it with a properly normalized distribution would settle whether the auxiliary-loss gains are real.","tokens_in":7455,"feed_emoji":"🎙️","tokens_out":6749,"duration_ms":54122,"temperature":0.7,"pith_summary":"This paper argues that the BEST-RQ recipe for self-supervised speech pre-training can be pushed to substantially lower word error rates without changing the downstream model size. The authors replace the single random-projection codebook with several codebooks, add a KL-divergence term that pushes the model's output distributions toward the quantizer's similarity scores, and assign each utterance a codebook chosen by clustering low-level acoustic features. In limited train-100 experiments, the multi-codebook step alone improves test-clean WER by 11.2% relative, and the KL loss adds a further gain; on the full 960-hour LibriSpeech pre-training, the combined recipe improves WER by up to 23.8% on test-clean and 30.6% on test-other. It also reports faster convergence and more stable validation loss during pre-training. If correct, this gives a simple, parameter-free quantizer path to better self-supervised speech models.","feed_headline":"Multi-codebook BEST-RQ cuts speech-recognition errors by 30.6%","feed_subtitle":"Codebook diversity, a KL-regularizing loss, and cluster-specific targets also speed and stabilize pre-training.","key_machinery":"The core mechanism is the random-projection quantizer (RQ), a fixed, randomly initialized projection matrix and codebook that maps each masked speech frame to the index of its nearest codebook vector, producing the training target. The paper keeps that quantizer frozen but modifies the objective and target structure in three ways: N parallel output heads and N codebooks whose losses are averaged; a KL-divergence term comparing each output distribution to the quantizer's cosine-similarity vector, scaled by a weight; and utterance-level clustering of low-level features so that each cluster gets its own codebook, with the matching codebook's loss weighted at least twice as heavily as the others. The argument is carried by the interaction of these three modifications with the fixed quantizer, which remains parameter-free and collapse-free.","core_discovery":"On the paper's own terms, the central discovery is that BEST-RQ's fixed random-projection quantizer is not a bottleneck: by using six independent codebooks with per-codebook output heads, by training with a weighted sum of cross-entropy and a KL-divergence regularizer against the quantizer's cosine-similarity vectors, and by specializing codebooks to utterance clusters from MFCC, spectral-contrast, spectral roll-off, and zero-crossing-rate features, the model learns a better representation for speech recognition. The authors report that this setup beats the single-codebook BEST-RQ baseline on every LibriSpeech dev and test set, with the largest relative WER reduction appearing on test-other (30.6%). They also observe that the modified pre-training reaches the baseline's best validation loss in about a third of the steps, and that the improved encoder transfers to fine-tuning at no extra model size.","pith_inferences":["Because the KL term only compares the model's output distribution to the quantizer's similarity scores, it should transfer to other self-supervised speech objectives; testing it on those objectives would show whether the gain is tied to BEST-RQ or to the regularizer itself.","Explicitly normalizing the similarity vectors before computing the KL term (for example with a softmax) would make the auxiliary loss a true divergence; a controlled ablation with and without normalization would isolate whether any improvement comes from the divergence or just from the extra smoothing.","The utterance-level clustering could be refined: instead of hand-chosen low-level features, a jointly learned clustering or clustering on deeper embeddings might give larger gains, and the same idea could apply to noise conditions or speakers.","If the preliminary pattern holds, the multi-codebook component likely contributes the largest share of the full-scale gain, but the paper does not demonstrate that breakdown at 960 hours."],"forward_implications":["With the full 960-hour LibriSpeech pre-training, the optimized recipe improves WER by 23.8% relative on test-clean and 30.6% on test-other compared with the single-codebook baseline.","In the train-100 preliminary setup, using six codebooks instead of one reduces test-clean WER by 11.2% relative.","Combining cross-entropy with the KL-divergence auxiliary loss improves WER over cross-entropy alone (7.8% on dev-clean and 8.4% on test-clean with one codebook; 4.6% and 4.5% with six) and reaches the baseline's best validation loss in roughly a third of the training steps.","Cluster-specific codebooks on top of the six-codebook CE+KL setup give a further WER reduction (to 16.13 dev-clean / 16.28 test-clean in the preliminary setup) and stabilize validation loss.","The extra pre-training complexity disappears at fine-tuning: the downstream encoder is the same size as the baseline's, so the WER gains do not cost inference compute."],"supporting_citations":[{"why":"Introduces BEST-RQ, the random-projection quantizer baseline that the paper optimizes.","marker":"[7]"},{"why":"Describes the multi-codebook extension and the use of cosine similarity for target selection that the paper adopts and extends.","marker":"[10]"},{"why":"Provides the open implementation and experimental setup used as the baseline and reference for the conformer encoder.","marker":"[13]"},{"why":"Defines the conformer block, the encoder architecture used for pre-training and fine-tuning.","marker":"[9]"},{"why":"Supplies the LibriSpeech corpus used for all pre-training and fine-tuning experiments.","marker":"[16]"},{"why":"Defines the MFCC features used in utterance-level clustering for cluster-specific codebooks.","marker":"[11]"},{"why":"Defines spectral contrast, one of the low-level features used to form utterance clusters.","marker":"[12]"}],"fun_headline_variants":["6-codebook BEST-RQ cuts WER 30.6% on LibriSpeech test-other","Multi-codebook BEST-RQ: 30.6% lower WER on speech recognition","BEST-RQ with 6 codebooks and KL loss: 30.6% WER drop","BEST-RQ with 6 codebooks trains 3x faster, cuts WER 30.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The auxiliary KL loss assumes that the quantizer's per-frame similarity scores form a proper probability distribution (nonnegative and summing to 1). The paper does not describe a normalization step for those cosine similarities, so if they do not sum to 1 the reported gains from the auxiliary loss rest on an invalid objective.","fun_headline_variants_meta":{"raw":{"variants":["6-codebook BEST-RQ cuts WER 30.6% on LibriSpeech test-other","Multi-codebook BEST-RQ: 30.6% lower WER on speech recognition","BEST-RQ with 6 codebooks and KL loss: 30.6% WER drop","BEST-RQ with 6 codebooks trains 3x faster, cuts WER 30.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001482,"raw_usage":{"total_tokens":5942,"prompt_tokens":922,"completion_tokens":5020,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":4914}},"tokens_in":538,"tokens_out":5020,"duration_ms":30989,"temperature":1.0,"reasoning_tokens":4914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:43:07.398783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the cosine-similarity vectors d_i from the random-projection quantizer on a few LibriSpeech batches and check whether each vector's entries are nonnegative and sum to 1. If the sums are not always 1, the KL term in Eq. (2) is not a well-defined divergence, and an ablation that replaces it with a properly normalized distribution would settle whether the auxiliary-loss gains are real.","supporting_citations":[{"cited_title":"Self-supervised learning with random-projection quantizer for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Introduces BEST-RQ, the random-projection quantizer baseline that the paper optimizes."},{"cited_title":"Open imple- mentation and study of best-rq for speech processing,","cited_arxiv_id":null,"evidence_quote":"Provides the open implementation and experimental setup used as the baseline and reference for the conformer encoder."},{"cited_title":"Conformer: Convolution- augmented transformer for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the conformer block, the encoder architecture used for pre-training and fine-tuning."},{"cited_title":"Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,","cited_arxiv_id":null,"evidence_quote":"Defines the MFCC features used in utterance-level clustering for cluster-specific codebooks."},{"cited_title":"Music type classification by spectral contrast feature,","cited_arxiv_id":null,"evidence_quote":"Defines spectral contrast, one of the low-level features used to form utterance clusters."}],"review_version":1}