{"id":"a1a8511e-28c0-4470-ac5c-876922227fae","arxiv_id":"2505.16207","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Applying differentiable k-means to token-based ASR lets the tokenizer, the speech encoder, and the recognizer be optimized together, improving WER and making tokens more phoneme-like.","lead":"A speech recognition system that converts sound into discrete tokens usually does this in a separate, fixed step. This paper makes the tokenization step trainable end-to-end with the recognizer, and reports better word error rates and more phoneme-like units.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies whether the ASR is trained on hard argmax tokens or on the soft Gumbel-Softmax vectors from Eq.","rationale":"I read the paper as a genuine attempt to make discrete tokens trainable end-to-end for ASR, and the cross-corpus results in Tables 1, 2, and 5 are positive evidence that the overall recipe helps. My concern is not with the existence of an improvement but with what exactly is being measured. The full-finetune condition updates WavLM and the centroids; any differentiable tokenizer (VQ-VAE, Gumbel-VQ, etc.) would likely also show gains. What distinguishes this submission is the claim that the discrete tokens themselves are optimized. That claim rests on the discreteness of the forward path, which the manuscript never pins down. Eq. (2) is a relaxation; Eq. (3) is switched off; the temperature schedule is described only as 'gradually decreased.' The reader's weakest assumption about assignment sharpness and centroid collapse is exactly the right area, and my concern narrows it: it is not just that assignments might stay soft, but that the training objective described in the paper is compatible with a fully soft input, in which case the method is not discrete-token ASR at all during training. The PNMI drop in Table 3 bolsters the suspicion that the 'phonetic purity' language outruns the evidence. I would not reject the paper on this basis—the fix is a one-page clarification plus a hard-forward ablation—but the central claim should be accepted only after that check, so I keep the reader's conditional verdict unchanged.","tokens_in":1124,"tokens_out":1127,"duration_ms":80568,"concrete_test":"Retrain the single-layer k=100 LibriSpeech-100 configuration of Table 1 with an explicit hard forward pass: tokens are argmax h_i at every step, gradients to θssl and M are obtained via straight-through through the hard selection using the Eq. (1) logits, α=0, and the same reported training schedule. Record the final τ and measure, at checkpoints and at test time, the fraction of frames whose soft assignment is within 0.01 of a one-hot vector. If WER stays near 5.4/9.2 while the hard-assignment fraction is high, the discrete-token claim is supported; if WER degrades toward the 7.6/13.2 baseline or hard-assignment fraction remains low, the published gains were carried by soft continuous training. A complementary check is to compare against VQ-VAE with the same full fine-tuning to see whether differentiable k-means itself, rather than generic differentiable discretization, is what matters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that differentiable k-means produces an optimized *discrete* tokenizer. The forward pass is never defined. Eq. (2) outputs a soft probability vector h_i, with τ initialized at 2.0 and only 'gradually decreased' during training; no final τ, schedule, or hard/soft consumption rule is given. If the ASR encoder consumes h_i (e.g., as a weighted mixture of token embeddings), then training is a continuous-input system with a learned soft codebook. The WER improvements over the frozen-k-means baseline could then come from the extra flexibility of soft mixtures and from WavLM full fine-tuning, not from better discrete units. If instead the encoder consumes argmax h_i, gradients through the discrete selection are zero almost everywhere, so a straight-through estimator or similar proxy is silently required, and the Gumbel-Softmax likelihood in Eq. (2) is not what is being optimized. Removing Lkm (α=0) removes the only loss that ties centroids to the SSL features, so assignment sharpness is guaranteed only by the undocumented annealing schedule. This is not a side detail: it determines whether the method is 'differentiable k-means for discrete-token ASR' or simply soft-input ASR with a learned codebook. The phonetic-purity claim is also strained in the same direction: single-layer PNMI drops from 0.4754 to 0.4561 under full finetune at k=100 (Table 3), so the evidence that units became more phoneme-like is based on MTER/TSL and multi-layer results only.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the standard non-differentiable k-means tokenizer in discrete-token ASR with a differentiable variant built on the concrete/Gumbel-Softmax k-means of [15]. The tokenizer layer computes soft assignments (Eq. (1)) and Gumbel-Softmax samples (Eq. (2)), through which the ASR cross-entropy loss (and optionally a k-means reconstruction loss, Eq. (3)) can backpropagate into the SSL encoder parameters θ_ssl, the centroids M, and the layer-weighting vector w (Eqs. (4)-(6)). Experiments use WavLM-large as the SSL model and a joint CTC/attention encoder-decoder ASR in ESPnet, comparing three update regimes: ASR-only (baseline), ASR+centroids (freeze SSL), and ASR+centroids+SSL (full finetune), with α=0 so that the reconstruction loss is disabled. On LibriSpeech-100, full finetune improves WER from 7.6/13.2 to 5.4/9.2 (k=100, single layer) and from 8.4/18.3 to 5.3/8.2 (k=100, multi-layer), approaching the continuous-feature result of 3.7/6.6. Token analyses on TIMIT show lower TSL and MTER (interpreted as more phoneme-like, speaker-robust tokens) and higher NQE, with multi-layer tokens also improving PNMI. Resynthesis experiments with Tacotron2 show better content preservation (OOD WER 31.4 to 6.1 at k=100 multi-layer), and gains are reported on LibriSpeech-960 and ML-SUPERB.","tokens_in":9649,"tokens_out":18074,"duration_ms":117169,"significance":"If the mechanism is as claimed, the paper makes a useful step: it shows how to carry gradients from an ASR loss through a discrete-equivalent tokenization into a large SSL encoder, with learned layer weighting, and it provides evidence that jointly optimized tokens approach continuous-feature performance while remaining discrete, compact, and usable for generation. The evaluation is broad by the standards of the area, covering held-out test sets (LibriSpeech test-clean/other, TIMIT, LJSpeech/VCTK, ML-SUPERB), multiple metrics (WER, PNMI, NQE, TSL, MTER, MCD, F0 RMSE, UTMOS), and a clean baseline/freeze/full ablation that separates which parameters are updated. The paper builds on the public ESPnet toolkit and standard checkpoints, so the experiments are in principle reproducible, and the headline gains at k=100 are substantial and practically meaningful if they hold. The main issues requiring revision are methodological rather than existential: the training-time forward pass through the tokenizer is underspecified (hard vs.","major_comments":[{"comment":"The forward computation of the tokenizer is never defined, and this is load-bearing for the central claim of training a discrete token-based ASR. Eq. (2) outputs a continuous vector h_i, and the paper never states whether the ASR encoder consumes argmax h_i (token indices) or the soft vector h_i (e.g., an expectation over token embeddings) during training and inference. The sentence in §2.2 that Gumbel-Softmax \"produces discrete outputs, i.e., arg max h_i\" conflates the finite-temperature relaxation with its limit; at the initial temperature τ=2.0 the outputs are not one-hot. If the ASR consumes soft mixtures, the system is trained as a continuous-input model with a learned soft codebook, and the WER gains could come from the added flexibility of soft mixtures rather than from better discrete units; if the ASR consumes argmax indices, a straight-through estimator is silently required and Eq. (2) is not the distribution actually differentiated by the ASR loss. Section 3.1 reports only that τ was initialized to 2.0 and \"gradually decreased,\" with no schedule, no final value, and no rule for producing the discrete tokens used at evaluation time (needed for the TIMIT clustering metrics and for the deduplicated token sequences used in resynthesis). Please specify the training- and inference-time forward passes, the temperature schedule, and the sampling/argmax rule for each experiment, and include some analysis of assignment sharpness (e.g., the entropy of h_i) to confirm that the trained tokenizer produces near-discrete assignments at the operating point.","section":"§2.2, Eqs. (1)–(2); §3.1."},{"comment":"The paper does not isolate the contribution of the differentiable k-means formulation from the contribution of SSL fine-tuning. With α=0 (§3.1), the k-means reconstruction loss of Eq. (3) is never used, so the tokenizer is a distance-based soft quantizer driven solely by the ASR loss. Table 1 shows that updating only the centroids (freeze SSL) yields no gain over baseline at any cluster size (e.g., 7.3/12.6 vs. 7.6/13.2 at k=100), while essentially all of the gains appear when θ_ssl is fine-tuned. Given the claim that tokenization is \"jointly optimized,\" the present experiments do not show whether the specific k-means structure of Eq. (1) matters for the reported gains. Please add (i) an ablation with α>0, (ii) a full-finetune condition with the centroids M frozen, and (iii) a comparison against an alternative differentiable quantizer (e.g., straight-through VQ-VAE-style codebook or a linear soft codebook) under otherwise identical SSL fine-tuning.","section":"§3.1 (α=0), §3.2, Table 1."},{"comment":"All reported results appear to be single runs without error bars or significance tests, yet the text makes equality and inequality claims that require them. In particular, §3.2 states that \"there was no significant difference in performance between cluster sizes of 500, 1000, and 2000,\" and Table 1 contains baseline/freeze SSL differences as small as 0.1–0.4 WER, which is within typical run-to-run variation for these settings. For the headline tables, please report variance over multiple training seeds (or an equivalent estimate) and either significance tests or reworded claims for the \"no difference\" statements.","section":"§3.2; Tables 1–5."}],"minor_comments":[{"comment":"At k=100, single-layer full finetune decreases PNMI from 0.4754 to 0.4561, yet the text describes the single-layer values as \"almost the same\"; please state this decrease explicitly and discuss whether it qualifies the phonetic-purity claim, which is currently carried by the multi-layer PNMI and by the TSL/MTER results.","section":"§3.4, Table 3."},{"comment":"In Table 4 at k=2000, the OOD resynthesis WER slightly worsens under single-layer full finetune (14.0 to 14.9) and the ID WER is flat (2.7 to 2.6); the discussion of the \"most balanced\" configuration highlights only the k=100 rows and should also comment on this regression.","section":"§3.5, Table 4."},{"comment":"The single-layer full-finetune condition updates θ_ssl for 60 epochs, whereas the multi-layer condition updates it only for the last 30 epochs, so the comparison between single- and multi-layer results also varies the training schedule; the stated rationale for the 30-epoch choice (\"w was effectively tuned\") is asserted without supporting ablation data.","section":"§3.3."},{"comment":"The claim that multi-layer full finetuning \"can condense this information into a single layer\" is not substantiated by any report of the learned layer weights w; showing w, or its evolution over training, would make the claim verifiable.","section":"§3.3, §3.5, Eq. (5)."},{"comment":"Several experimental definitions are incomplete: for MTER it is not stated whether token sequences are deduplicated before computing pairwise TER (TSL is defined on deduplicated sequences, so the relation between the two metrics should be clarified); the ML-SUPERB evaluation in §3.6 is described in one sentence with no details of the subset, recipe, or definition of \"normal/few-shot\"; and σ²=1.0 in Eq. (1) is given no sensitivity analysis despite high-dimensional WavLM features.","section":"§3.1, §3.4, §3.6."},{"comment":"Typos and formatting issues to correct: \"LibriSpecch-960\" in the Table 5 caption, \"freeeze SSL\" in §3.3, \"custer sizes\" in §3.4, the inline \"baseline) θasr, freeze SSL) θasr, w, and M, or full finetune) ...\" enumeration in §3.3, and \"pre-trained on the train clean 100\" in §3.2.","section":"Throughout."}],"recommendation":"major_revision","confidential_remarks":"An editorial point worth checking: the paper does not compare its reimplemented k-means baselines against the numbers reported in [11] and [12] for the same configurations, so a reader cannot verify that the baseline is on par with previously published discrete-token ASR results. The citations to works from the same research group ([11, 12, 17, 18]) are appropriate given the task, and I do not see a circularity problem because all headline claims are measured on held-out test sets. The novelty over those works is incremental, and the five-page format leaves real gaps in the method description (temperature schedule, hard/soft forward pass, seed variance); if the target venue is a journal, those gaps should be addressed in the expanded version rather than by simply moving the experimental setup text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper gets the right headline result—fine-tuning the tokenizer and the SSL encoder through a Gumbel-Softmax relaxation of k-means clearly improves ASR WER over frozen tokenization—but it leaves the central mechanism underspecified. The stress-test note is on target.\n\nWhat's new and good: applying concrete k-means [15] to SSL-based discrete-token ASR, together with joint SSL fine-tuning and learned layer weights, is a sensible extension. The WER tables support the improvement: k=100 single-layer goes from 7.6/13.2 to 5.4/9.2, and multi-layer from 8.4/18.3 to 5.3/8.2, approaching the continuous-feature ceiling. The token analysis using MTER/TSL is useful, and the resynthesis results—OOD WER dropping from 31.4 to 6.1—are genuinely striking. The paper is honest about the PNMI drop in the single-layer case and the expected NQE increase.\n\nSoft spots, in order of size:\n\n1. The forward pass is never defined. Eq. (2) gives a soft vector h_i, and tau is annealed from 2.0 with no final value or schedule specified. If the ASR encoder consumes h_i, training is continuous and the improvements could come from extra flexibility plus WavLM fine-tuning, not from better discrete units. If it consumes argmax h_i, gradients vanish without a straight-through estimator. The paper needs to say which.\n\n2. alpha=0 means the k-means reconstruction loss is never used. The centroids become learned embeddings, not cluster centers that reconstruct SSL features. The method is closer to differentiable soft-assignment with a codebook than to differentiable k-means. No ablation of alpha is provided.\n\n3. No error bars; every result is a single run. For a paper with gains this large, that is less critical, but still a gap.\n\n4. The phonetic-purity claim is overstated: single-layer PNMI drops at k=100 (0.4754 -> 0.4561). The claim rests on multi-layer results and should be scaled back.\n\n5. No comparison to VQ-VAE or other differentiable discretizations, so the specific benefit of the k-means formulation is not isolated. Code and hyperparameters (tau schedule, etc.) would help.\n\nWho is this for? People working on discrete speech units for ASR or speech-LLMs. The empirical findings are valuable even if the mechanism is murky. I would send this to referees rather than desk-reject; the current version is not ready to accept without revision.","headline":"A promising but underspecified application of concrete k-means to discrete-token ASR: the WER gains are real and worth refereeing, but the paper never clarifies whether training is actually discrete, and alpha=0 leaves the k-means reconstruction objective out entirely.","tokens_in":10206,"tokens_out":6206,"would_cite":true,"duration_ms":47587,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Integrating differentiable k-means into discrete-token ASR lets the tokenizer, SSL encoder, and layer weights be trained jointly on the ASR objective, improving accuracy and phonetic purity.","keywords":["differentiable k-means","discrete speech tokens","self-supervised learning","automatic speech recognition","tokenization","layer weighting","speech resynthesis","phonetic information"],"falsifier":"During the full-finetune run on LibriSpeech-100 with 100 clusters, compute the mean entropy of the Gumbel-Softmax assignment distribution per frame and the fraction of centroids that receive at least one argmax assignment at the end of training; if the distribution is near-uniform or many centroids are dead, the reported gains cannot be attributed to the differentiable clustering mechanism.","tokens_in":9060,"feed_emoji":"🎙️","tokens_out":7003,"duration_ms":52641,"temperature":0.7,"pith_summary":"This paper tries to establish that discrete speech tokens for ASR should be trained, not just clustered: replacing ordinary arg-min k-means with a differentiable version lets the tokenizer, the SSL encoder, and the layer-fusion weights all be optimized jointly against the ASR loss. On LibriSpeech-100, word error rate improves at every cluster size tested, with the biggest relative gains at 100 clusters (7.6/13.2 down to 5.4/9.2 for a single SSL layer). The learned tokens also carry cleaner phonetic information, measured by lower mean token error rate across speakers and shorter deduplicated sequences, and they improve resynthesis intelligibility. A careful reader would care because this points toward compact, task-specific discrete units that keep the efficiency of tokenization while narrowing the accuracy gap with continuous SSL features.","feed_headline":"Trainable k-means cuts LibriSpeech word error rate by a third","feed_subtitle":"Discrete speech tokens shaped by the ASR objective also become more phonetic and help resynthesis.","key_machinery":"The load-bearing object is the differentiable k-means operator: soft cluster-assignment probabilities $p(j \\mid s_i) = \\exp(-\\sigma^2 \\|s_i - \\mu_j\\|^2) \\,/\\, \\sum_c \\exp(-\\sigma^2 \\|s_i - \\mu_c\\|^2)$, sampled with Gumbel-Softmax at temperature $\\tau$, so that the discrete argmax has a continuous surrogate for gradients. The total loss $L = L_{\\mathrm{ASR}}(\\mathrm{ASR}(\\mathrm{DiffKM}(\\mathrm{SSL}(X)), M), \\theta_{\\mathrm{asr}}) + \\alpha L_{\\mathrm{km}}$, with $\\alpha=0$ in all reported experiments, routes the ASR error through the assignments to update the SSL parameters $\\theta_{\\mathrm{ssl}}$, centroids $M$, and in the multi-layer variant the weights $w$ over SSL layers. The k-means loss $L_{\\mathrm{km}}$ is defined but deliberately unused, which makes the experiment a test of whether the ASR signal alone can shape the clusters.","core_discovery":"The central claim is that replacing standard arg-min k-means tokenization with a Gumbel-Softmax differentiable k-means operator lets the ASR objective backpropagate through the discrete assignment into the SSL encoder, the cluster centroids, and the layer weights, so the token vocabulary itself is optimized for recognition rather than for reconstruction fidelity. With the k-means reconstruction loss weighted at zero, full fine-tuning of the SSL model improves LibriSpeech-100 WER from 7.6/13.2 to 5.4/9.2 (single layer, 100 clusters) and from 8.4/18.3 to 5.3/8.2 (multi-layer, 100 clusters). The resulting tokens have lower mean token error rate across speakers who say the same sentence, shorter deduplicated token sequences, and better out-of-domain resynthesis intelligibility, which the paper reads as evidence that the tokens converge toward phoneme-level content.","pith_inferences":["A natural extension the authors mention only in passing: the same differentiable tokenizer could be driven by other objectives, such as speech translation or TTS intelligibility, to shape tokens for those tasks.","Because freezing the SSL encoder while updating only centroids gives little gain, the paper's comparison suggests most of the benefit comes from fine-tuning the encoder itself; a straight-through hard-assignment baseline would isolate the contribution of the soft differentiable mechanism.","Setting $\\alpha=0$ works better than including a reconstruction loss, which implies that clustering objectives can actively hurt downstream-shaped tokenization; sweeping $\\alpha$ would make this tradeoff quantitative.","Task-optimized small vocabularies could be especially useful for LLM-based speech models, where shorter token sequences directly reduce memory and decoding cost."],"forward_implications":["At 100 clusters, full fine-tuning brings single-layer WER from 7.6/13.2 to 5.4/9.2, and performance becomes nearly flat from 500 to 2000 clusters, so compact token vocabularies stop costing accuracy.","Multi-layer fusion with full fine-tuning reaches 5.3/8.2 WER, approaching the continuous-feature system (3.7/6.7), so discrete tokens narrow the gap with continuous SSL features.","TIMIT evaluation shows lower mean token error rate (28.8% vs 42.2% at 100 clusters) and shorter token sequences, indicating increased robustness to speaker and pronunciation variation.","Resynthesis with the optimized 100-cluster tokens improves out-of-domain WER from 31.4% to 6.1%, suggesting the units retain linguistic content needed for generation.","Improvements also hold on LibriSpeech-960 and the multilingual ML-SUPERB benchmark, so the effect is not limited to the small English corpus."],"supporting_citations":[{"why":"Supplies the differentiable k-means formulation (soft assignment plus Gumbel-Softmax sampling) that the paper adapts from autoencoders to ASR.","marker":"[15]"},{"why":"Provides the Gumbel-Softmax reparameterization that makes the discrete cluster assignments differentiable.","marker":"[23]"},{"why":"Defines the discrete-token ASR baseline the paper extends and compares against.","marker":"[11]"},{"why":"Contributes the experimental configuration, including the 21st-layer feature choice, used for single-layer experiments.","marker":"[12]"},{"why":"Gives the joint CTC/attention encoder-decoder architecture used as the ASR model.","marker":"[19]"},{"why":"WavLM-large is the upstream SSL model whose parameters are fine-tuned.","marker":"[4]"},{"why":"Motivates the multi-layer weighting strategy for discrete tokens that the method optimizes jointly.","marker":"[13]"}],"fun_headline_variants":["Differentiable k-means tunes speech tokens for ASR, cutting WER by a third","ASR-optimized discrete tokens via differentiable k-means beat clustering","Jointly training k-means and ASR yields better tokens and lower WER","Differentiable k-means tailors speech tokens to ASR, improving accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, with the reconstruction loss switched off ($\\alpha=0$) and $\\sigma^2$ fixed at 1.0, the Gumbel-Softmax soft assignments still pass gradients that push the SSL encoder and centroids toward useful phonetic structure instead of collapsing or staying too uniform.","fun_headline_variants_meta":{"raw":{"variants":["Differentiable k-means tunes speech tokens for ASR, cutting WER by a third","ASR-optimized discrete tokens via differentiable k-means beat clustering","Jointly training k-means and ASR yields better tokens and lower WER","Differentiable k-means tailors speech tokens to ASR, improving accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3314,"prompt_tokens":880,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2348}},"tokens_in":496,"tokens_out":2434,"duration_ms":15348,"temperature":1.0,"reasoning_tokens":2348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:05:05.685990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"During the full-finetune run on LibriSpeech-100 with 100 clusters, compute the mean entropy of the Gumbel-Softmax assignment distribution per frame and the fraction of centroids that receive at least one argmax assignment at the end of training; if the distribution is near-uniform or many centroids are dead, the reported gains cannot be attributed to the differentiable clustering mechanism.","supporting_citations":[{"cited_title":"WavLM: Large-scale self-supervised pre-training for full stack speech process- ing,","cited_arxiv_id":null,"evidence_quote":"Defines the discrete-token ASR baseline the paper extends and compares against."},{"cited_title":"Speechgpt: Empowering large language models with intrinsic cross-modal conversational abili- ties,","cited_arxiv_id":null,"evidence_quote":"Supplies the differentiable k-means formulation (soft assignment plus Gumbel-Softmax sampling) that the paper adapts from autoencoders to ASR."},{"cited_title":"Neural discrete represen- tation learning,","cited_arxiv_id":null,"evidence_quote":"Provides the Gumbel-Softmax reparameterization that makes the discrete cluster assignments differentiable."},{"cited_title":"w2v-BERT: Combining contrastive learning and masked language modeling for self- supervised speech pre-training,","cited_arxiv_id":null,"evidence_quote":"Contributes the experimental configuration, including the 21st-layer feature choice, used for single-layer experiments."},{"cited_title":"Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,","cited_arxiv_id":null,"evidence_quote":"Gives the joint CTC/attention encoder-decoder architecture used as the ASR model."},{"cited_title":"Since SSL fea- ture spaces are changed by finetuning, they are divided by the averaged L2 norm for normalization","cited_arxiv_id":null,"evidence_quote":"WavLM-large is the upstream SSL model whose parameters are fine-tuned."},{"cited_title":"SUPERB: Speech processing universal performance benchmark,","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-layer weighting strategy for discrete tokens that the method optimizes jointly."}],"review_version":1}