{"id":"79457311-2a7d-4012-a6ad-97ea9e5c10a2","arxiv_id":"2506.00843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HASRD factorizes SSL speech representations into a first semantic codebook and residual acoustic codebooks, reporting improved ASR and reconstruction at 3.1 kbps versus SpeechTokenizer's 6.0 kbps.","lead":"HASRD splits speech into two kinds of discrete tokens: semantic tokens from a self-supervised model and acoustic tokens from the leftover signal. The authors report better speech recognition and audio reconstruction than SpeechTokenizer at about half the bitrate, but the headline WER gain depends on which comparison you choose.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's '44% relative WER improvement' is not supported for quantized full tokens: Table 4 shows HASRD combined tokens (24.2 WER) are worse than SpeechTokenizer's 23.0, and the 44% figure only matches continuous latent features.","rationale":"The reader correctly flagged the WER comparison as ambiguous and noted Table 4, but identified Eq. (2)'s residual assumption as the weakest assumption. I see the more load-bearing issue as the evaluation protocol itself: the 44% relative WER improvement cited in the abstract is not reproducible from the quantized rows of Table 3, and the full-token WER in Table 4 (24.2) is actually worse than SpeechTokenizer's reported quantized WER (23.0). This directly undermines the strongest claim in the paper, independent of whether the residual in Eq. (2) is clean. The reconstruction results at 3.1 kbps (mel distance 0.64 vs 0.76, ViSQOL 4.50 vs 4.26) are credible and give the paper real value, and the encoder efficiency gains are useful. But the headline WER statement, which is the primary hook of the abstract, appears to compare semantic-only HASRD tokens against SpeechTokenizer's full token sequence. A clarifying re-evaluation with matched token sets could change the central claim substantially. I therefore agree with the reader's CONDITIONAL verdict rather than moving to full acceptance; my concern reinforces the existing conditions rather than overturning them.","tokens_in":8930,"tokens_out":7299,"duration_ms":68031,"concrete_test":"Run the ASR evaluation with identical training data and the same BLSTM-CTC recipe on: (a) HASRD full token sequence (first semantic codebook plus all eight acoustic RVQ codebooks), (b) HASRD semantic-only, (c) SpeechTokenizer full token sequence, and (d) SpeechTokenizer first-codebook-only. Then recompute relative WER improvements. If HASRD full-token WER is not below SpeechTokenizer full-token WER, or if (a) is worse than (b), the abstract's '44% relative WER improvement' must be withdrawn or restricted to continuous latent features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—'Compared to SpeechTokenizer, HASRD achieves a 44% relative WER improvement'—is not supported by the reported numbers when comparing the token sequences a spoken language model would actually consume. In Table 3, the only 44%-class relative improvement is in the continuous latent row: SpeechTokenizer 21.5 → HASRD (BestRQ+) 12.0 = 44.2% relative. The quantized-row comparison is only 8.7% relative (21.0 vs 23.0). More seriously, Table 4 reports HASRD's combined semantic+acoustic token WER as 24.2, which is worse than SpeechTokenizer's quantized WER of 23.0. Table 3's HASRD quantized value (21.0) matches Table 4's semantic-only value, suggesting the headline WER gain is obtained by evaluating only the first HASRD codebook while SpeechTokenizer is evaluated on its full (semantic+acoustic) output. Unless the paper specifies and equalizes the token sets used for ASR, the headline improvement is an artifact of an asymmetric evaluation, and the '44%' figure itself appears to come from continuous features rather than the discrete tokens that matter for SLMs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HASRD, a two-stage framework that factorizes SSL speech representations into discrete semantic and acoustic tokens. In the first stage, a CNN-Conformer encoder is pre-trained with a BestRQ-style masked language modeling objective using random-projection quantization; in the second stage, the frozen encoder's layer representations are combined with learned weights, a k-means quantized semantic layer becomes the first RVQ codebook, and subsequent RVQ codebooks encode the acoustic residual via a reconstruction objective. The authors report improved computational efficiency of the encoder, better reconstruction quality than SpeechTokenizer at lower bitrate, and improved ASR WER, along with an analysis of acoustic/semantic disentanglement using speaker similarity and ASR.","tokens_in":9211,"tokens_out":4231,"duration_ms":39123,"significance":"If fully substantiated, the method would be a valuable contribution to spoken language model tokenization: it aims to preserve semantic performance while adding acoustic fidelity, with interpretable codebook roles and a 2x bitrate reduction over SpeechTokenizer. The paper's strengths include a systematic set of ablations on LibriSpeech (100h and 960h), evaluation with external ASR, ViSQOL, and speaker-verification metrics, and a clear visual and quantitative analysis of layer-wise contributions to acoustic versus semantic objectives. The efficiency improvements to the DAC-style encoder are believable and potentially useful independently. However, the headline quantitative claim about WER improvement is not supported by the numbers as reported for discrete tokens, and the full token sequence behaves differently than the semantic-only token sequence, which is central to the paper's premise for SLM use.","major_comments":[{"comment":"The abstract's claim of a 44% relative WER improvement over SpeechTokenizer is not supported by the quantized token comparison. In Table 3, the quantized rows show HASRD (BestRQ+) at 21.0 WER versus SpeechTokenizer at 23.0, which is a 8.7% relative improvement; the 44% figure matches only the continuous latent row (12.0 vs. 21.5). Since spoken language models consume discrete tokens, the abstract should report the quantized comparison or explicitly state that the 44% figure refers to continuous latent features.","section":"Abstract / Table 3"},{"comment":"Table 4 reports that the combined semantic+acoustic token sequence yields WER 24.2, which is worse than both HASRD's semantic-only WER of 21.0 and SpeechTokenizer's full-token WER of 23.0. This directly weakens the claim that HASRD improves ASR while adding acoustic fidelity for the token set an SLM would actually consume. Please report ASR on the exact token set used by an SLM (e.g., all codebooks) and analyze why adding acoustic codebooks degrades WER relative to semantic-only tokens.","section":"Table 4 / Section 4.3"},{"comment":"The acoustic residual Ha = HW - H_s assumes that subtracting the k-means quantized semantic term removes essentially all semantic content. The results in Table 4, where acoustic-only tokens achieve 70.2% WER, indicate that substantial semantic information remains in the acoustic tokens, so the factorization is not clean. Provide a quantitative leakage analysis, for example ASR on acoustic-only tokens with a controlled training setup or a probing task, and discuss the consequence for the disentanglement claim.","section":"Section 2.2, Eq. (2)"},{"comment":"The evaluation protocol is not fully symmetric across models. The Table 3 caption states that for HASRD only the encoder is used for ASR, but it does not specify which token set is used for the quantized rows of SpeechTokenizer or for HASRD. If the HASRD quantized row uses only the first codebook while SpeechTokenizer is evaluated on its full RVQ output, the comparison is asymmetric and the reported improvement is not meaningful for downstream SLM use. Please specify the token set used for every row and, if necessary, retrain the ASR model on matching token sets.","section":"Table 3 and Section 3 (Evaluation)"}],"minor_comments":[{"comment":"The text says 'Increasing the attention dimension from 516 to 768' but the encoder description in Section 3 states an attention dimension of 512; also 'HASARD' appears to be a typo for 'HASRD'.","section":"Section 4.2, text after Table 3"},{"comment":"The table reports inference speedups of 2x and 3x and a GPU memory reduction of 25%, but the text only says 'inference speed also improves by 2x on CPU'; please clarify the hardware and measurement conditions for all speedup numbers.","section":"Table 1"},{"comment":"The notation for the quantized semantic representation is inconsistent: H_s is used for the k-means output in the text, while Eq. (2) uses a hat notation (H^s); please unify the notation.","section":"Section 2.2, Eq. (2)"},{"comment":"The statement that the semantic layer index n_s = 8 was selected as the best performing layer for ASR is a legitimate design choice, but it should be clearly acknowledged in the text as a free parameter selected on the dev set (which the paper does later); please make this explicit in Section 3 as well.","section":"Section 3, evaluation setup"},{"comment":"The comparison with SpeechTokenizer is at different bitrates (3.1 vs. 6.0 kbps); while this is favorable to HASRD, it would strengthen the paper to include a same-bitrate comparison or to discuss the rate-distortion tradeoff more explicitly.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for Interspeech and the core idea is worth publishing if the quantitative claims are aligned with the evaluation protocol. The main concern is that the abstract and Section 4.2 overstate the WER improvement by using the continuous-latent comparison rather than the quantized-token comparison, and Table 4 shows that the full token sequence is worse than semantic-only tokens for ASR. These issues are fixable with re-analysis and clearer reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: HASRD is a useful, incremental extension of SpeechTokenizer and DAC. The reconstruction results are the strongest part: at 3.1 kbps versus 6.0 kbps, it reports mel distance 0.64 vs 0.76 and ViSQOL 4.50 vs 4.26, which looks credible. The ASR story is where the abstract overreaches. The 44% relative WER improvement over SpeechTokenizer comes from the latent-feature row in Table 3 (21.5 to 12.0). For the quantized tokens that actually feed an SLM, the comparison is 23.0 to 21.0, about 8.7% relative. Worse, Table 4 reports combined semantic+acoustic tokens at 24.2 WER, which is worse than SpeechTokenizer's full quantized output. So the headline appears to come from comparing HASRD's semantic-only codebook against SpeechTokenizer's full token set. That is an asymmetric comparison, and the paper needs to state exactly which token sets were used.\n\nThe residual assumption in Eq. (2) is only partially validated. Table 4 shows acoustic tokens alone carry some semantic leakage (70% WER is poor but not ceiling), and combined tokens driving ASR to 24.2 suggests the residual is not purely acoustic-complementary. That is not fatal—the disentanglement still gets some support from the speaker-similarity numbers—but it deserves a clear discussion.\n\nThe paper does several things well: the fbank-input CNN encoder with depthwise separable convolutions is a practical efficiency gain, the layer-weight analysis in Fig. 2 is informative, and the authors cite and position against concurrent work. The main soft spots are the overstated abstract claim, the unclear ASR token-set comparison, and the missing code and error bars. Selecting n_s on ASR and then evaluating on ASR is mild feature selection, not fatal circularity.\n\nVerdict: worth refereeing at Interspeech—the reconstruction contribution is solid and the tokenizer design is useful—but the WER claims need recomputation on equal footing and the abstract needs to be toned down before acceptance.","headline":"Reconstruction results are credible, but the headline 44% WER gain comes from an asymmetric comparison and does not survive on quantized tokens.","tokens_in":9739,"tokens_out":2914,"would_cite":true,"duration_ms":24797,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speech SSL representations separate into semantic and acoustic token streams, preserving ASR while halving bitrate.","keywords":["self-supervised learning","speech tokenization","residual vector quantization","spoken language models","automatic speech recognition","neural audio codecs","representation disentanglement"],"falsifier":"Reconstruct speech using only the acoustic RVQ codebooks and decode it with a strong ASR model; if the residual is semantic-free the WER should be near chance, whereas the paper's own acoustic-only WER of 70.2% suggests a stronger test would reveal how much semantic leakage remains.","tokens_in":1562,"feed_emoji":"🎙️","tokens_out":1777,"duration_ms":77624,"temperature":0.7,"pith_summary":"HASRD proposes that self-supervised speech representations can be factorized into two complementary discrete streams: a semantic stream carried by the first residual quantizer codebook, and an acoustic stream carried by the remaining codebooks. The paper claims this separation preserves automatic speech recognition accuracy while delivering high-quality waveform reconstruction at roughly half the bitrate of SpeechTokenizer (3.1 vs 6.0 kbps), and that a more efficient CNN pre-encoder improves both speed and WER. Compared with SpeechTokenizer, HASRD reports a 44% relative WER improvement on continuous encoder features (12.0% vs 21.5%) while also improving mel-spectrogram distance from 0.76 to 0.64. If correct, a single tokenizer could serve both spoken-language modeling, which wants semantic tokens, and synthesis, which wants acoustic detail, removing the need for separate codecs.","feed_headline":"Speech tokens split semantic from acoustic at half the bitrate","feed_subtitle":"HASRD keeps ASR accuracy in one codebook and speaker detail in the others, beating SpeechTokenizer on both.","key_machinery":"The central mechanism is residual vector quantization over SSL latents with a semantic first codebook: a convex combination of SSL layer representations is learned, the best semantic layer (layer 8) is k-means quantized as $\\widetilde{\\mathbf{H}}_s$, the acoustic residual is computed as $\\mathbf{H}_a = \\mathbf{H}\\mathbf{W} - \\widetilde{\\mathbf{H}}_s$ with a learnable projection $\\mathbf{W}$, and the final quantized representation is $\\widehat{\\mathbf{H}} = \\widetilde{\\mathbf{H}}_s + \\sum_{m=1}^{M-1} \\widetilde{\\mathbf{H}}_{a,m}$. The efficient front-end uses fbank input, depthwise separable convolutions, and random-projection (BestRQ) pre-training, while reconstruction relies on the DAC-style discriminators and losses.","core_discovery":"The paper claims that SSL latent layers contain separable semantic and acoustic information, and that this separation can be made operational by quantizing one chosen layer with k-means to form the first RVQ codebook, then quantizing the residual $\\mathbf{H}_a = \\mathbf{H}\\mathbf{W} - \\widetilde{\\mathbf{H}}_s$ with subsequent codebooks. On LibriSpeech test-clean, semantic-only tokens reach 21.0% WER, better than SpeechTokenizer's quantized 23.0%, while the full codebook stack reconstructs speech with mel distance 0.64 at 3.1 kbps versus SpeechTokenizer's 0.76 at 6.0 kbps. Speaker-similarity measurements support the intended split: semantic-only tokens give low speaker similarity (0.15) and good ASR, while acoustic-only tokens give high speaker similarity (0.67) and poor ASR, indicating that speaker identity lives in the acoustic codebooks and linguistic content in the first codebook.","pith_inferences":["Beyond the paper, if the acoustic codebooks truly carry speaker identity, swapping them between two utterances while keeping the same semantic codebook should perform one-shot voice conversion; the speaker-similarity results are consistent with this, but the paper does not test it.","The paper's own Table 4 shows that full tokens (semantic plus acoustic) give worse ASR WER than semantic-only tokens (24.2 vs 21.0), suggesting that downstream speech models should not blindly consume all codebooks for recognition tasks; this design rule is an editorial consequence, not stated by the authors.","A direct probe of the residual is testable: reconstruct speech from acoustic-only codebooks and run ASR; their 70.2% WER is far above chance but not maximal, so a stronger recognizer would quantify whether any semantic leakage remains in the acoustic codebooks."],"forward_implications":["A single tokenizer can serve both semantic tasks and high-quality synthesis: use the first codebook for ASR or language modeling and the full codebook stack for waveform reconstruction.","At 3.1 kbps with better reconstruction than SpeechTokenizer at 6.0 kbps, storage and transmission costs for spoken-language-model training drop roughly twofold.","The learned layer weights reveal which SSL layers encode semantic versus acoustic content, making SSL representations more interpretable and guiding layer selection for downstream tasks.","Because HASRD with a HuBERT teacher retains its teacher's WER rather than degrading during distillation, the approach may generalize to stronger SSL teachers without losing semantic performance."],"supporting_citations":[{"why":"SpeechTokenizer is the main baseline; its approach of distilling semantic information into the first codebook is extended and compared against.","marker":"[5]"},{"why":"Supplies the DAC encoder/decoder design, RVQ codebooks, and the discriminator and loss functions that HASRD adapts.","marker":"[16]"},{"why":"BestRQ provides the random-projection quantizer and masked language modeling objective used to pre-train the semantic representation.","marker":"[24]"},{"why":"Open implementation and study of BestRQ is the reproduced baseline whose encoder and normalization are improved.","marker":"[29]"},{"why":"HuBERT serves as the SSL teacher for the HuBERT variant of HASRD and for SpeechTokenizer.","marker":"[7]"},{"why":"LibriSpeech provides the training and evaluation data for both ASR and reconstruction.","marker":"[27]"},{"why":"ECAPA-TDNN is used to measure speaker similarity, providing the evidence that acoustic codebooks carry speaker identity.","marker":"[31]"}],"fun_headline_variants":["HASRD splits speech tokens: semantic first, acoustic residual after","HASRD halves bitrate and improves ASR by disentangling speech","One codebook for meaning, rest for sound: HASRD","HASRD disentangles speech: semantics first, acoustics after","HASRD separates speech meaning from sound at half bitrate"],"cache_read_input_tokens":11904,"weakest_assumption_plain":"The factorization is clean only if subtracting the k-means quantized semantic layer from the projected SSL representation leaves a residual that contains no semantic content; if the residual retains word or phone information, the two codebooks are not truly disentangled.","fun_headline_variants_meta":{"raw":{"variants":["HASRD splits speech tokens: semantic first, acoustic residual after","HASRD halves bitrate and improves ASR by disentangling speech","One codebook for meaning, rest for sound: HASRD","HASRD disentangles speech: semantics first, acoustics after","HASRD separates speech meaning from sound at half bitrate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3045,"prompt_tokens":885,"completion_tokens":2160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2070}},"tokens_in":501,"tokens_out":2160,"duration_ms":17178,"temperature":1.0,"reasoning_tokens":2070,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:56:44.406855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reconstruct speech using only the acoustic RVQ codebooks and decode it with a strong ASR model; if the residual is semantic-free the WER should be near chance, whereas the paper's own acoustic-only WER of 70.2% suggests a stronger test would reveal how much semantic leakage remains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SpeechTokenizer is the main baseline; its approach of distilling semantic information into the first codebook is extended and compared against."},{"cited_title":"Ex- ploration of efficient end-to-end ASR using discretized input from self-supervised learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the DAC encoder/decoder design, RVQ codebooks, and the discriminator and loss functions that HASRD adapts."},{"cited_title":"A ConvNet for the 2020s,","cited_arxiv_id":null,"evidence_quote":"Open implementation and study of BestRQ is the reproduced baseline whose encoder and normalization are improved."},{"cited_title":"ReVISE: Self-supervised speech resynthesis with visual input for univer- sal and generalized speech regeneration,","cited_arxiv_id":null,"evidence_quote":"LibriSpeech provides the training and evaluation data for both ASR and reconstruction."},{"cited_title":"Mel- GAN: Generative adversarial networks for conditional wave- form synthesis,","cited_arxiv_id":null,"evidence_quote":"ECAPA-TDNN is used to measure speaker similarity, providing the evidence that acoustic codebooks carry speaker identity."}],"review_version":1}