{"id":"6f06459a-f62c-469a-97d3-e0ff260e899d","arxiv_id":"2506.15456","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A factorized speech codec with separate acoustic, phonetic, and lexical token streams shows that a lexical token layer can act as a word-level detector while preserving usable reconstruction quality.","lead":"This paper introduces a neural audio encoder that turns speech into three separate sets of discrete symbols: one for fine sound detail, one for phonemes, and one for word meaning. A generalist might read it because these layered symbolic representations could help speech AI handle both natural-sounding audio and semantic accuracy in one system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Disentanglement is asserted, not enforced: the additive Z_Q = Z_Qp + Z_Qa + Z_Ql has no orthogonality or residual constraint, and the reported PNMI/word-F1 only show each branch matches its own KD teacher.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing point: the additive decomposition relies on independently trained codebooks carrying disjoint information without interference. My stress-test concurs, and I would emphasize that the architecture does not even provide a residual pathway: RVQacoust sees the same encoder output Z as the phonetic and lexical branches, and the decoder sees only the sum. The reported evaluations are consistent with a weaker reading of the paper — a lexical branch trained with LaBSE/SAMU distillation does produce word-detector tokens, and a phonetic branch trained with HuBERT distillation does align with phonemes. But this is expected from the training objectives and does not establish that the acoustic branch is informationally disjoint from the other two. The codebook-size confound in Figure 4 further weakens the lexical-disentanglement evidence. I did not find a formal internal inconsistency in the equations; rather, the gap is between the strong claim of three cleanly separated token sets and the indirect, supervised metrics used to support it. The proposed leave-one-out ablation and linear-probe test would directly measure functional independence and would settle whether the central claim holds. Since the reader already recommended a conditional verdict, my analysis does not change that verdict, but it reinforces the need for this additional evidence before the strong claim can be accepted.","tokens_in":15778,"tokens_out":7307,"duration_ms":80245,"concrete_test":"Run inference-time ablations on the trained HAC-14-LaBSE-HuB-T-en model: set Z_Qa=0, Z_Qp=0, or Z_Ql=0 before feeding the sum to the decoder, and measure reconstruction (Mel/STFT/SI-SDR/ViSQOL), phoneme PNMI/ABX, and word-detector F1 from the remaining token sets. Also fit linear probes on Z_Qa alone for phoneme and word labels using the LibriSpeech forced alignments. True disentanglement predicts that removing Z_Ql collapses word-F1 but leaves PNMI and reconstruction nearly unchanged, removing Z_Qp collapses PNMI but not word-F1, removing Z_Qa degrades reconstruction but leaves both linguistic metrics unchanged, and Z_Qa probes are at chance. If instead any branch retains the capability of another branch after it is removed, the claimed factorization is not functional.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HAC yields three informationally disjoint token sets rests on the assumption that the factorized bottleneck in Section 2 separates acoustic, phonetic, and lexical information. The architecture provides no mechanism for this. RVQacoust is applied to the encoder output Z directly, not to a residual Z - Z_Qp - Z_Ql, and the final decoder input is the unconstrained sum Z_Q = Z_Qp + Z_Qa + Z_Ql. There is no orthogonality penalty, no mutual-information penalty, and no adversarial or reconstruction term that explicitly prevents the acoustic codebook from encoding the same phonetic or lexical information carried by the other two branches. The KD losses align the phonetic and lexical branches to their teachers, but cosine-similarity losses on two branches do not constrain the third. The evidence in Figures 5 and 6 shows that the lexical branch is a good word detector and the phonetic branch correlates with phonemes, but this is largely a check that each branch learned its own distillation target. It does not establish that the acoustic branch is linguistically empty, nor that the three token sets are usable independently. The word-F1 comparison is additionally confounded by codebook size: HAC's lexical codebook is 14-bit with 16,384 entries, while the DAC and ST baselines use 10-bit codebooks with only 1,024 entries, so the higher number of word detectors may partly reflect increased codebook capacity rather than true lexical specialization. Because the paper's headline contribution is the three-way disentanglement, and the load-bearing assumption of separability is not enforced or directly measured, the central claim is currently under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Hierarchical Audio Codec (HAC), a neural speech codec with a factorized bottleneck intended to produce three disentangled token streams: acoustic, phonetic, and lexical. HAC is trained with a GAN-based reconstruction objective plus two knowledge-distillation losses, one from HuBERT for phonetic structure and one from LaBSE/SAMU for lexical information. Experiments compare HAC with SpeechTokenizer and Descript Audio Codec variants on ABX discrimination, phoneme normalized mutual information (PNMI), word-detector F1, and reconstruction metrics on English and multilingual data. The central claim is that HAC yields token sets that align with phonemes and words, preserve speech naturalness, and outperform single-level baselines in disentanglement and reconstruction.","tokens_in":16052,"tokens_out":4365,"duration_ms":41609,"significance":"If substantiated, the HAC architecture would be a useful contribution: a single codec that supplies acoustic, phonetic, and lexical discrete units could serve speech language modeling, speech-to-speech translation, and spoken understanding. The paper's strengths include evaluation with external phoneme and word labels rather than teacher embeddings, which grounds the main results independently of the authors' own teacher model; the LaBSE-based variant reproduces the lexical effect without relying on SAMU; and the evaluation spans multiple languages. However, the central claim of full disentanglement is not yet supported: no explicit mechanism enforces separation among the three token streams, and several quantitative comparisons contradict the abstract's overstatement. These issues are fixable with additional diagnostics and revised claims, but they are load-bearing for the paper's contribution.","major_comments":[{"comment":"The factorization Z_Q = Z_Qp + Z_Qa + Z_Ql is presented without any explicit constraint enforcing information separation among the three codebook streams. RVQ_acoust is applied directly to Z, not to a residual after removing Z_Qp and Z_Ql, and the KD losses only align the phonetic and lexical branches to their respective teachers. Nothing in the objective prevents the acoustic codebook from encoding phonetic or lexical content. The layer-wise PNMI and word-F1 results show that the phonetic and lexical branches match their teachers, but they do not establish that the acoustic branch is linguistically empty or that the three token sets are informationally disjoint, which is precisely the paper's central claim.","section":"Section 2 (architecture and training objective)"},{"comment":"The abstract states that HAC 'outperforms single-level baselines in both disentanglement and reconstruction quality,' but the reported numbers do not support this for the DAC baselines. In Table 1, DAC has lower Mel-D (0.55 vs. 0.58), lower STFT-D (1.34 vs. 1.37), higher SI-SDR (7.82 vs. 7.42), and higher ViSQOL (4.50 vs. 4.34) than HAC. In Figures 2 and 3, DAC models achieve lower ABX error and higher PNMI than HAC models. The abstract should be revised to accurately reflect that HAC trails DAC on several metrics and outperforms the ST baseline specifically, and that the claimed advantage is in lexical-word detection rather than in reconstruction or phoneme-level disentanglement.","section":"Abstract, Table 1, Figures 2-3"},{"comment":"The word-detector F1 comparison is confounded by codebook size. HAC's lexical VQ layer has a 14-bit codebook with 16,384 entries, while the DAC and ST baselines use 10-bit codebooks with 1,024 entries. Reporting the absolute number of tokens that exceed an F1 threshold therefore gives HAC a structural advantage. The authors should report the fraction of codebook entries that act as word detectors, or compare against a baseline with a matched codebook size, to support the claim that HAC's lexical branch is genuinely more lexically specialized rather than merely larger.","section":"Figure 4"},{"comment":"There is an inconsistency in model naming that makes the central results hard to interpret. Section 3 defines HAC-14-SAMU-HuB-T-en and HAC-14-LaBSE-HuB-T-en, but Figure 5 and Table 1 refer to an undefined model 'HAC-10-SAMU-HuB-T-en'. It is unclear whether this is a typo or a distinct configuration, and the reader cannot tell which model is actually being evaluated in the layer-wise PNMI and reconstruction comparisons. This should be clarified.","section":"Section 3, Figure 5, Table 1"}],"minor_comments":[{"comment":"No confidence intervals or significance tests are reported for any of the quantitative comparisons, and several differences are small (e.g., Mel-D 0.55 vs. 0.58). Adding error bars or significance statements would strengthen the reliability of the claims.","section":"Figures 2-3 and Table 1"},{"comment":"The word-F1 analysis for the acoustic tokens shows that virtually no acoustic codebook entries act as word detectors, but no analogous phoneme-level analysis (e.g., PNMI of the acoustic tokens) is reported. Adding acoustic-token PNMI would directly address the concern that the acoustic branch may still carry phonetic information.","section":"Figure 6"},{"comment":"The word-detector evaluation is performed on English words only, while the multilingual HAC models are trained on 16 languages. The multilingual lexical claim would be better supported by at least a small multilingual word-detector or lexical-alignment evaluation.","section":"Section 3, word-detector evaluation"},{"comment":"The KD losses average teacher representations over all layers, which is a coarse aggregation. The choice of layer averaging is not discussed, and a comparison of different teacher layer choices would help establish the robustness of the phonetic and lexical alignments.","section":"Section 2, Eq. (3)-(4)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is the first codec I've seen that adds a dedicated discrete lexical token stream to an RVQ-GAN, using KD from text embeddings. The experiments show that stream does behave like word detectors, the phonetic stream tracks phonemes, and the acoustic stream mostly doesn't. That is a real, useful extension of SpeechTokenizer, and the LaBSE vs SAMU comparison is well designed: the lexical effect survives when the teacher is a speech encoder, so it isn't an artifact of needing transcripts at inference.\n\nWhat it does well: the layer-wise analyses (Figs 5 and 6) are the right kind of evidence. They show the lexical and phonetic branches carry different linguistic information, and the acoustic branch is relatively empty of both. The multilingual results are a plus. Reconstruction quality is close to DAC, though not better.\n\nWhere it's soft: the abstract overstates the comparison. Table 1 shows DAC beats HAC on every reconstruction metric, and Figs 2-3 show HAC behind DAC on ABX and PNMI. 'Outperforming single-level baselines' only holds against ST, not against a strong acoustic codec. The architecture doesn't enforce disentanglement: Z_Q is an unconstrained sum of three branches, with no orthogonality or residual loss on the acoustic branch. The paper's evidence is indirect—it shows each branch matches its own teacher and the acoustic branch isn't a word detector. That's decent support, but it doesn't prove the token sets are informationally disjoint. The codebook-size confound in Fig 4 is real but not fatal, because within HAC the phonetic and lexical codebooks are both 14-bit and the lexical one wins clearly. The HAC-10/HAC-14 naming inconsistency is sloppy. No error bars, no released code or checkpoints.\n\nBottom line: the central idea is solid and the evidence, while indirect, is suggestive enough to take seriously. The paper deserves a full peer review. I'd ask for a softened abstract, error bars, a direct measure of inter-token-set redundancy (e.g., MI between branch tokens), and artifacts before accepting. Worth engaging.","headline":"A genuinely new lexical token stream in a codec, with solid but indirect disentanglement evidence and an overstated abstract.","tokens_in":16684,"tokens_out":3951,"would_cite":true,"duration_ms":38337,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Hierarchical Audio Codec (HAC), which factorizes a neural codec's bottleneck into disentangled acoustic, phonetic, and lexical token sets, and argues these can coexist in one model without sacrificing reconstruction…","keywords":["neural audio codec","speech tokenization","residual vector quantization","knowledge distillation","disentangled representation","phonetic tokens","lexical tokens","multilingual speech"],"falsifier":"Take a trained HAC and measure word-detection F1 or phoneme-level mutual information on the acoustic token stream. If acoustic tokens predict words or phonemes nearly as well as the lexical or phonetic streams, the levels have not separated and the central claim fails.","tokens_in":15529,"feed_emoji":"🎧","tokens_out":6013,"duration_ms":58053,"temperature":0.7,"pith_summary":"The paper introduces Hierarchical Audio Codec (HAC), a neural speech codec that factorizes its discrete token bottleneck into three levels: acoustic, phonetic, and lexical. It trains the phonetic and lexical branches by distillation from self-supervised speech and text encoders, leaving the acoustic residual vector-quantization branch to carry the remaining detail. The claim is that the resulting token sets are disentangled: one aligns with phonemes, another with words, while the acoustic tokens preserve natural-sounding speech. This matters because a single codec could then supply discrete units for both speech generation and language understanding without stitching token streams from separate models.","feed_headline":"Codec splits speech into acoustic, phonetic, and lexical tokens","feed_subtitle":"HAC keeps audio quality while adding token streams that align with phonemes and words.","key_machinery":"The load-bearing object is the factorized VQ bottleneck: three quantizers in parallel, one residual RVQ for acoustics and two plain VQ layers for phonetic and lexical content, whose outputs are summed before decoding. The mechanism that separates the levels is the pair of knowledge-distillation losses, which pull the phonetic and lexical codebook embeddings toward HuBERT and LaBSE (or SAMU) representations; the acoustic branch is left free to explain everything else. Transformer encoders before the phonetic and lexical VQ layers, plus the low-dimensional code lookup procedure from the underlying codec architecture, support the separation.","core_discovery":"HAC's central discovery is that an additive bottleneck decomposition $Z_Q = Z_{Q_p} + Z_{Q_a} + Z_{Q_l}$ in a residual vector-quantization generative adversarial network, trained with two knowledge-distillation losses, produces token sets that isolate linguistic levels. The phonetic branch matches averaged HuBERT embeddings, the lexical branch matches LaBSE text embeddings of forced-aligned transcripts, and the acoustic branch, left unconstrained, learns the residual spectral detail. In the reported evaluations, phonetic tokens show phoneme alignment through ABX and phoneme-normalized mutual information, lexical tokens act as word detectors, and acoustic tokens show little lexical content while reconstruction quality remains comparable to a codec trained only for reconstruction.","pith_inferences":["An untested but direct consequence of the additive decomposition is ablatability: setting one branch to zero should remove exactly one linguistic level from the reconstruction, leaving the other levels mostly intact.","The three-level tokenization could serve as a controllable interface for generation, such as swapping lexical tokens to change content while keeping acoustic tokens to preserve voice, which the paper does not demonstrate.","The same distillation-into-a-factorized-bottleneck recipe could be applied to other teacher signals, such as prosody or speaker embeddings, to create further token dimensions within one codec.","A reader could directly estimate mutual information between the phonetic and lexical token streams; near-zero mutual information would confirm strong disentanglement, and the paper does not report such a measurement."],"forward_implications":["A single trained HAC can emit token streams for three abstraction levels from one forward pass, removing the need to merge or align tokens from separate codecs.","Downstream systems can choose token type by task: acoustic tokens for high-fidelity synthesis, phonetic tokens for phoneme-related structure, and lexical tokens for semantic and syntactic content.","Because SAMU can replace LaBSE as the lexical teacher, lexical tokens can be learned without text transcripts at inference, extending the approach to languages without forced alignments.","The additive residual formulation means the acoustic branch can be trained largely unsupervised while distillation guides the other branches, potentially easing extension to new languages or new linguistic levels.","Multilingual training results suggest the disentanglement transfers across languages, so one codec could serve speech understanding and generation for many languages."],"supporting_citations":[{"why":"Supplies the RVQ-GAN backbone, low-dimensional code lookup, and reconstruction and adversarial losses that HAC builds on.","marker":"[3]"},{"why":"Provides the two-level phonetic and acoustic tokenizer baseline and the knowledge-distillation loss formulation that HAC extends to three levels.","marker":"[11]"},{"why":"Supplies HuBERT averaged layer embeddings as the phonetic teacher signal in the distillation loss.","marker":"[8]"},{"why":"Supplies LaBSE text embeddings as the lexical teacher signal for the lexical distillation loss.","marker":"[12]"},{"why":"Supplies SAMU, a speech-based semantic encoder that removes the need for transcripts when computing the lexical distillation loss.","marker":"[19]"},{"why":"LibriSpeech is the English training corpus and evaluation testbed for the English-only models.","marker":"[14]"},{"why":"mHuBERT is the multilingual phonetic teacher used in the 16-language HAC variants.","marker":"[18]"},{"why":"The ABX discrimination task is used to measure how well the phonetic tokens preserve phoneme-level distinctions.","marker":"[20]"}],"fun_headline_variants":["Codec factors speech into acoustic, phonetic, and lexical tokens","HAC disentangles speech into three linguistic token sets","RVQ-GAN with distilled losses isolates phonemes and words","One codec, three speech levels: acoustic, phonetic, lexical"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that training two of the three parallel quantizers to imitate speech and text encoders keeps the three token streams from overlapping, leaving the untrained acoustic stream with only the leftover acoustic detail.","fun_headline_variants_meta":{"raw":{"variants":["Codec factors speech into acoustic, phonetic, and lexical tokens","HAC disentangles speech into three linguistic token sets","RVQ-GAN with distilled losses isolates phonemes and words","One codec, three speech levels: acoustic, phonetic, lexical"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2373,"prompt_tokens":826,"completion_tokens":1547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":1477}},"tokens_in":442,"tokens_out":1547,"duration_ms":11827,"temperature":1.0,"reasoning_tokens":1477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:34:39.632530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained HAC and measure word-detection F1 or phoneme-level mutual information on the acoustic token stream. If acoustic tokens predict words or phonemes nearly as well as the lexical or phonetic streams, the levels have not separated and the central claim fails.","supporting_citations":[{"cited_title":"These objectives are described in detail in [3]","cited_arxiv_id":null,"evidence_quote":"Supplies the RVQ-GAN backbone, low-dimensional code lookup, and reconstruction and adversarial losses that HAC builds on."},{"cited_title":"AudioLM: a language modeling approach to audio gener- ation,","cited_arxiv_id":null,"evidence_quote":"Provides the two-level phonetic and acoustic tokenizer baseline and the knowledge-distillation loss formulation that HAC extends to three levels."},{"cited_title":"Direct speech-to-speech translation with discrete units,","cited_arxiv_id":null,"evidence_quote":"Supplies LaBSE text embeddings as the lexical teacher signal for the lexical distillation loss."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,","cited_arxiv_id":null,"evidence_quote":"Supplies SAMU, a speech-based semantic encoder that removes the need for transcripts when computing the lexical distillation loss."},{"cited_title":"HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,","cited_arxiv_id":null,"evidence_quote":"LibriSpeech is the English training corpus and evaluation testbed for the English-only models."},{"cited_title":"Language-agnostic BERT sentence embedding,","cited_arxiv_id":null,"evidence_quote":"mHuBERT is the multilingual phonetic teacher used in the 16-language HAC variants."}],"review_version":2}