{"id":"6cbf54c3-7fdb-45c6-8254-8e500012b1aa","arxiv_id":"2608.08286","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ReLMCodec's preserve-control-refine design yields a single-codebook speech codec that improves the predictability-reconstruction trade-off at 650 and 800 bps, with gains in downstream TTS.","lead":"This paper studies why some speech tokens are easier for AI language models to predict and introduces ReLMCodec, a low-bitrate single-stream speech codec that keeps linguistic structure from a frozen self-supervised model while adding acoustic detail. At 650 and 800 bits per second, it improves both reconstruction quality and downstream text-to-speech intelligibility, pushing the known trade-off between predictable tokens and high-fidelity audio.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The frontier claim pairs native reconstruction with predictability measured by a separate P-VQ that bypasses the deployed quantizer; P-ACC may not track N-ACC for actual codec tokens, and no baseline N-ACC is reported.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: P-ACC is a matched-probe proxy, and the paper itself states that P-ACC and N-ACC are not directly comparable. The frontier claim requires that the probe ranking transfer to native token sequences, but the only native-token evidence is N-ACC for ReLMCodec's own quantizer ablation, with no baseline N-ACC for comparison. This is not an internal inconsistency: the paper is transparent about the probe, and the reconstruction and TTS results are plausibly real. The issue is whether the predictability axis of the central claim measures what a speech LM actually consumes. The proposed check is concrete and uses existing checkpoints plus the paper's own training recipe, so it can settle the concern without new methodology. Since the reader already conditioned the verdict on this assumption, my read does not move the verdict; CONDITIONAL remains appropriate until the native-token comparison is run.","tokens_in":21720,"tokens_out":3729,"duration_ms":35086,"concrete_test":"For X-Codec2, FocalCodec, SemantiCodec, ReLMCodec@8K, and ReLMCodec@64K, train a single Qwen2-1.5B LM on each system's native token sequences using the Appendix B recipe (same optimization, splits, and evaluation), and report top-1 next-token accuracy (N-ACC) on LibriSpeech test-clean/test-other. If ReLMCodec's native N-ACC gain over its rate-matched baselines is comparable to its P-ACC gain (e.g., ReLMCodec@64K vs X-Codec2: P-ACC 9.65% vs 5.12%), the frontier claim survives; if N-ACC is flat or reversed, the predictability claim is an artifact of the P-VQ probe and the paper should be revised to a reconstruction-plus-TTS claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central frontier claim is computed on two different quantizers for the same codec: reconstruction uses ReLMCodec's native EMA-VQ, while predictability is measured with the matched P-VQ (8K codewords, feature-reconstruction objective, no waveform decoder) that bypasses the native quantizer. Appendix B explicitly states that P-ACC and N-ACC \"are not directly comparable,\" and Table A5 reports N-ACC only for ReLMCodec (8.72% for EMA-VQ versus 6.56% for FSQ); no baseline codec is given an N-ACC value. Therefore the claimed predictability gain over FocalCodec, SemantiCodec, or X-Codec2 could be an artifact of how the probe quantizes each representation, rather than a property of the native token sequences a speech LM actually models. The downstream TTS result does not close this gap, because TTS quality can improve with reconstruction fidelity alone, independent of token predictability. Moreover, the module ablations and the alpha=0.1 selection are all tuned on P-ACC; if P-ACC is not monotone in native N-ACC across codecs, the design principle and the frontier claim rest on a proxy of unknown validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces ReLMCodec, a single-codebook low-bitrate speech codec designed around the idea that pre-quantization phoneme structure predicts autoregressive token predictability. The authors first build a matched probing setup: 24 frozen codec and SSL representations are standardized, projected into a common 8,192-codeword probing vector quantizer (P-VQ), tokenized, and modeled by a shared Qwen2-1.5B LM; phoneme separability metrics are then correlated with probe accuracy (P-ACC) and perplexity (P-PPL). Based on this diagnosis, they instantiate a preserve-control-refine architecture: a frozen W2v-BERT 2.0 L17 main path at the quantizer input, a Pre-quantization Anchor-Preserving Adaptation (PAPA) residual module that adds acoustic detail while keeping the SSL anchor, and a training-only WavLM-Large L24 teacher via the JMAS loss. Evaluations report improved reconstruction at 650 and 800 bps relative to several released single-stream codecs, and improved downstream EmoVoice TTS intelligibility and speaker similarity for ReLMCodec tokens.","tokens_in":22050,"tokens_out":9410,"duration_ms":81845,"significance":"The paper's strengths are its controlled diagnostic (matched quantizer, LM, and evaluation protocol), bootstrap confidence intervals for the correlation analysis, extensive ablations (SSL-role, module, residual scale, quantizer), an efficiency benchmark, a human MOS study, and a plan to release code, checkpoints, and probing scripts. If the central frontier claim survives validation on native tokens, ReLMCodec would be a useful operating point for low-bitrate single-stream speech LMs, and the matched probing protocol would be a reusable measurement tool. The principal risk is that the predictability axis is measured on a separately trained P-VQ rather than on the codec's native tokens, and the paper does not currently supply the native-token measurements needed to close that gap.","major_comments":[{"comment":"The headline frontier claim (e.g., the 650/800 bps comparison in Table 2 and the sentence in Section 4.1 that the matched P-VQ probe raises P-ACC from 5.12% to 9.65%) pairs reconstruction metrics computed on native tokens with predictability measured on P-VQ tokens. Appendix B explicitly states that P-ACC and N-ACC are not directly comparable, and Table A5 gives N-ACC only for ReLMCodec's two quantizers, with no baseline codec given an N-ACC value. The P-VQ has a feature-reconstruction objective, no waveform decoder, and a different codebook distribution than the native EMA-VQ, so the measured P-ACC gain may be a property of the probe rather than of the token sequences an autoregressive speech LM actually consumes. Please report N-ACC or native-token perplexity for FocalCodec, SemantiCodec, X-Codec2, and the ReLMCodec variants under a shared LM training recipe, and show that the P-ACC ordering tracks the native-token ordering across these systems. Until then, the design choices in Tables 3-4 and the alpha=0.1 selection in Figure 4 rest on a proxy of unvalidated validity.","section":"End-to-End Reconstruction and Predictability; Appendix B; Table A5"},{"comment":"The core empirical claim is a set of point estimates from single checkpoints: no confidence intervals, no multiple seeds, and no significance tests are reported for WER, SIM, PESQ, STOI, or UTMOS. Several margins are small (e.g., SIM 0.749 for both FocalCodec and ReLMCodec@8K; PESQ 2.17 versus 2.19 across alpha in Figure 4), and Table 2 compares released baseline checkpoints with the authors' trained ReLMCodec checkpoints, so training-recipe differences are confounded with architecture differences. Please provide bootstrap or multi-seed intervals for the headline comparisons, and consider matched-training baselines for at least FocalCodec and X-Codec2; at minimum, the baseline checkpoint provenance and training budgets should be stated in the main text.","section":"Tables 2 and 6"},{"comment":"The quantizer ablation uses N-ACC to prefer EMA-VQ over FSQ, but the main design analyses (PAPA alpha, teacher role, module ablations) are all evaluated with P-ACC, and no P-ACC value is reported for the FSQ variant. This leaves the relationship between P-ACC and N-ACC untested even within a single codec architecture. A minimal check is to report P-ACC for both quantizers under the same P-VQ protocol with the same frozen encoder, together with N-ACC under a shared native-token LM; this would directly test whether the proxy is monotonic in native predictability for at least ReLMCodec.","section":"Table A5 and Section 4.2"}],"minor_comments":[{"comment":"The ReLMCodec@64K row renders as '13.96' where the codebook count and WER appear concatenated; please fix the column separation so the row reads as Codebooks=1, WER=3.96.","section":"Table 2"},{"comment":"The human MOS evaluation should report the number of listeners, the number of utterances rated, and inter-rater agreement; the current figure shows only rating distributions.","section":"Appendix Figure A5"},{"comment":"Equation (3) uses J_R^T without defining the Jacobian or its dimensions; please define all symbols in the PAPA derivation.","section":"Equation (3)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal. The main risk is the proxy-validity gap between P-ACC and native-token predictability, which is fixable with additional native-token experiments. I do not see evidence of problematic citation behavior; the related work is broad and the limitations are stated candidly. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is solid and worth your time. The genuinely new piece is the controlled probing protocol: twenty-four representations, one shared 8K P-VQ, one shared Qwen LM, bootstrap CIs. It makes the old intuition about phoneme structure and token predictability quantitative, and the correlation table is credible. The ReLMCodec design—frozen W2v-BERT anchor, scaled residual path, training-only WavLM teacher—is simple, and the ablations are the best part. PAPA retaining 99.55% KNN retention versus 40.62% for direct adaptation, with corresponding P-ACC gains, is a clean demonstration that the parameterization matters.\n\nThe main soft spot is the mismatch between the probe and the deployed quantizer. The headline claim about moving the predictability-reconstruction frontier pairs native reconstruction metrics with P-ACC measured on a separately trained P-VQ that bypasses the codec's actual EMA-VQ. The paper acknowledges P-ACC and N-ACC are not directly comparable, and it reports N-ACC only for its own two quantizer variants. No baseline codec gets an N-ACC value, so the specific claim that ReLMCodec's native tokens are easier to model than FocalCodec's or SemantiCodec's is not actually measured. The downstream TTS results use native tokens and show real gains, but those could come from better reconstruction alone. Since every module ablation and the alpha selection are tuned on P-ACC, the design principle would be on much firmer ground if we saw native-token N-ACC for at least a couple of baselines.\n\nMinor issues: Table 2 compares released baseline checkpoints against freshly trained ReLMCodec (standard in this field, but it does favor the new model), and reconstruction/TTS metrics come without significance tests or error bars.\n\nNone of that is fatal. The diagnostic is well-constructed within its declared scope, the transparency about limitations is genuine, and the downstream gains are encouraging. I'd send this to a careful referee. The referee should ask for baseline N-ACC on native tokens, or evidence that the P-ACC ranking transfers, plus error bars on the headline table.","headline":"Solid empirical paper with a genuinely useful matched diagnostic; the predictability claim rests on a probe quantizer rather than native tokens, but the TTS results and ablations make it worth serious review.","tokens_in":22506,"tokens_out":4067,"would_cite":true,"duration_ms":36265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that phoneme structure in the representation entering a speech quantizer predicts how predictable the resulting tokens are for autoregressive models, and that a codec can preserve this structure while adding acoustic…","keywords":["speech codec","neural audio compression","speech tokens","autoregressive speech modeling","phoneme structure","self-supervised speech representations","single-codebook quantization","text-to-speech"],"falsifier":"Train the same autoregressive language model directly on ReLMCodec's native 8K and 64K token sequences and on X-Codec2's, then compare native next-token accuracy: the design is confirmed only if the native-token advantage at 800 bps resembles the probe advantage (9.65% versus 5.12%), since the paper reports P-ACC on a separately trained probing quantizer, not on the codec's own tokens.","tokens_in":21461,"feed_emoji":"🎙️","tokens_out":7990,"duration_ms":65665,"temperature":0.7,"pith_summary":"Speech tokens that an autoregressive model can predict well and tokens that reconstruct waveforms well are usually different things. This paper establishes a diagnostic link: across 24 codec and self-supervised speech representations, the clarity of phoneme grouping before quantization strongly tracks next-token prediction accuracy under a matched probing protocol. The paper then builds ReLMCodec, a single-codebook codec that keeps frozen self-supervised features at the quantizer input, adds acoustic detail through a fixed-scale residual path, and uses a training-only teacher to refine post-quantization structure. At 650 and 800 bits per second, ReLMCodec improves both reconstruction and downstream text-to-speech intelligibility and speaker similarity relative to rate-matched baselines. The takeaway is that the design principle, not any single component, is the contribution.","feed_headline":"Phoneme clarity before quantization predicts token ease","feed_subtitle":"A single-codebook codec keeps self-supervised phoneme structure, adds acoustic detail, and moves the 650/800 bps frontier.","key_machinery":"The load-bearing object is the pre-quantization representation\\textemdash{}the continuous frame-level features that enter the quantizer before discrete code assignment\\textemdash{}because the paper's whole diagnostic measures phoneme structure there. The matched probing vector quantizer (P-VQ) is the instrument that makes the comparison fair: every representation is quantized with the same 8,192-codeword probe and scored with the same autoregressive language model, yielding probe accuracy (P-ACC) and probe perplexity (P-PPL). PAPA is the architectural mechanism that translates the finding into a codec: it writes the quantizer input as $z_t = s_t + \\alpha \\Delta_t$ with frozen SSL anchor $s_t$ and $\\alpha=0.1$, so the trainable branch can add acoustic detail without rewriting the SSL geometry. The JMAS loss is the refinement mechanism that shapes the quantized latent space toward the teacher's phoneme structure without adding inference-time cost.","core_discovery":"The central claim is that pre-quantization phoneme structure is a consistent predictor of autoregressive token predictability, with Spearman correlation 0.911 between KNN phoneme accuracy and probe next-token accuracy across 24 representations, and that this structure can be preserved inside a waveform codec without sacrificing reconstruction. ReLMCodec realizes the claim by composing three mechanisms: a frozen W2v-BERT 2.0 L17 anchor written with coefficient one into the quantizer input, a Pre-quantization Anchor-Preserving Adaptation (PAPA) that adds an acoustic residual at fixed scale $\\alpha=0.1$, and a joint\\textendash{}marginal alignment loss (JMAS) that pulls quantized latents toward WavLM-Large L24 teacher structure during training only. The result is an empirical frontier move at 650 bps and 800 bps: ReLMCodec@8K and ReLMCodec@64K beat rate-matched single-stream baselines on reconstruction metrics while raising probe next-token accuracy, and the tokens also improve downstream TTS word error rate and speaker similarity.","pith_inferences":["The paper reports P-ACC on a separately trained probing quantizer and, in Appendix B, states that P-ACC and native-token N-ACC are not directly comparable; a reader should not treat the 9.65% versus 5.12% figure as native-token evidence without checking native-token results.","Because the observational claim is limited to 50-Hz English LibriSpeech, the preserve\\textendash{}control\\textendash{}refine recipe may need re-validation for multilingual, noisy, or other-frame-rate speech before it is treated as a general design law.","The layer-dependence of phoneme structure suggests that selecting a teacher by final-layer accuracy alone is not generally safe; the right role assignment may need to be re-derived per encoder and per task.","A direct test of the design principle would replace the frozen SSL anchor with a trainable encoder of the same capacity: if P-ACC retention and KNN retention collapse, the anchor itself, rather than the added parameters, carries the predictability benefit."],"forward_implications":["If the correlation holds beyond the 24 evaluated representations, codec designers can screen candidate tokenizers by measuring pre-quantization phoneme KNN accuracy before training a full codec.","Single-codebook, single-stream codecs at 650 and 800 bps can achieve both strong reconstruction and predictable tokens, reducing the need for multi-stream residual quantizer interfaces in this regime.","Because the WavLM teacher and speaker-verification models are frozen and used only in training, ReLMCodec adds no inference-time computation beyond the SSL encoder, acoustic encoder, quantizer, and decoder.","Downstream TTS trained on ReLMCodec tokens shows lower word error rate and higher speaker similarity than rate-matched baselines, indicating the tokens carry usable linguistic and speaker information for generation.","The matched probing protocol itself is reusable: future codecs can be compared under identical quantizer capacity and language-model optimization rather than through their native token streams."],"supporting_citations":[{"why":"Supplies the LibriSpeech corpus used for the probing protocol, ReLMCodec training, and reconstruction and TTS evaluation.","marker":"[Panayotov et al., 2015]"},{"why":"Provides the Montreal Forced Aligner phoneme boundaries that define the phoneme-structure labels for the diagnostic analysis.","marker":"[McAuliffe et al., 2017]"},{"why":"Supplies the frozen W2v-BERT 2.0 features that serve as the preserve anchor at the quantizer input.","marker":"[Seamless Communication et al., 2023]"},{"why":"Supplies WavLM-Large L24 as the training-only structure teacher and WavLM-SV for speaker-similarity evaluation.","marker":"[Chen et al., 2022]"},{"why":"Introduces the joint\\textendash{}marginal alignment loss that ReLMCodec adapts into its JMAS post-quantization regularizer.","marker":"[Cheng et al., 2026]"},{"why":"Supplies the Qwen2-1.5B autoregressive language model used to compute P-ACC and P-PPL for every representation.","marker":"[Yang et al., 2024a]"},{"why":"Provides X-Codec2, the rate-matched 800 bps single-stream baseline and a key comparison point for the frontier claim.","marker":"[Ye et al., 2025b]"},{"why":"Provides FocalCodec, the rate-matched 650 bps single-stream baseline that ReLMCodec@8K is compared against.","marker":"[Della Libera et al., 2025]"},{"why":"Supplies WavTokenizer as a single-codebook low-bitrate codec baseline and part of the reconstruction comparison.","marker":"[Ji et al., 2025]"}],"fun_headline_variants":["Phoneme-preserving codec moves predictability frontier","ReLMCodec: phoneme structure yields easier LM tokens","Single-codebook codec preserves phonemes, boosts LM prediction","Pre-quant phoneme clarity drives autoregressive ease","Codec with phoneme anchor improves token predictability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the probing quantizer's next-token accuracy (P-ACC) measures the predictability of the codec's actual native tokens in downstream autoregressive speech models; if that proxy does not carry over, the design principle is not supported for real token sequences.","fun_headline_variants_meta":{"raw":{"variants":["Phoneme-preserving codec moves predictability frontier","ReLMCodec: phoneme structure yields easier LM tokens","Single-codebook codec preserves phonemes, boosts LM prediction","Pre-quant phoneme clarity drives autoregressive ease","Codec with phoneme anchor improves token predictability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000369,"raw_usage":{"total_tokens":2012,"prompt_tokens":1013,"completion_tokens":999,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":920}},"tokens_in":629,"tokens_out":999,"duration_ms":8523,"temperature":1.0,"reasoning_tokens":920,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:10:32.297081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same autoregressive language model directly on ReLMCodec's native 8K and 64K token sequences and on X-Codec2's, then compare native next-token accuracy: the design is confirmed only if the native-token advantage at 800 bps resembles the probe advantage (9.65% versus 5.12%), since the paper reports P-ACC on a separately trained probing quantizer, not on the codec's own tokens.","supporting_citations":[{"cited_title":"Montreal forced aligner: Trainable text-speech alignment using Kaldi","cited_arxiv_id":null,"evidence_quote":"Provides the Montreal Forced Aligner phoneme boundaries that define the phoneme-structure labels for the diagnostic analysis."}],"review_version":1}