{"id":"97d7ad12-2c4b-453a-999a-25415eccc2b5","arxiv_id":"2509.02349","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A four-part benchmark plus a semantic/acoustic token taxonomy for comparing audio codecs, with correlation analysis across ten models.","lead":"This paper introduces a benchmark that measures audio codecs on reconstruction quality, token stability, language-model perplexity, and downstream task performance. It also proposes a taxonomy of audio tokens as semantic, acoustic, fused, or decoupled, and reports correlations between these dimensions across ten models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) PPL normalization is arbitrary and confounded by token rate and training-data overlap; the Table 7 correlations central to the paper's claim are not robust.","rationale":"The paper's abstract and conclusion assert that the benchmark results validate the proposed semantic/acoustic definitions and show correlations among the four dimensions. The most consequential empirical evidence for this is Table 7, where PPL is correlated with probe, ID-stability, and reconstruction metrics across codecs. The reader flagged the arbitrary PPL normalization in Eq. (1); I agree and wish to underscore that this is load-bearing because the correlation analysis is the only quantitative link between the PPL dimension and the other three dimensions. The normalization assumes PPL scales linearly with codebook size, but no theoretical or empirical justification is given. The codecs differ in token rate and training corpora, and the paper's own unexplained anomalies—EnCodec's low PPL, Mimi's high PPL, DAC/WavTokenizer's inverted domain trend—suggest confounds. With only 6-8 data points and no error bars, the correlations are fragile. A concrete robustness check, as described, would settle whether the claimed correlations survive. I considered the potential circularity of defining 'semantic' via text-describability and then validating it on text-describable tasks; while real, that is a conceptual limitation that doesn't invalidate the benchmark's utility. The PPL comparability issue is more directly threatening to the quantitative claim. Therefore the reader's conditional verdict is appropriate, pending the proposed check.","tokens_in":14848,"tokens_out":10156,"duration_ms":114913,"concrete_test":"Using the released code and dataset, recompute the Table 7 correlations after (a) replacing Eq. (1) with raw PPL (no size normalization), (b) using only the first codebook's PPL for multi-codebook codecs, and (c) computing PPL on held-out corpora that no codec was trained on (e.g., LibriTTS test-clean for speech, GTZAN test for music). If the sign or significance of the key correlations (PPL vs WER_CTC, PPL vs ESC_ACC, PPL vs MRC/OS) changes for any variant, the paper's central conclusion is an artifact of the chosen PPL definition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that the four evaluation dimensions are consistent with the semantic/acoustic taxonomy depends on the perplexity results and their correlations in Table 7. This dependence is the least secure. Equation (1) normalizes PPL by dividing exp(LCE) by Scb/1024, which assumes the LM's cross-entropy scales linearly with log codebook size; no justification is given. The codecs also differ in token rate (12.5-100 tokens/s), number of codebooks (1/2/8), and whether they were trained on the same corpus as the PPL LM (Emilia-EN for speech, MTG-Jamendo for music). These confounds are visible in the paper's own anomalies: EnCodec, despite being 'acoustic,' has unexpectedly low PPL; Mimi, which uses semantic distillation, does not; and DAC/WavTokenizer have lower PPL on music than speech, which the authors attribute to MTG-Jamendo training overlap. Thus PPL appears to reflect codec-specific statistical properties rather than 'semantic content.' Moreover, the Pearson correlations in Table 7 are computed over only 6-8 codecs with no error bars; the sign of a correlation can be reversed by a single outlier. Without a robustness check on the normalization or a held-out evaluation, the claimed correlation—and the assertion that introducing semantics benefits LM modeling—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AudioCodecBench, a benchmark for evaluating neural audio codecs and SSL-based tokenizers across four dimensions: reconstruction fidelity, codebook ID stability under perturbation, decoder-only LM perplexity, and downstream probe-task performance. It proposes a taxonomy in which a semantic feature is defined as one strictly describable by text, an acoustic feature as one not describable by text, and further distinguishes semantic-acoustic fused and decoupled features. Ten models are classified accordingly and evaluated on speech, music, and sound datasets. The main claim is that the experimental results validate the proposed definitions and show correlations among reconstruction metrics, ID stability, probe-task performance, and perplexity, specifically that semantic tokens are easier for LMs to model.","tokens_in":15240,"tokens_out":5832,"duration_ms":72977,"significance":"If the central claim were established, the benchmark would be a useful practical resource for comparing audio tokenizers for MLLMs. Its strengths are concrete: the authors release code and datasets; the benchmark covers three audio domains and a broad set of probe tasks; and the four-dimensional framework goes beyond reconstruction-only or ASR-only evaluation. The taxonomy, although stipulative, is clear enough to classify existing codecs. However, the empirical support for the load-bearing claims is currently weak: the PPL normalization in Eq. (1) is not justified, the cross-codec correlations in Table 7 have no significance measures, and part of the validation is circular because the probe tasks are grouped by the same text-describability criterion used to define the feature classes. With additional analysis and more careful claims, the benchmark could be a valuable contribution.","major_comments":[{"comment":"The cross-codec PPL comparison rests on Eq. (1), PPL = exp(LCE)/(Scb/1024). This assumes that perplexity scales linearly with codebook size, but no derivation or empirical justification is given. Codecs differ not only in codebook size but also in token rate (12.5 to 100 tokens/s), number of codebooks (1, 2, or 8), and training corpus. The paper itself notes anomalies consistent with these confounds: EnCodec, labeled acoustic, has unexpectedly low PPL; Mimi, despite semantic distillation, has high PPL; and DAC/WavTokenizer have lower PPL on MTG-Jamendo than on Emilia-EN, attributed to training overlap. The normalized PPL therefore likely reflects codec-specific statistical properties rather than semantic content. Since Table 7 and the conclusion that semantic tokens are easier to model depend directly on this normalized PPL, the load-bearing comparison is not established. Please also cla","section":"Perplexity, Eq. (1)"},{"comment":"The Pearson correlations in Table 7 are computed over only 6-8 codecs, with no p-values, confidence intervals, or robustness checks. A single outlier can reverse the sign of such correlations. More seriously, the interpretation of the ID-sensitivity correlations appears inconsistent: Table 7 reports positive correlations between PPL and MRC/OS (higher PPL associated with higher same-ID stability), and the text says this indicates that semantics brings stable ID patterns that benefit LM modeling. Since lower PPL is better, a beneficial relation should be negative. Either the correlation sign is wrong, the table's metric convention is not what the text assumes, or the interpretation is reversed. This needs to be corrected and supported with significance testing (e.g., Spearman rho, leave-one-out, or confidence intervals).","section":"Table 7 and Perplexity/ID-sensitivity interpretation"},{"comment":"The validation is partly circular. The probe tasks are labeled 'semantic-driven' using the same text-describability criterion that defines the semantic feature class, and then superior performance by semantic codecs on those tasks is presented as validating the definition ('These results also validate our definition of semantic'). This is at most internal consistency, not independent evidence. To claim validation, the authors should include held-out tasks not selected under the taxonomy, or explicitly reframe the results as consistency rather than proof. The phrase 'proving that introducing semantics can effectively capture high-level, symbolizable information in music' overclaims.","section":"Probe and Audio Feature Classification"},{"comment":"All experimental results are point estimates without error bars, significance tests, or seed information. For example, in Table 5 the Emotion Detection R2 values differ by small amounts (e.g., 0.06 vs. 0.07 for several codecs), and in Table 6 the MELD ED accuracy values are all between 0.48 and 0.52; these differences may be within run-to-run variability. Claims such as 'semantic codecs show better performance than acoustic codecs' need variance estimates over multiple seeds/splits, or at least an explicit statement of the number of runs and a statistical test. Without this, the relative rankings that support the taxonomy are not reliable.","section":"Tables 3, 5, 6 and Figures 3-4"}],"minor_comments":[{"comment":"The phrase 'existing research is unsuitable in the definitions' is grammatically unclear; consider 'inadequate in defining.' The abstract also says the results show correlation 'among reconstruction metrics, codebook ID stability, downstream probe tasks and perplexity,' but Table 7 only reports correlations between PPL and each other metric, not pairwise correlations among all dimensions.","section":"Abstract/Introduction"},{"comment":"The notation in Eq. (1) is not fully defined: LCE is called 'average cross-entropy loss,' but it is unclear whether it is averaged over tokens, codebooks, or both. Table 4 uses '-' for missing codebooks; a table note should explain that Mimi and SpeechTokenizer are not evaluated on music and WavTokenizer has one codebook.","section":"Table 4 / Eq. (1)"},{"comment":"The table does not report the number of codecs used for each correlation, the direction convention for metrics where lower is better (e.g., WER, PPL), or whether any correlation is statistically significant. The label 'WERCTC' and 'CERC TC' is likely a typographical artifact.","section":"Table 7"},{"comment":"The figures would benefit from explicit axis labels and legends. The appendix figures appear to repeat the same information without summary statistics or confidence intervals, making it hard to assess variability across codebooks and runs.","section":"Figures 3-4 and Appendix"},{"comment":"There is a typo in the section heading: 'Conlusion' should be 'Conclusion.' The conclusion also overstates the degree to which the results 'show the correctness' of the definitions; see major comment 3.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The benchmark assets and task coverage are valuable, and the taxonomy is a reasonable starting point. The main risk is that the current evidence for the central claim is built on an unjustified PPL normalization and on small-N correlations without significance testing, with at least one apparent sign inconsistency in the ID-sensitivity interpretation. These issues are fixable by reanalysis, additional robustness checks, and softened claims, so I do not recommend rejection. I would want to see the revised manuscript before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is worth a look if you work on audio tokenization. It consolidates a lot of evaluation practice into a single benchmark and proposes a clean four-way taxonomy—acoustic, semantic, fused, decoupled—based on whether the feature is strictly text-describable. That taxonomy is a helpful organizing device, even if it is a definition rather than a derived result. The benchmark covers reconstruction, ID stability, LM perplexity, and a wide set of probe tasks across speech, music, and sound, and the authors ship code and data. Those are real assets.\n\nThe reconstruction and probe results look internally consistent, and the pattern that semantic codecs do better on text-describable tasks and worse on pure reconstruction is plausible. The paper also reports anomalies (EnCodec's low PPL, Mimi's high PPL, DAC and WavTokenizer on music) without papering over them, which I appreciate.\n\nThe soft spot is the perplexity analysis. Equation (1) normalizes PPL by dividing by Scb/1024, where Scb is codebook size. That is an arbitrary scaling. PPL already depends on codebook size, but dividing by a linear codebook-size factor assumes a relationship that isn't justified. More importantly, codecs differ in token rate, number of codebooks, and whether they were trained on the same corpus as the LM used for PPL. The authors even attribute the DAC/WavTokenizer music result to training overlap. So the normalized PPL is not a clean measure of semantic content.\n\nThe correlations in Table 7 are computed over six to eight codecs with no error bars or significance tests. A single outlier can flip a correlation sign. The paper's central claim—that the four evaluation dimensions are consistent with the taxonomy—leans heavily on these correlations. That claim is not established, and the PPL column is the least secure part of the benchmark.\n\nThere is also a mild circularity in the probe validation: the tasks are classified as semantic- or acoustic-driven according to the same text-describability criterion used to define the features, and then the results are taken as evidence for the definition. That doesn't invalidate the task results, but it means the taxonomy's utility rests on whether the classification is useful, not on an independent test.\n\nBottom line: as a benchmark and a conversation starter, it is solid and worth engaging with. For peer review, I would send it out with the expectation that the authors re-derive or drop the PPL normalization, add error bars or per-codec variance, and reframe the correlation claim as exploratory rather than confirmatory. The benchmark itself deserves a serious referee; the current correlations do not.","headline":"A useful codec benchmark and taxonomy, but the PPL normalization and low-n correlations are too shaky to support the paper's central validation claim.","tokens_in":15678,"tokens_out":2904,"would_cite":true,"duration_ms":30512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AudioCodecBench claims the semantic/acoustic line in audio tokens is text-describability, and backs it with a four-part evaluation whose metrics line up accordingly.","keywords":["audio codec evaluation","semantic tokens","acoustic tokens","token taxonomy","perplexity","codebook ID stability","downstream probes","audio language modeling"],"falsifier":"Run the perplexity experiment with the same LM and corpora but normalize by bitrate or token rate instead of codebook size; if the codec ordering changes materially, the claimed semantics–perplexity link is an artifact of the formula. Alternatively, find one codec that scores well on semantic probe tasks yet has high normalized perplexity—that single counterexample breaks the reported correlation.","tokens_in":14834,"feed_emoji":"🎧","tokens_out":8351,"duration_ms":92480,"temperature":0.7,"pith_summary":"The paper proposes that the line between semantic and acoustic audio features is whether the content can be strictly described in text. From that criterion it builds a four-part taxonomy—acoustic, semantic, fused, and decoupled tokens—and evaluates eight codecs and two self-supervised speech models on reconstruction quality, codebook ID stability under noise and time shifts, perplexity of a small language model trained on the token streams, and downstream probe tasks across speech, music, and sound. Its results show that text-describable content is easier for a language model to model: perplexity correlates positively with semantic probe scores and ID stability, and negatively with raw acoustic reconstruction fidelity. The practical payoff is a benchmark that lets developers compare audio tokenizers on the dimension that matters for multimodal LLMs instead of on a single task.","feed_headline":"Text-describable tokens are the ones LMs model best","feed_subtitle":"A four-way evaluation shows text-describable audio tokens are the easiest for language models to model.","key_machinery":"The carrying mechanism is the four-part evaluation framework. Reconstruction uses PESQ, STOI, speaker similarity, and reconstructed WER/CER to place a codec on the acoustic end. ID sensitivity measures the unchanged rate of codebook indices across ten rounds of reconstruction and after 2 ms time shifts, placing codecs on a robustness axis. Perplexity is measured by training a 100M decoder-only transformer from scratch on each token stream, with perplexity normalized to a reference codebook size of 1024 via PPL = exp(L_CE)/(S_cb/1024), so token sequences can be compared across codebooks. Downstream probes use 11 tasks across speech, music, and sound, including a Conformer+CTC ASR head for tex","core_discovery":"The central claim, stated on the paper's own terms, is that 'semantic' should not be equated with ASR performance or with any single task; a semantic feature is one that must be strictly describable by text. Based on this, the paper classifies discrete audio representations into acoustic, semantic, semantic-acoustic fused, and semantic-acoustic decoupled features. It then measures codecs across four dimensions and finds that the dimensions agree: codecs classified as semantic or fused achieve lower perplexity, higher ID stability under time shifts, and better scores on probe tasks whose labels are text-like (genre, key, instrument, or transcribed speech), while acoustic codecs win on reconst","pith_inferences":["Inference: the 'text-describable' criterion is a practical proxy, not a demonstrated theory of meaning; a control that varies label granularity (phonetic, lexical, semantic) would test it more sharply.","Inference: the perplexity comparison may hinge on the hand-chosen normalization; dividing by bitrate or token rate instead of codebook size could reorder codecs and weaken the correlations.","Inference: a generative test—letting a frozen LM continue token sequences and scoring acoustic versus semantic coherence—would show whether low perplexity translates into usable generation, which the paper does not measure.","Inference: if the taxonomy is right, training a codec with an explicit text-alignment loss should lower its perplexity on held-out music even without reconstruction gains; that is a direct, testable prediction."],"forward_implications":["A new codec can be positioned on the semantic–acoustic axis by running these four probes, without waiting for a dedicated application benchmark.","Language-model perplexity becomes a cheap, task-agnostic proxy for how much text-describable content a token stream carries.","Codec design for multimodal LLMs can target semantic richness rather than reconstruction fidelity alone, since the two goals trade off in the measured correlations.","Music tasks with text-like labels—genre, key, instrument—can serve as semantic probes, so semantics stops being measured solely by speech recognition.","Decoupled codecs that separate semantic and acoustic streams are the configuration most likely to satisfy both LM modeling and reconstruction, a design direction the correlations support."],"supporting_citations":[{"why":"Supplies the HuBERT SSL model whose hidden-unit tokens anchor the paper's 'semantic feature' category.","marker":"Hsu et al. 2021"},{"why":"wav2vec 2.0 supplies the MLM-style contextual representation that motivates the text-describable semantic definition.","marker":"Baevski et al. 2020"},{"why":"EnCodec is one of the acoustic codecs whose reconstruction-first behavior sets the acoustic end of the taxonomy.","marker":"D´efossez et al. 2022"},{"why":"DAC provides the other high-fidelity acoustic codec baseline in reconstruction and probe evaluations.","marker":"Kumar et al. 2024"},{"why":"SpeechTokenizer defines the semantic/acoustic token split and the distillation scheme that the paper classifies as fused.","marker":"Zhang et al. 2024"},{"why":"SemantiCodec supplies the decoupled dual-encoder example whose separate semantic and acoustic streams illustrate the fourth feature class.","marker":"Liu et al. 2024"},{"why":"Code Drift contributes the multi-round reconstruction stability idea that the ID-sensitivity experiments adapt.","marker":"O'Reilly et al. 2025"},{"why":"SUPERB provides the downstream-probing methodology that the probe component extends to music and sound.","marker":"wen Yang et al. 2021"},{"why":"MARBLE supplies the cross-domain probe-task template and many of the datasets the music probes reuse.","marker":"Yuan et al. 2023"}],"fun_headline_variants":["Text-describable audio tokens are best for LMs","Semantic = text-describable: LMs prefer these tokens","Four dimensions agree: text-describable tokens win","For LMs, audio tokens should be describable in text","The best audio tokens for LMs are text-describable"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing assumption is that perplexity values from different codecs can be fairly compared after dividing only by codebook size; if token rate, number of codebooks, or training data must also be accounted for, the rankings and the reported correlations shift.","fun_headline_variants_meta":{"raw":{"variants":["Text-describable audio tokens are best for LMs","Semantic = text-describable: LMs prefer these tokens","Four dimensions agree: text-describable tokens win","For LMs, audio tokens should be describable in text","The best audio tokens for LMs are text-describable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1351,"prompt_tokens":722,"completion_tokens":629,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":466,"tokens_out":629,"duration_ms":7553,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:34:31.150001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the perplexity experiment with the same LM and corpora but normalize by bitrate or token rate instead of codebook size; if the codec ordering changes materially, the claimed semantics–perplexity link is an artifact of the formula. Alternatively, find one codec that scores well on semantic probe tasks yet has high normalized perplexity—that single counterexample breaks the reported correlation.","supporting_citations":[{"cited_title":"H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A","cited_arxiv_id":null,"evidence_quote":"Supplies the HuBERT SSL model whose hidden-unit tokens anchor the paper's 'semantic feature' category."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"wav2vec 2.0 supplies the MLM-style contextual representation that motivates the text-describable semantic definition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DAC provides the other high-fidelity acoustic codec baseline in reconstruction and probe evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SemantiCodec supplies the decoupled dual-encoder example whose separate semantic and acoustic streams illustrate the fourth feature class."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Code Drift contributes the multi-round reconstruction stability idea that the ID-sensitivity experiments adapt."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MARBLE supplies the cross-domain probe-task template and many of the datasets the music probes reuse."}],"review_version":1}