Pith. sign in

REVIEW 4 major objections 5 minor 53 references

AudioCodecBench claims the semantic/acoustic line in audio tokens is text-describability, and backs it with a four-part evaluation whose metrics line up accordingly.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A four-part benchmark plus a semantic/acoustic token taxonomy for comparing audio codecs, with correlation analysis across ten models.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful codec benchmark and taxonomy, but the PPL normalization and low-n correlations are too shaky to support the paper's central validation claim. the 4 major comments →

arxiv 2509.02349 v2 pith:A4IOET5P submitted 2025-09-02 cs.SD cs.AIcs.LG

AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation

Lu Wang , Hao Chen , Siyu Wu , Zhiyue Wu , Hao Zhou , Chengfeng Zhang , Ting Wang , Haodi Zhang This is my paper
classification cs.SD cs.AIcs.LG
keywords audio codec evaluationsemantic tokensacoustic tokenstoken taxonomyperplexitycodebook ID stabilitydownstream probesaudio language modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that the line between semantic and acoustic audio features is whether the content can be strictly described in text. From that criterion it builds a four-part taxonomy—acoustic, semantic, fused, and decoupled tokens—and evaluates eight codecs and two self-supervised speech models on reconstruction quality, codebook ID stability under noise and time shifts, perplexity of a small language model trained on the token streams, and downstream probe tasks across speech, music, and sound. Its results show that text-describable content is easier for a language model to model: perplexity correlates positively with semantic probe scores and ID stability, and negatively with raw acoustic reconstruction fidelity. The practical payoff is a benchmark that lets developers compare audio tokenizers on the dimension that matters for multimodal LLMs instead of on a single task.

Core claim

The central claim, stated on the paper's own terms, is that 'semantic' should not be equated with ASR performance or with any single task; a semantic feature is one that must be strictly describable by text. Based on this, the paper classifies discrete audio representations into acoustic, semantic, semantic-acoustic fused, and semantic-acoustic decoupled features. It then measures codecs across four dimensions and finds that the dimensions agree: codecs classified as semantic or fused achieve lower perplexity, higher ID stability under time shifts, and better scores on probe tasks whose labels are text-like (genre, key, instrument, or transcribed speech), while acoustic codecs win on reconst

What carries the argument

The carrying mechanism is the four-part evaluation framework. Reconstruction uses PESQ, STOI, speaker similarity, and reconstructed WER/CER to place a codec on the acoustic end. ID sensitivity measures the unchanged rate of codebook indices across ten rounds of reconstruction and after 2 ms time shifts, placing codecs on a robustness axis. Perplexity is measured by training a 100M decoder-only transformer from scratch on each token stream, with perplexity normalized to a reference codebook size of 1024 via PPL = exp(L_CE)/(S_cb/1024), so token sequences can be compared across codebooks. Downstream probes use 11 tasks across speech, music, and sound, including a Conformer+CTC ASR head for tex

Load-bearing premise

The load-bearing assumption is that perplexity values from different codecs can be fairly compared after dividing only by codebook size; if token rate, number of codebooks, or training data must also be accounted for, the rankings and the reported correlations shift.

What would settle it

Run the perplexity experiment with the same LM and corpora but normalize by bitrate or token rate instead of codebook size; if the codec ordering changes materially, the claimed semantics–perplexity link is an artifact of the formula. Alternatively, find one codec that scores well on semantic probe tasks yet has high normalized perplexity—that single counterexample breaks the reported correlation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A new codec can be positioned on the semantic–acoustic axis by running these four probes, without waiting for a dedicated application benchmark.
  • Language-model perplexity becomes a cheap, task-agnostic proxy for how much text-describable content a token stream carries.
  • Codec design for multimodal LLMs can target semantic richness rather than reconstruction fidelity alone, since the two goals trade off in the measured correlations.
  • Music tasks with text-like labels—genre, key, instrument—can serve as semantic probes, so semantics stops being measured solely by speech recognition.
  • Decoupled codecs that separate semantic and acoustic streams are the configuration most likely to satisfy both LM modeling and reconstruction, a design direction the correlations support.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the 'text-describable' criterion is a practical proxy, not a demonstrated theory of meaning; a control that varies label granularity (phonetic, lexical, semantic) would test it more sharply.
  • Inference: the perplexity comparison may hinge on the hand-chosen normalization; dividing by bitrate or token rate instead of codebook size could reorder codecs and weaken the correlations.
  • Inference: a generative test—letting a frozen LM continue token sequences and scoring acoustic versus semantic coherence—would show whether low perplexity translates into usable generation, which the paper does not measure.
  • Inference: if the taxonomy is right, training a codec with an explicit text-alignment loss should lower its perplexity on held-out music even without reconstruction gains; that is a direct, testable prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AudioCodecBench, a benchmark for evaluating neural audio codecs and SSL-based tokenizers across four dimensions: reconstruction fidelity, codebook ID stability under perturbation, decoder-only LM perplexity, and downstream probe-task performance. It proposes a taxonomy in which a semantic feature is defined as one strictly describable by text, an acoustic feature as one not describable by text, and further distinguishes semantic-acoustic fused and decoupled features. Ten models are classified accordingly and evaluated on speech, music, and sound datasets. The main claim is that the experimental results validate the proposed definitions and show correlations among reconstruction metrics, ID stability, probe-task performance, and perplexity, specifically that semantic tokens are easier for LMs to model.

Significance. If the central claim were established, the benchmark would be a useful practical resource for comparing audio tokenizers for MLLMs. Its strengths are concrete: the authors release code and datasets; the benchmark covers three audio domains and a broad set of probe tasks; and the four-dimensional framework goes beyond reconstruction-only or ASR-only evaluation. The taxonomy, although stipulative, is clear enough to classify existing codecs. However, the empirical support for the load-bearing claims is currently weak: the PPL normalization in Eq. (1) is not justified, the cross-codec correlations in Table 7 have no significance measures, and part of the validation is circular because the probe tasks are grouped by the same text-describability criterion used to define the feature classes. With additional analysis and more careful claims, the benchmark could be a valuable contribution.

major comments (4)
  1. [Perplexity, Eq. (1)] The cross-codec PPL comparison rests on Eq. (1), PPL = exp(LCE)/(Scb/1024). This assumes that perplexity scales linearly with codebook size, but no derivation or empirical justification is given. Codecs differ not only in codebook size but also in token rate (12.5 to 100 tokens/s), number of codebooks (1, 2, or 8), and training corpus. The paper itself notes anomalies consistent with these confounds: EnCodec, labeled acoustic, has unexpectedly low PPL; Mimi, despite semantic distillation, has high PPL; and DAC/WavTokenizer have lower PPL on MTG-Jamendo than on Emilia-EN, attributed to training overlap. The normalized PPL therefore likely reflects codec-specific statistical properties rather than semantic content. Since Table 7 and the conclusion that semantic tokens are easier to model depend directly on this normalized PPL, the load-bearing comparison is not established. Please also cla
  2. [Table 7 and Perplexity/ID-sensitivity interpretation] The Pearson correlations in Table 7 are computed over only 6-8 codecs, with no p-values, confidence intervals, or robustness checks. A single outlier can reverse the sign of such correlations. More seriously, the interpretation of the ID-sensitivity correlations appears inconsistent: Table 7 reports positive correlations between PPL and MRC/OS (higher PPL associated with higher same-ID stability), and the text says this indicates that semantics brings stable ID patterns that benefit LM modeling. Since lower PPL is better, a beneficial relation should be negative. Either the correlation sign is wrong, the table's metric convention is not what the text assumes, or the interpretation is reversed. This needs to be corrected and supported with significance testing (e.g., Spearman rho, leave-one-out, or confidence intervals).
  3. [Probe and Audio Feature Classification] The validation is partly circular. The probe tasks are labeled 'semantic-driven' using the same text-describability criterion that defines the semantic feature class, and then superior performance by semantic codecs on those tasks is presented as validating the definition ('These results also validate our definition of semantic'). This is at most internal consistency, not independent evidence. To claim validation, the authors should include held-out tasks not selected under the taxonomy, or explicitly reframe the results as consistency rather than proof. The phrase 'proving that introducing semantics can effectively capture high-level, symbolizable information in music' overclaims.
  4. [Tables 3, 5, 6 and Figures 3-4] All experimental results are point estimates without error bars, significance tests, or seed information. For example, in Table 5 the Emotion Detection R2 values differ by small amounts (e.g., 0.06 vs. 0.07 for several codecs), and in Table 6 the MELD ED accuracy values are all between 0.48 and 0.52; these differences may be within run-to-run variability. Claims such as 'semantic codecs show better performance than acoustic codecs' need variance estimates over multiple seeds/splits, or at least an explicit statement of the number of runs and a statistical test. Without this, the relative rankings that support the taxonomy are not reliable.
minor comments (5)
  1. [Abstract/Introduction] The phrase 'existing research is unsuitable in the definitions' is grammatically unclear; consider 'inadequate in defining.' The abstract also says the results show correlation 'among reconstruction metrics, codebook ID stability, downstream probe tasks and perplexity,' but Table 7 only reports correlations between PPL and each other metric, not pairwise correlations among all dimensions.
  2. [Table 4 / Eq. (1)] The notation in Eq. (1) is not fully defined: LCE is called 'average cross-entropy loss,' but it is unclear whether it is averaged over tokens, codebooks, or both. Table 4 uses '-' for missing codebooks; a table note should explain that Mimi and SpeechTokenizer are not evaluated on music and WavTokenizer has one codebook.
  3. [Table 7] The table does not report the number of codecs used for each correlation, the direction convention for metrics where lower is better (e.g., WER, PPL), or whether any correlation is statistically significant. The label 'WERCTC' and 'CERC TC' is likely a typographical artifact.
  4. [Figures 3-4 and Appendix] The figures would benefit from explicit axis labels and legends. The appendix figures appear to repeat the same information without summary statistics or confidence intervals, making it hard to assess variability across codebooks and runs.
  5. [Conclusion] There is a typo in the section heading: 'Conlusion' should be 'Conclusion.' The conclusion also overstates the degree to which the results 'show the correctness' of the definitions; see major comment 3.

Circularity Check

1 steps flagged

Probe-task validation of the semantic definition is partly self-definitional: semantic-driven tasks are selected by the same text-describability criterion that defines semantic features, so the results are a consistency check rather than an independent confirmation.

specific steps
  1. self definitional [Audio Feature Classification; Experiments and Analysis, Probe (paragraph after Table 5)]
    "These tasks share the common feature that their labels (e.g., “Pop,” “A major”) can be strictly described by text or symbols, with a correspondence between musical content and labels. We refer to these tasks as semantic-driven tasks. Therefore, in these tasks, semantic codecs show better performance than acoustic codecs. These results also validate our definition of “semantic,” proving that introducing semantics can effectively capture high-level, symbolizable information in music."

    The paper defines a semantic feature as one that 'must be strictly describable by text' (Audio Feature Classification). The probe tasks are then designated 'semantic-driven' using exactly that criterion: their labels are 'strictly described by text or symbols.' Observing that codecs labeled 'semantic' perform better on tasks labeled 'semantic-driven' is therefore a consistency check built from the definition, not an independent validation of it. The classification of tasks and codecs both follow from the same stipulated text-describability rule, so the 'validation' is partly circular. The performance ranking itself is empirical and not logically forced, hence the circularity is partial, not total.

full rationale

The benchmark is largely self-contained and externally grounded: reconstruction metrics, ID-stability measurements, and LM perplexity are computed from the codecs' outputs without fitting the taxonomy. There is no load-bearing self-citation or imported uniqueness theorem. The main circularity is the probe-task validation loop: tasks are labeled 'semantic-driven' iff their labels are text-describable, which is the same criterion used to define 'semantic,' so the probe results confirm the classification convention rather than independently testing it. The PPL normalization in Eq. (1) is arguably arbitrary and confounded by token rate and training-data overlap, but that is a robustness/validity concern, not a circularity. Overall, the central four-dimension evaluation has independent content, but the 'correctness of the definitions' claim is partially self-definitional, yielding a moderate score.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The benchmark introduces no new physical or mathematical entities. The free parameters are experimental choices (normalization constant, time shift, round count, codebook count) that directly affect the reported correlations. The axioms are the text-describability definition, the validity of a small LM's perplexity as a proxy, and the interpretability of correlations over a tiny sample.

free parameters (4)
  • PPL normalization reference codebook size = 1024
    Eq (1) normalizes each codec's perplexity by dividing by (Scb/1024); the choice of 1024 is hand-chosen and directly affects cross-codec PPL comparisons.
  • Time shift duration = 2 ms
    The ID sensitivity time-shift experiment uses 2 ms shifts, 'inspired by Code Drift' but no sensitivity analysis.
  • Multi-round reconstruction count = 10
    The ID stability multi-round reconstruction uses n=10 rounds; the paper does not study the effect of this choice.
  • Number of codebooks evaluated = 8
    For multi-codebook codecs, the first 8 codebooks are used; codecs with fewer codebooks are used entirely, making the comparison asymmetric.
axioms (3)
  • domain assumption Semantic features must be strictly describable by text
    The paper proposes this definition and uses it to classify codecs and select probe tasks; it is not proven or derived.
  • domain assumption Perplexity of a 100M Qwen2 LM is a valid measure of LM modeling efficiency
    The PPL experiment trains one small LM from scratch; the transfer to real large MLLMs is assumed.
  • standard math Pearson correlation across the evaluated models is interpretable
    Table 7 computes Pearson r over about 8 models without significance tests; the sample is small.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation." pith.science (2026). https://pith.science/paper/A4IOET5P

@misc{pith2026250902349,
  author       = {Pith},
  title        = {Pith review of: AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4IOET5P}},
  note         = {Machine review of arXiv:2509.02349}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal Large Language Models (MLLMs) have been widely applied in speech and music. This tendency has led to a focus on audio tokenization for Large Models (LMs). Unlike semantic-only text tokens, audio tokens must both capture global semantic content and preserve fine-grained acoustic details. Moreover, they provide a discrete method for speech and music that can be effectively integrated into MLLMs. However, existing research is unsuitable in the definitions of semantic tokens and acoustic tokens. In addition, the evaluation of different codecs typically concentrates on specific domains or tasks, such as reconstruction or Automatic Speech Recognition (ASR) task, which prevents fair and comprehensive comparisons. To address these problems, this paper provides suitable definitions for semantic and acoustic tokens and introduces a systematic evaluation framework. This framework allows for a comprehensive assessment of codecs' capabilities which evaluate across four dimensions: audio reconstruction metric, codebook index (ID) stability, decoder-only transformer perplexity, and performance on downstream probe tasks. Our results show the correctness of the provided suitable definitions and the correlation among reconstruction metrics, codebook ID stability, downstream probe tasks and perplexity.

Figures

Figures reproduced from arXiv: 2509.02349 by Chengfeng Zhang, Hao Chen, Haodi Zhang, Hao Zhou, Lu Wang, Siyu Wu, Ting Wang, Zhiyue Wu.

Figure 1
Figure 1. Figure 1: AudioCodecBench data distribution overview. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The proposed AudioCodecBench framework. Users provide pre-trained codec and obtain token-level outputs through [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The percentage of the same ID in each codebook of the codecs after multi-round reconstruction, cb stands for code [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The proportion of identical IDs in each codebook of the codecs after time shift processing and reconstruction. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 9
Figure 9. Figure 9: Multi-round Reconstruction results of SpeechTok [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Multi-round Reconstruction results of XCodec. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Multi-round Reconstruction results of WavTok [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 8
Figure 8. Figure 8: Multi-round Reconstruction results of Semanti [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 16
Figure 16. Figure 16: Audio Time Shift results of SemantiCodec. [PITH_FULL_IMAGE:figures/full_fig_p012_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Audio Time Shift results of SpeechTokenizer. [PITH_FULL_IMAGE:figures/full_fig_p012_17.png] view at source ↗
Figure 15
Figure 15. Figure 15: Audio Time Shift results of Mimi [PITH_FULL_IMAGE:figures/full_fig_p012_15.png] view at source ↗
Figure 19
Figure 19. Figure 19: Audio Time Shift results of WavTokenizer. [PITH_FULL_IMAGE:figures/full_fig_p013_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Audio Time Shift results of YuE. Appendix C: Visualization of music, speech and sound probe task results [PITH_FULL_IMAGE:figures/full_fig_p013_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Visualization for the speech and sound probe [PITH_FULL_IMAGE:figures/full_fig_p013_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages · 2 internal anchors

  1. [1]

    M.; and Weber, G

    Ardila, R.; Branson, M.; Davis, K.; Henretty, M.; Kohler, M.; Meyer, J.; Morais, R.; Saunders, L.; Tyers, F. M.; and Weber, G. 2020. Common Voice: A Massively-Multilingual Speech Corpus. arXiv:1912.06670

    Pith/arXiv arXiv 2020

  2. [2]

    Asgari, M.; Kiss, G.; Van Santen, J.; Shafran, I.; and Song, X. 2014. Automatic measurement of affective valence and arousal in speech. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), 965--969. IEEE

    work page 2014

  3. [3]

    Baevski, A.; Zhou, Y.; Mohamed, A.; and Auli, M. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, 12449--12460

    work page 2020

  4. [4]

    Bogdanov, D.; Won, M.; Tovstogan, P.; Porter, A.; and Serra, X. 2019. The mtg-jamendo dataset for automatic music tagging. In Proceedings of the Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), 1--3

    work page 2019

  5. [5]

    Borsos, Z.; Marinier, R.; Vincent, D.; Kharitonov, E.; Pietquin, O.; Sharifi, M.; Teboul, O.; Grangier, D.; Tagliasacchi, M.; and Zeghidour, N. 2023. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 2523--2533

    work page 2023

  6. [6]

    Chen, S.; Wang, C.; Chen, Z.; Wu, Y.; Liu, S.; Chen, Z.; Li, J.; Kanda, N.; Yoshioka, T.; Xiao, X.; et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6): 1505--1518

    work page 2022

  7. [7]

    Chiu, C.-C.; Qin, J.; Zhang, Y.; Yu, J.; and Wu, Y. 2022. Self-supervised learning with random-projection quantizer for speech recognition. In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 391...

    work page 2022

  8. [8]

    Chollet, F. 2017. Xception: Deep Learning with Depthwise Separable Convolutions. arXiv:1610.02357

    Pith/arXiv arXiv 2017

  9. [9]

    Chu, Y.; Xu, J.; Yang, Q.; Wei, H.; Wei, X.; Guo, Z.; Leng, Y.; Lv, Y.; He, J.; Lin, J.; Zhou, C.; and Zhou, J. 2024. Qwen2-Audio Technical Report. arXiv:2407.10759

    Pith/arXiv arXiv 2024

  10. [10]

    Défossez, A.; Copet, J.; Synnaeve, G.; and Adi, Y. 2022. High Fidelity Neural Audio Compression. arXiv:2210.13438

    Pith/arXiv arXiv 2022

  11. [11]

    Défossez, A.; Mazaré, L.; Orsini, M.; Royer, A.; Pérez, P.; Jégou, H.; Grave, E.; and Zeghidour, N. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037

    Pith/arXiv arXiv 2024

  12. [12]

    Engel, J.; Resnick, C.; Roberts, A.; Dieleman, S.; Norouzi, M.; Eck, D.; and Simonyan, K. 2017. Neural audio synthesis of musical notes with wavenet autoencoders. In Proceedings of the International Conference on Machine Learning (ICML), 1068--1077

    work page 2017

  13. [13]

    Gong, Y.; Jin, L.; Deng, R.; Zhang, D.; Zhang, X.; Cheng, Q.; Fei, Z.; Li, S.; and Qiu, X. 2025. XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs. arXiv:2506.23325

    Pith/arXiv arXiv 2025

  14. [14]

    Gong, Y.; Yu, J.; and Glass, J. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 151--155

    work page 2022

  15. [15]

    Graves, A.; Fern\' a ndez, S.; Gomez, F.; and Schmidhuber, J. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, 369–376. New York, NY, USA: Association for Computing Machinery. ISBN 1595933832

    work page 2006

  16. [16]

    Gulati, A.; Qin, J.; Chiu, C.-C.; Parmar, N.; Zhang, Y.; Yu, J.; Han, W.; Wang, S.; Zhang, Z.; Wu, Y.; and Pang, R. 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv:2005.08100

    Pith/arXiv arXiv 2020

  17. [17]

    He, H.; Shang, Z.; Wang, C.; Li, X.; Gu, Y.; Hua, H.; Liu, L.; Yang, C.; Li, J.; Shi, P.; Wang, Y.; Chen, K.; Zhang, P.; and Wu, Z. 2024. Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation. arXiv:2407.05361

    Pith/arXiv arXiv 2024

  18. [18]

    H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A

    Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 3451--3460

    work page 2021

  19. [19]

    Hu, J.; Shen, L.; Albanie, S.; Sun, G.; and Wu, E. 2019. Squeeze-and-Excitation Networks. arXiv:1709.01507

    Pith/arXiv arXiv 2019

  20. [20]

    o ck, S.; H \

    Knees, P.; Faraldo, A.; Herrera, P.; Vogl, R.; B \"o ck, S.; H \"o rschl \"a ger, F.; and Le Goff, M. 2015. Two Data Sets for Tempo Estimation and Key Detection in Electronic Dance Music Annotated from User Corrections. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 364--370

    work page 2015

  21. [21]

    Kumar, R.; Seetharaman, P.; Luebs, A.; Kumar, I.; and Kumar, K. 2024. High-fidelity audio compression with improved rvqgan. In Advances in Neural Information Processing Systems (NeurIPS), volume 36

    work page 2024

  22. [22]

    La Quatra, M.; Koudounas, A.; Vaiani, L.; Baralis, E.; Cagliero, L.; Garza, P.; and Siniscalchi, S. M. 2024. Benchmarking Representations for Speech, Music, and Acoustic Events. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 505–509. IEEE

    work page 2024

  23. [23]

    I.; Bay, M.; and Downie, J

    Law, E.; West, K.; Mandel, M. I.; Bay, M.; and Downie, J. S. 2009. Evaluation of algorithms using games: The case of music tagging. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 387--392

    work page 2009

  24. [24]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. arXiv:2304.08485

    Pith/arXiv arXiv 2023

  25. [25]

    Liu, H.; Xu, X.; Yuan, Y.; Wu, M.; Wang, W.; and Plumbley, M. D. 2024. SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound. IEEE Journal of Selected Topics in Signal Processing, 18(8): 1448–1461

    work page 2024

  26. [26]

    Mentzer, F.; Minnen, D.; Agustsson, E.; and Tschannen, M. 2023. Finite Scalar Quantization: VQ-VAE Made Simple. arXiv:2309.15505

    Pith/arXiv arXiv 2023

  27. [27]

    D.; Duret, J.; Ploujnikov, A.; Subakan, C.; and Ravanelli, M

    Mousavi, P.; Libera, L. D.; Duret, J.; Ploujnikov, A.; Subakan, C.; and Ravanelli, M. 2024. DASB - Discrete Audio and Speech Benchmark. arXiv:2406.14294

    Pith/arXiv arXiv 2024

  28. [28]

    Mousavi, P.; Maimon, G.; Moumen, A.; Petermann, D.; Shi, J.; Wu, H.; Yang, H.; Kuznetsova, A.; Ploujnikov, A.; Marxer, R.; Ramabhadran, B.; Elizalde, B.; Lugosch, L.; Li, J.; Subakan, C.; Woodland, P.; Kim, M.; yi Lee, H.; Watanabe, S.; Adi, Y.; and Ravanelli, M. 2025. Discrete Audio Tokens: More Than a Survey! arXiv:2506.10274

    arXiv 2025

  29. [29]

    O'Reilly, P.; Seetharaman, P.; Su, J.; Jin, Z.; and Pardo, B. 2025. Code Drift: Towards Idempotent Neural Audio Codecs. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5

    work page 2025

  30. [30]

    Ott, M.; Edunov, S.; Baevski, A.; Fan, A.; Gross, S.; Ng, N.; Grangier, D.; and Auli, M. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations

    work page 2019

  31. [31]

    Poria, S.; Hazarika, D.; Majumder, N.; Naik, G.; Cambria, E.; and Mihalcea, R. 2019. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. arXiv:1810.02508

    Pith/arXiv arXiv 2019

  32. [32]

    Rix, A.; Beerends, J.; Hollier, M.; and Hekstra, A. 2001. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), volume 2, 749--752 vol.2

    work page 2001

  33. [33]

    VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music

    Shi, J.; jin Shim, H.; Tian, J.; Arora, S.; Wu, H.; Petermann, D.; Yip, J. Q.; Zhang, Y.; Tang, Y.; Zhang, W.; Alharthi, D. S.; Huang, Y.; Saito, K.; Han, J.; Zhao, Y.; Donahue, C.; and Watanabe, S. 2025. VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music. arXiv:2412.17667

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    N.; Schmidt, E

    Soleymani, M.; Caro, M. N.; Schmidt, E. M.; Sha, C. Y.; and Yang, Y. H. 2013. 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia, 1--6

    work page 2013

  35. [35]

    H.; Hendriks, R

    Taal, C. H.; Hendriks, R. C.; Heusdens, R.; and Jensen, J. 2010. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, 4214--4217

    work page 2010

  36. [36]

    Team, C. 2025. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv:2405.09818

    Pith/arXiv arXiv 2025

  37. [37]

    Trowitzsch, I.; Taghia, J.; Kashef, Y.; and Obermayer, K. 2020. The NIGENS General Sound Events Database. arXiv:1902.08314

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  38. [38]

    R.; Raj, B.; Schuller, B

    Turian, J.; Shier, J.; Khan, H. R.; Raj, B.; Schuller, B. W.; Steinmetz, C. J.; Malloy, C.; Tzanetakis, G.; Velarde, G.; McNally, K.; Henry, M.; Pinto, N.; Noufi, C.; Clough, C.; Herremans, D.; Fonseca, E.; Engel, J.; Salamon, J.; Esling, P.; Manocha, P.; Watanabe, S.; Jin, Z.; and Bisk, Y. 2022. Hear: Holistic evaluation of audio representations. In Proc...

    work page 2022

  39. [39]

    Tzanetakis, G.; and Cook, P. 2002. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5): 293--302

    work page 2002

  40. [40]

    Wang, C.; Tang, Y.; Ma, X.; Wu, A.; Okhonko, D.; and Pino, J. 2020. fairseq S2T: Fast Speech-to-Text Modeling with fairseq. In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations

    work page 2020

  41. [41]

    J.; Lakhotia, K.; Lin, Y

    wen Yang, S.; Chi, P.-H.; Chuang, Y.-S.; Lai, C.-I. J.; Lakhotia, K.; Lin, Y. Y.; Liu, A. T.; Shi, J.; Chang, X.; Lin, G.-T.; Huang, T.-H.; Tseng, W.-C.; tik Lee, K.; Liu, D.-R.; Huang, Z.; Dong, S.; Li, S.-W.; Watanabe, S.; Mohamed, A.; and yi Lee, H. 2021. SUPERB: Speech processing Universal PERformance Benchmark. arXiv:2105.01051

    Pith/arXiv arXiv 2021

  42. [42]

    Wilkins, J.; Seetharaman, P.; Wahl, A.; and Pardo, B. 2018. Vocalset: A singing voice dataset. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 468--474

    work page 2018

  43. [43]

    H.; and yi Lee, H

    Wu, H.; Chung, H.-L.; Lin, Y.-C.; Wu, Y.-K.; Chen, X.; Pai, Y.-C.; Wang, H.-H.; Chang, K.-W.; Liu, A. H.; and yi Lee, H. 2024. Codec-SUPERB: An In-Depth Analysis of Sound Codec Models. arXiv:2402.13071

    Pith/arXiv arXiv 2024

  44. [44]

    Yang, D.; Liu, S.; Guo, H.; Zhao, J.; Wang, Y.; Wang, H.; Ju, Z.; Liu, X.; Chen, X.; Tan, X.; Wu, X.; and Meng, H. 2025. ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling. arXiv:2504.10344

    Pith/arXiv arXiv 2025

  45. [45]

    Yang, D.; Liu, S.; Huang, R.; Tian, J.; Weng, C.; and Zou, Y. 2023. HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec. arXiv:2305.02765

    Pith/arXiv arXiv 2023

  46. [46]

    Ye, Z.; Sun, P.; Lei, J.; Lin, H.; Tan, X.; Dai, Z.; Kong, Q.; Chen, J.; Pan, J.; Liu, Q.; Guo, Y.; and Xue, W. 2024. Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model. arXiv preprint arXiv:2408.17175

    Pith/arXiv arXiv 2024

  47. [47]

    Yuan, R.; Lin, H.; Guo, S.; Zhang, G.; Pan, J.; Zang, Y.; Liu, H.; Liang, Y.; Ma, W.; Du, X.; Du, X.; Ye, Z.; Zheng, T.; Jiang, Z.; Ma, Y.; Liu, M.; Tian, Z.; Zhou, Z.; Xue, L.; Qu, X.; Li, Y.; Wu, S.; Shen, T.; Ma, Z.; Zhan, J.; Wang, C.; Wang, Y.; Chi, X.; Zhang, X.; Yang, Z.; Wang, X.; Liu, S.; Mei, L.; Li, P.; Wang, J.; Yu, J.; Pang, G.; Li, X.; Wang,...

    arXiv 2025

  48. [48]

    Yuan, R.; Ma, Y.; Li, Y.; Zhang, G.; Chen, X.; Yin, H.; others; and Fu, J. 2023. Marble: Music audio representation benchmark for universal evaluation. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 39626--39647

    work page 2023

  49. [49]

    Zeghidour, N.; Luebs, A.; Omran, A.; Skoglund, J.; and Tagliasacchi, M. 2021. SoundStream: An End-to-End Neural Audio Codec. arXiv:2107.03312

    Pith/arXiv arXiv 2021

  50. [50]

    J.; Jia, Y.; Chen, Z.; and Wu, Y

    Zen, H.; Dang, V.; Clark, R.; Zhang, Y.; Weiss, R. J.; Jia, Y.; Chen, Z.; and Wu, Y. 2019. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. arXiv:1904.02882

    Pith/arXiv arXiv 2019

  51. [51]

    Zhang, X.; Zhang, D.; Li, S.; Zhou, Y.; and Qiu, X. 2024. SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. arXiv:2308.16692

    Pith/arXiv arXiv 2024

  52. [52]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  53. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.