Pith. sign in

REVIEW 5 major objections 6 minor 48 references

PAST: Phonetic-Acoustic Speech Tokenizer

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PAST, a speech tokenizer trained with direct phonetic supervision instead of distillation from pretrained self-supervised models, outperforms existing hybrid tokenizers on phonetic, reconstruction, and speech-language-modeling benchmarks.

desk verdict PAST has a genuinely new training recipe for hybrid speech tokenizers, but its headline comparison to SSL-distillation baselines is confounded by training on the same labeled domains used for evaluation. read the letter →

arxiv 2505.14470 v2 pith:QPVGYJV3 submitted 2025-05-20 cs.SD cs.CLcs.LGeess.AS

Nadav Har-Tuv , Or Tal , Yossi Adi This is my paper · ORCID
classification cs.SDcs.CLcs.LGeess.AS
keywords speechtokenizationphonetictokenshybridtokenizersauxiliaryCTClossphonemeclassificationresidualvectorquantizationlanguagemodelsstreaming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a speech tokenizer can capture both phonetic content and acoustic detail better by using explicit supervised phonetic labels instead of distilling pseudo-phonetic features from pretrained self-supervised models. The authors build PAST on top of the EnCodec architecture and add two auxiliary training objectives, a CTC character loss and a phoneme classification loss, applied to the first quantized codebook stream. They report that PAST surpasses hybrid baselines like SpeechTokenizer and X-Codec across phonetic metrics (PNMI, ABX, WER), reconstruction quality (SISNR, ViSQOL, PESQ), and downstream speech language modeling (sWUGGY). A streamable causal variant is also introduced, which retains most of the gains while enabling real-time applications. If the claim holds, it suggests that direct phonetic supervision can replace SSL distillation in hybrid tokenizers, simplifying training and improving performance.

What carries the argument

The key machinery is the set of auxiliary heads that operate on the first RVQ stream, z-hat-1: a CTC head (linear projection, BiLSTM, linear projection) aligned to character transcriptions, and a simple linear phoneme classifier aligned to TIMIT phoneme labels. These losses, L_ctc and L_phn, are added to the full EnCodec reconstruction objective with weights 12 and 5, respectively. A transformer encoder placed before the quantizer, trained with a stochastic skip-connection dropout, provides sequence modeling capacity; the dropout forces the model to use the transformer rather than bypass it. Together, the auxiliary heads inject phonetic structure directly into the quantized codebook stream that carries the bulk of the phonetic information, while the reconstruction loss preserves acoustic fidelity.

What would settle it

Retrain PAST on a non-English corpus without TIMIT phoneme labels, using only automatically generated character alignments, and compare PNMI, ABX, and downstream sWUGGY against SpeechTokenizer and X-Codec; if PAST no longer outperforms the SSL-distilled baselines, the central claim of replacing SSL distillation with supervised phonetic data collapses.

Watch

Extended reading notes

Core claim

The central discovery is that supervising the first residual vector quantization stream with phonetic targets—using a CTC loss over character transcriptions and a cross-entropy loss over TIMIT phoneme labels—produces discrete speech tokens that are simultaneously more phonetically informative and more acoustically faithful than tokens distilled from pretrained SSL models. PAST achieves this without an external vocoder or a pretrained teacher during tokenizer training, and the auxiliary losses are sufficient to push phonetic information into the quantized latent space. The paper demonstrates this with consistent improvements over SpeechTokenizer and X-Codec on phonetic representation, signal reconstruction, and speech language model evaluation, and shows via ablations that the CTC objective is the most critical component and that the transformer with skip-connection dropout is needed to realize the gains.

Load-bearing premise

The method's phonetic gains depend on the accuracy of character-level alignments produced by a pretrained Wav2Vec2 model and on TIMIT phoneme labels, so if these supervision signals are noisy or unavailable, the advantage over SSL distillation may not transfer to other languages or domains.

Editorial extensions

If this is right

  • If PAST's results hold, hybrid tokenizers can be trained without a pretrained SSL teacher or a separate vocoder, simplifying the tokenization pipeline and reducing training cost.
  • The streamable causal variant, requiring only a 20 ms look-ahead, makes PAST suitable for real-time speech applications such as live speech-to-speech translation and interactive spoken dialogue systems.
  • PAST's superior sWUGGY scores when used as the representation for a speech language model suggest that direct phonetic supervision yields a better foundation for spoken language generation than SSL-distilled tokens.
  • The ablation identifying CTC as the most crucial auxiliary objective implies that character-level alignment supervision, rather than phoneme labels alone, drives the phonetic encoding gains.
  • Because PAST replaces SSL distillation with supervised targets, its training may require much less unlabeled speech data, potentially easing application to low-resource languages if labeled alignments are available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's claim of eliminating pretrained models is only partially true: the CTC training targets are obtained from a pretrained Wav2Vec2 model that generates character-level alignments, so a pretrained model remains in the data preparation pipeline even though it is not part of the tokenizer's forward pass.
  • If the supervised approach generalizes, it may shift the field's focus from improving SSL pseudo-label quality to obtaining better forced alignments and phonetic annotations, possibly making speech tokenization more controllable and interpretable.
  • A testable extension is to apply PAST to a language with no phonetic dictionary, using only character-level CTC supervision from an existing ASR system; the paper's results suggest that even partial phonetic supervision (10% of batches) could yield competitive phonetic tokens.
  • The reason X-Codec's SISNR is far lower, which the paper attributes to missing point-wise metric loss, hints that reconstruction quality in hybrid tokenizers is sensitive to the exact loss composition; PAST's design may benefit from adding an explicit waveform-matching term.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces PAST, an end-to-end speech tokenizer that combines EnCodec-style waveform reconstruction with auxiliary CTC character and phoneme classification losses on the first RVQ codebook, replacing the SSL-distillation objectives used by previous hybrid tokenizers. A causal/streamable variant is also proposed. The authors evaluate PAST against SpeechTokenizer, X-Codec, EnCodec, and a HuBERT k-means baseline on phonetic metrics (PNMI, ABX, WER), reconstruction metrics (SISNR, ViSQOL, PESQ), and speech language modeling (sWUGGY), reporting that PAST outperforms the observed baselines on essentially all metrics. They additionally provide an ablation study of the transformer, auxiliary heads, and skip-connection dropout.

Significance. If the empirical claims hold, PAST is a useful contribution: it shows that direct phonetic supervision can replace SSL distillation in hybrid tokenization, it avoids a separate vocoder, it includes a streaming variant, and the authors open-source code, checkpoints, and samples. The ablation study, especially the skip-connection dropout analysis, is informative and goes beyond what is typical in tokenizer papers. However, the central comparison is undermined by a training/evaluation confound: PAST is trained on the same corpora and label types used for evaluation, while the baselines are used as fixed pretrained models. The significance of the reported gains therefore cannot be assessed without a controlled comparison or cross-corpus evaluation.

major comments (5)
  1. [Section 4.1 and Tables 1-3] PAST is trained on all LibriSpeech training subsets and the TIMIT training set, using character and phoneme labels, while the baseline tokenizers are not trained or fine-tuned on these labels. The evaluation metrics are computed on the same domains: PNMI and ABX on the TIMIT test set, WER on LibriSpeech test sets, and sWUGGY on LibriSpeech audio. This creates a direct confound: PAST's reported superiority across Tables 1-3 may reflect in-domain supervised training rather than a fundamental advantage of the proposed architecture. A controlled comparison, such as retraining or fine-tuning the baselines with the same auxiliary losses and labels, or evaluating PAST on a held-out corpus/language not used in training, is needed to support the paper's central claim.
  2. [Section 5.1] Mimi is excluded because the authors' reproduced results 'were far from being on par with the observed baselines,' but no reproduced numbers are reported. Since Mimi is a recent hybrid tokenizer and the paper claims superiority over 'existing evaluated baseline tokenizers,' the omission should be documented with concrete numbers, or at least the reproduced metrics should be included in an appendix or supplementary material.
  3. [Section 4.1 and Abstract/Introduction] The abstract and introduction claim that PAST eliminates the need for external pretrained models, yet Section 4.1 states that character-level alignments for CTC supervision are obtained using a pretrained Wav2Vec2 model. This is an external pretrained model used in the training pipeline, so the claim is overstated as written. The paper should either scope the claim to inference-time behavior or discuss this training-time dependence as a limitation.
  4. [Section 5.3 and Table 4] The text states that including the transformer 'improves sequence modeling and signal reconstruction, as reflected in the ABX and SISNR trends.' However, the relevant Table 4 rows show SISNR decreasing from 5.24 (row X V V, no transformer) to 4.84 (row V V V, with transformer), while only ABX improves. The claim about SISNR is therefore not supported by the table and should be corrected.
  5. [Tables 1-3] No error bars, confidence intervals, or significance tests are reported for any metric. Several reported margins are small (e.g., PNMI 0.75 vs. 0.72, sWUGGY Inter 71.8 vs. 67.9, sWUGGY OOV 57.5 vs. 55.4), so without variance estimates it is unclear whether these differences are meaningful. Reporting standard deviations across training seeds or bootstrap confidence intervals would substantially strengthen the evaluation.
minor comments (6)
  1. [Abstract] The link 'seepages.cs.huji.ac.il/adiyoss-lab/PAST' is missing a space after 'see' and should be formatted as a proper URL.
  2. [Section 4.1] The phrase 'To achieve this, we obtain character-level alignment for paired text transcriptions using pretrained Wav2Vec2' is logically disconnected from the preceding sentence about segmenting audio; obtaining alignments is not needed for 3-second segmentation, and the sentence should be rephrased.
  3. [Table 2] The entries for X-Codec (SISNR -7.12, ViSQOL 4.46, PESQ 3.33) are run together in the source text; ensure proper spacing and alignment in the camera-ready version.
  4. [Section 4.4.2] The WER description says training and validation were performed on LibriSpeech train-clean-100 and dev-clean, but PAST itself was trained on all LibriSpeech training subsets. Clarify that this description refers to the DASB probing model, not the tokenizer.
  5. [Equation (1)] The auxiliary loss weights are given as λ_ctc=12 and λ_phn=5 without any sensitivity analysis or explanation of how they were chosen; a sentence on this would help reproducibility.
  6. [Section 5.1] The terms 'top-line' and 'topline' are used inconsistently; choose one form and use it throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PAST's auxiliary labels are external and all reported metrics are computed on held-out data; self-citations are architectural context, not load-bearing.

full rationale

I examined the derivation chain for reductions by construction or load-bearing self-citation. PAST minimizes a weighted sum of EnCodec reconstruction loss, a CTC character loss, and a phoneme cross-entropy loss (Eq. 1). The evaluation metrics (PNMI, ABX, WER, SISNR, ViSQOL, PESQ, sWUGGY) are computed on held-out TIMIT and LibriSpeech test sets; none of these metrics is defined by the training objective, and no parameter is fitted to the evaluation outcome. The phoneme CE loss is related to PNMI in expectation, but the paper does not claim PNMI is the training loss, and the reported comparison against baselines is an empirical benchmark rather than a derivation. The paper's citations to prior work by overlapping authors (EnCodec, AudioGen, MusicGen, and related studies) are used for architecture and context, not as a uniqueness theorem or as the sole justification of the central claim. The Discussion's limitation about reliance on labeled phonetic data is a scope constraint, not a circular step. There is a legitimate external-validity concern that PAST is trained on the same domains and supervision types used for evaluation, which may advantage it over baselines, but that is an evaluation confound, not a circular reduction: the paper's equations do not make the reported quantities equal to its inputs by construction, and the metrics are measured on held-out data. Therefore no circular step meets the required evidence standard.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the auxiliary training signals and the evaluation protocol. The loss weights and sampling ratios are hand-chosen; the reuse of the EnCodec objective and the validity of the metrics are taken as given.

free parameters (5)
  • lambda_ctc = 12
    Weight of the CTC character loss in Eq. 1; chosen by hand in Section 4.2.
  • lambda_phn = 5
    Weight of the phoneme classification loss; chosen by hand in Section 4.2.
  • p_trns-only = 0.3
    Probability of feeding only the transformer output to the quantizer during training; tuned in Section 4.2 and studied in Table 5.
  • p_skip-only = 0.1
    Probability of feeding only the encoder skip-connection output; tuned in Section 4.2.
  • LS:T sampling ratio = 9:1
    Proportion of LibriSpeech vs TIMIT samples in a batch, controlling phoneme supervision coverage; set in Section 4.1.
assumptions (4)
  • domain assumption The EnCodec reconstruction objective L_EnCodec is a sufficient training signal for high-fidelity audio codecs.
    The paper reuses it without modification, relying on prior validation in [7].
  • domain assumption TIMIT phoneme labels and LibriSpeech character transcriptions are accurate ground truth.
    Used as supervised targets for the auxiliary heads; errors would propagate into the learned token space.
  • domain assumption Pretrained Wav2Vec2 forced alignments correctly split audio into 3-second segments and provide the right character sequence for each segment.
    Section 4.1 states: 'we obtain character-level alignment for paired text transcriptions using pretrained Wav2Vec2 model'.
  • domain assumption The evaluation metrics (PNMI, ABX, WER, SISNR, ViSQOL, PESQ, sWUGGY) are valid proxies for tokenizer quality.
    Adopted from prior literature without justification that they capture downstream usefulness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAST: Phonetic-Acoustic Speech Tokenizer." pith.science (2026). https://pith.science/paper/QPVGYJV3

@misc{pith2026250514470,
  author       = {Pith},
  title        = {Pith review of: PAST: Phonetic-Acoustic Speech Tokenizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QPVGYJV3}},
  note         = {Machine review of arXiv:2505.14470}
}
read the original abstract

We present PAST, a novel end-to-end framework that jointly models phonetic information alongside signal reconstruction, eliminating the need for external pretrained models. Unlike previous approaches that rely on pretrained self-supervised models, PAST employs supervised phonetic data, directly integrating domain knowledge into the tokenization process via auxiliary tasks. Additionally, we introduce a streamable, causal variant of PAST, enabling real-time speech applications. Results demonstrate that PAST surpasses existing evaluated baseline tokenizers across common evaluation metrics, including phonetic representation and speech reconstruction. Notably, PAST also achieves superior performance when serving as a speech representation for speech language models, further highlighting its effectiveness as a foundation for spoken language generation. To foster further research, we release the full implementation. For code, model checkpoints, and samples see: https://pages.cs.huji.ac.il/adiyoss-lab/PAST

Figures

Figures reproduced from arXiv: 2505.14470 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 28 canonical work pages

  1. [1]

    These models usually operate over acoustic tokens or phonetic speech tokens (also known as semantic tokens)

    Introduction Speech and audio language models have recently attracted sig- nificant attention in the research community, showcasing re- markable performance across a range of tasks [1–6]. These models usually operate over acoustic tokens or phonetic speech tokens (also known as semantic tokens). Acoustic tokenizers, such as EnCodec [7] and SoundStream [8]...

    work page

  2. [2]

    We propose a novel approach for jointly learning phonetic and acoustic representations using supervised data, eliminat- ing the need for pretrained models or external vocoders

    work page

  3. [3]

    We achieve superior performance compared to hybrid to- kenizers across both phonetic and acoustic benchmarks., demonstrating the effectiveness of our approach

    work page

  4. [4]

    We introduce a streaming-compatible variant of PAST that operates causally, ensuring that it relies only on previous con- text, making it suitable for real-time speech applications

    work page

  5. [5]

    We open-source our implementation, including training and inference pipelines in addition to model checkpoints

    work page

  6. [6]

    Acoustic tokenizersaim to compress speech into discrete rep- arXiv:2505.14470v2 [cs.SD] 4 Jun 2025 resentations optimized for high-fidelity reconstruction

    Related Work Approaches to tokenization can be roughly categorized into phonetic tokenizers, acoustic tokenizers, and hybrid-tokenizers. Acoustic tokenizersaim to compress speech into discrete rep- arXiv:2505.14470v2 [cs.SD] 4 Jun 2025 resentations optimized for high-fidelity reconstruction. En- Codec [7] and SoundStream [8] have set the foundation for st...

    arXiv 2025

  7. [7]

    Problem Setup Our model is composed of three main components: Encoder, Quantizer, and Decoder

    Method 3.1. Problem Setup Our model is composed of three main components: Encoder, Quantizer, and Decoder. Given a waveform signalx∈R fs·t of durationt[sec], sampled atf s[Hz], the encoder transformsx into a dense latent representationz∈R D×T . Here,T=f r ·t denotes the temporal resolution of the latent space, determined by the frame ratef r, andDrepresen...

    work page

  8. [8]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

Show all 48 references
  1. [9]

    Data We use all training subsets of LibriSpeech [29] and TIMIT [30] for our training set, yielding a total of965hours of raw audio

    Experimental Setup 4.1. Data We use all training subsets of LibriSpeech [29] and TIMIT [30] for our training set, yielding a total of965hours of raw audio. To improve training efficiency and avoid redundant padding, we sample3-second audio segments from each data sample. This ...

  2. [10]

    Baseline Comparison We compare PAST with two baseline hybrid models, Speech- Tokenizer and X-Codec, on both reconstruction and phonetic information metrics

    Results 5.1. Baseline Comparison We compare PAST with two baseline hybrid models, Speech- Tokenizer and X-Codec, on both reconstruction and phonetic information metrics. For context, we use EnCodec as an acous- tic topline, since its objective is purely signal fidelity with- o...

  3. [11]

    Discussion We introduce PAST, a unified phonetic-acoustic novel speech tokenizer that integrates supervised phonetic information into the tokenization process while maintaining high-fidelity recon- struction. Unlike existing approaches that rely on SSL models and external voco...

  4. [12]

    Generative spoken language model based on continuous word-sized audio tokens,

    R. Algayreset al., “Generative spoken language model based on continuous word-sized audio tokens,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing. Association for Computational Linguistics, 2023, pp. 3008– 3028

    2023

  5. [13]

    Textless acoustic model with self-supervised distillation for noise-robust expressive speech-to-speech transla- tion,

    M.-J. Hwanget al., “Textless acoustic model with self-supervised distillation for noise-robust expressive speech-to-speech transla- tion,”Findings of the Association for Computational Linguistics: ACL 2024, 2024

    2024

  6. [14]

    Audiolm: a language modeling approach to au- dio generation,

    Z. Borsoset al., “Audiolm: a language modeling approach to au- dio generation,”IEEE/ACM transactions on audio, speech, and language processing, vol. 31, pp. 2523–2533, 2023

    2023

  7. [15]

    Text-free prosody-aware generative spoken language modeling,

    E. Kharitonovet al., “Text-free prosody-aware generative spoken language modeling,”Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666–8681, 2022

    2022

  8. [16]

    On generative spoken language modeling from raw audio,

    K. Lakhotiaet al., “On generative spoken language modeling from raw audio,”Transactions of the Association for Computa- tional Linguistics, vol. 9, pp. 1336–1354, 2021

    2021

  9. [17]

    Textually pretrained speech language mod- els,

    M. Hassidet al., “Textually pretrained speech language mod- els,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  10. [18]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint: 2210.13438, 2022

    2022 arXiv

  11. [19]

    Pyramidcodec: Hierarchical codec for long-form music generation in audio domain,

    J. Chenet al., “Pyramidcodec: Hierarchical codec for long-form music generation in audio domain,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 4253– 4263

    2024

  12. [20]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  13. [21]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  14. [22]

    Analysing discrete self supervised speech representation for spoken language modeling,

    A. Sicherman and Y . Adi, “Analysing discrete self supervised speech representation for spoken language modeling,” inICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, p. 1–5

    2023

  15. [23]

    Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,”Advances in neural information processing systems, vol. 33, pp. 17 022– 17 033, 2020

    2020

  16. [24]

    Speechtok- enizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech language models,” inThe Twelfth International Conference on Learning Representa- tions, 2024

    2024

  17. [25]

    Codec does matter: Exploring the semantic short- coming of codec for audio language model,

    Z. Yeet al., “Codec does matter: Exploring the semantic short- coming of codec for audio language model,”arXiv preprint: 2408.17175, 2024

    2024 arXiv

  18. [26]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”Technical report, Kyutai, 2024

    2024

  19. [27]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chenet al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, p. 1505–1518, Oct. 2022

    2022

  20. [28]

    loss, denoted asL ctc =CTC(y|chars), is then applied to align the predicted sequence with the transcription target. Phoneme classification.The second auxiliary head is a simple linear projection that takes ˆz1 as input and outputs a distribu- tion over the set of all phonemesP...

  21. [29]

    Hifi- codec: Group-residual vector quantization for high fidelity audio codec,

    D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y . Zou, “Hifi- codec: Group-residual vector quantization for high fidelity audio codec,”arXiv preprint: 2305.02765, 2023

    2023 arXiv

  22. [30]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”Transactions on Machine Learning Research, 2023, featured Certification, Reproducibility Certifica- tion

    2023

  23. [31]

    Audiodec: An open-source streaming high- fidelity neural audio codec,

    Y .-C. Wuet al., “Audiodec: An open-source streaming high- fidelity neural audio codec,” inICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  24. [32]

    Funcodec: A funda- mental, reproducible and integrable open-source toolkit for neural speech codec,

    Z. Du, S. Zhang, K. Hu, and S. Zheng, “Funcodec: A funda- mental, reproducible and integrable open-source toolkit for neural speech codec,” inICASSP 2024-2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 591–595

    2024

  25. [33]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,

    S. Jiet al., “Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,” 2024

    2024

  26. [34]

    Scaling speech-text pre-training with synthetic interleaved data,

    A. Zeng, Z. Du, M. Liu, L. Zhang, S. Jiang, Y . Dong, and J. Tang, “Scaling speech-text pre-training with synthetic interleaved data,” arXiv preprint: 2411.17607, 2024

    2024 arXiv

  27. [35]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  28. [36]

    Last: Language model aware speech tokenization,

    A. Turetzky and Y . Adi, “Last: Language model aware speech tokenization,”arXiv preprint: 2409.03701, 2024

    2024 arXiv

  29. [37]

    Nast: Noise aware speech tokenization for speech language models,

    S. Messica and Y . Adi, “Nast: Noise aware speech tokenization for speech language models,”arXiv preprint: 2406.11037, 2024

    2024 arXiv

  30. [38]

    A systematic compar- ison of phonetic aware techniques for speech enhancement,

    O. Tal, M. Mandel, F. Kreuk, and Y . Adi, “A systematic compar- ison of phonetic aware techniques for speech enhancement,” in Interspeech 2022, 2022, pp. 1193–1197

    2022

  31. [39]

    Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,

    A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” inProceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376

    2006

  32. [40]

    Lib- rispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  33. [41]

    Timit acoustic-phonetic continuous speech corpus,

    J. Garofolo, L. Lamel, W. Fisher, J. Fiscus, D. Pallett, N. Dahlgren, and V . Zue, “Timit acoustic-phonetic continuous speech corpus,”Linguistic Data Consortium, 11 1992

    1992

  34. [42]

    Visqol v3: An open source production ready objec- tive speech and audio metric,

    M. Chinen, F. S. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “Visqol v3: An open source production ready objec- tive speech and audio metric,” in2020 twelfth international con- ference on quality of multimedia experience (QoMEX). IEEE, 2020, pp. 1–6

    2020

  35. [43]

    Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

    A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in2001 IEEE In- ternational Conference on Acoustics, Speech, and Signal Process- ing. Proceedings (C...

    2001

  36. [44]

    Evaluating speech features with the minimal- pair abx task: analysis of the classical mfc/plp pipeline,

    T. Schatzet al., “Evaluating speech features with the minimal- pair abx task: analysis of the classical mfc/plp pipeline,” inInter- speech 2013, 2013, pp. 1781–1785

    2013

  37. [45]

    Dasb - discrete audio and speech benchmark,

    P. Mousavi, L. D. Libera, J. Duret, A. Ploujnikov, C. Subakan, and M. Ravanelli, “Dasb - discrete audio and speech benchmark,” arXiv preprint: 2406.14294, 2024

    2024 arXiv

  38. [46]

    Audiogen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided audio generation,”arXiv preprint: 2209.15352, 2023

    2023 arXiv

  39. [47]

    Simple and controllable music gen- eration,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. Défossez, “Simple and controllable music gen- eration,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  40. [48]

    The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language model- ing,

    T. A. Nguyenet al., “The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language model- ing,”Self-Supervised Learning for Speech and Audio Processing Workshop @ NeurIPS, 2020

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.