Pith. sign in

REVIEW 2 cited by

ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.15897 v2 pith:3K3UJ32W submitted 2024-09-24 eess.AS cs.SD

Jiatong Shi , Jinchuan Tian , Yihan Wu , Jee-weon Jung , Jia Qi Yip , Yoshiki Masuyama , William Chen , Yuning Wu
show 12 more authors
Yuxun Tang Massa Baali Dareen Alharhi Dong Zhang Ruifan Deng Tejes Srivastava Haibin Wu Alexander H. Liu Bhiksha Raj Qin Jin Ruihua Song Shinji Watanabe
This is my paper · ORCID
classification eess.AScs.SD
keywords evaluationespnet-codecaudiotrainingapplicationscodeccodecsneural
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream applications are investigated, challenges have arisen in ensuring fair comparisons across diverse applications. To address these issues, we present a new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on neural codec training and evaluation. ESPnet-Codec offers various recipes in audio, music, and speech for training and evaluation using several widely adopted codec models. Together with ESPnet-Codec, we present VERSA, a standalone evaluation toolkit, which provides a comprehensive evaluation of codec performance over 20 audio evaluation metrics. Notably, we demonstrate that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse applications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training

    cs.SD 2025-06 conditional novelty 6.0 of 10

    Using Mimi neural codec features with label-delayed training reduces endpoint cutoff errors by 42.7% (single-stream) and 37.5% (two-stream) at 160 ms median latency.

  2. Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Staged ASR-to-text-response-to-TTS training makes open end-to-end spoken dialogue systems trainable on 300 hours of public human-human data and more coherent than one-step speech-to-speech models.

Pith tools