Pith. sign in

REVIEW 3 major objections 5 minor 57 references

DeCodec claims one audio codec can learn separately controllable speech and background-sound streams, with speech further split into semantic and paralinguistic codes, so that swapping quantized streams alone yields enhancement, voice conve

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:31 UTC pith:HTQCGNCQ

load-bearing objection A genuinely new codec architecture with a solid joint ablation for decoupling, but the theoretical guarantee is invalid and the background-extraction claim is undermined by the model's own negative SDR-B numbers. the 3 major comments →

arxiv 2509.09201 v1 pith:HTQCGNCQ submitted 2025-09-11 cs.SD

DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners

Xiaoxue Luo , Jinwei Huang , Runyan Yang , Yingying Gao , Junlan Feng , Chao Deng , Shilei Zhang This is my paper
classification cs.SD
keywords neural audio codecrepresentation disentanglementorthogonal subspacesspeech-background decouplingspeech enhancementvoice conversionnoise-robust ASRcontrollable TTS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a neural audio codec can be trained to factor a mixed audio signal into orthogonal quantized streams: one for speech, one for background sound, and, within speech, one semantic and one residual paralinguistic stream. If true, a single codec front-end can replace separate speech-separation, enhancement, and feature-extraction pipelines. The authors show that by selecting and recombining these streams, one trained model can reconstruct audio, suppress or isolate background sound, perform one-shot voice conversion on noisy speech, deliver noise-robust features to an ASR model, and let a TTS model keep or drop background sound. They report reconstruction quality comparable to standard codecs while adding these capabilities, and they position DeCodec as the first codec to integrate all of them.

Core claim

The paper's central claim is that the codec's encoder embedding Y of a mixture y=s+n can be split, by two trained linear projections, into orthogonal subspaces S and N, with S further quantized into a semantic code Zc and a residual paralinguistic code Zr. The claimed mechanism is a subspace orthogonal projection (SOP) module that enforces S⊥N, a representation swap training (RST) procedure that reconstructs s1+n2 from swapped codes and thereby pushes the speech stream to drop background content, and semantic guidance that aligns the first speech quantizer with a self-supervised speech representation. On this view the quantized streams are individually controllable: dropping N gives enhanced

What carries the argument

The machinery is the combination of a subspace orthogonal projection (SOP) module, a representation swap training (RST) procedure, and parallel residual vector quantizers with semantic guidance (SG). SOP applies two trainable linear projections to the encoder embedding, with an orthogonality loss enforcing ⟨S,N⟩≈0 so the embedding space decomposes as VS⊕VN. RST takes two mixtures s1+n1 and s2+n2, reconstructs s1+n2 from the swapped quantized speech and background codes, and trains with an L1 loss on that hybrid; the paper argues this forces the speech stream to contain no background information and vice versa. The speech quantizer's first layer is aligned with a semantic representation, sepa

Load-bearing premise

The load-bearing premise is that the orthogonality loss plus the swap-reconstruction loss provably forces the speech quantized stream to contain no background-sound information and the background stream no speech information, and that the first speech quantizer carries semantics rather than speaker traits; the paper's argument for this relies on an unverified covariance condition and a mean-value-theorem step that is not valid as written.

What would settle it

Train or take a fixed DeCodec and construct two mixtures s1+n1 and s1+n2 with identical speech but different backgrounds; if the speech stream truly contains no n1 information, swapping only background codes must leave the recovered speech content unchanged. A falsifying observation would be that swapping background codes changes the decoded speech's WER or speaker similarity beyond what the residual speech code can carry. A second, more direct check: on held-out embeddings, compute the norm of the product of the projection matrices PS^T PN; if it is not near zero, the claimed orthogonal proje

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • One trained codec can perform audio reconstruction, speech enhancement, background-sound extraction, one-shot voice conversion, and VC with noise removal by selecting which quantized streams to keep or swap.
  • Speech enhancement from representation recombination reaches DNSMOS scores above or matching dedicated enhancement models, including on real recordings.
  • ASR models trained on DeCodec semantic tokens achieve lower word error rates on noisy speech than those trained on tokens from a cascade of a denoiser plus a speech tokenizer.
  • A TTS model built on DeCodec tokens inherits controllable preservation or suppression of input background sound without additional fine-tuning.
  • Because decoupling happens in the representation domain, downstream tasks avoid the distortion and error accumulation of a separate time-domain separation front-end.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's formal proof that RST forces Zs to be background-free is not airtight: the mean-value-theorem step's Jacobian evaluation point depends on Zs1, so the stated mathematical argument does not by itself rule out Zs carrying n1 information; the empirical ablations are what actually support the decoupling.
  • The orthogonality argument also assumes the encoder's embedding covariance has a special 'angular matrix' form so that ⟨S,N⟩=0 implies the projection matrices satisfy PSPN^T=0; if that condition fails on held-out inputs, the claimed universal guarantee reduces to a property of the training distribution.
  • A natural stress test would move beyond additive, mutually independent background noise to reverberation, overlapping speech, music, or nonstationary noise; the additive model in the paper may not generalize to those.
  • If the disentanglement holds broadly, one shared codec tokenizer could serve as the common front-end for speech recognition, synthesis, enhancement, and audio editing, eliminating task-specific feature extractors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DeCodec proposes a neural audio codec that jointly disentangles speech versus background sound and, within speech, semantic versus paralinguistic content. The architecture combines a subspace orthogonal projection (SOP) block, a representation swap training (RST) procedure, and semantic guidance (SG) on a parallel residual vector quantizer. The paper claims that this single codec can serve as a universal front-end for reconstruction, speech enhancement, background sound extraction, one-shot voice conversion, noise-robust ASR, and controllable TTS. The empirical sections report competitive reconstruction numbers, strong DNSMOS for SE, improved ASR WER on noisy speech, and plausible subjective TTS scores. The central theoretical premise is that SOP plus the swap loss provably guarantees that the speech and background streams carry no cross-information.

Significance. If the disentanglement claims held as stated, DeCodec would be a valuable contribution: a single codec front-end that supports multiple downstream speech and audio tasks without task-specific separation or enhancement modules. The paper deserves credit for a broad evaluation, including standard metrics (DNSMOS, WER, SDR, subjective MOS), a causal/non-causal comparison, and an ablation study (Table 5) that isolates the contribution of the SOP module and RST procedure. The t-SNE visualization of semantic versus speaker information and the quantitative ASR/TTS results are useful empirical evidence. However, the theoretical proof that the swap loss enforces content independence is invalid as written, and the reported background-stream reconstruction quality (negative SDR-B in Table 5) directly conflicts with the strength of the paper's central claim. The empirical framework is promising, but the manuscript currently overstates the formal guarantees and the completeness of the disentanglement.

major comments (3)
  1. [§4.5, Eqs. (13)–(16)] The mean value theorem step does not prove that Zs is independent of n1. Subtracting Eq. (13) from Eq. (14) gives Dec(Zs1+Zn2) − Dec(Zs1+Zn1) ≈ n2−n1. The MVT then yields ∂Dec/∂Zn evaluated at some ξ between Zn1 and Zn2, multiplied by (Zn2−Zn1). The evaluation point ξ generally depends on Zs1 through the Jacobian, so the left-hand side is not independent of Zs1. Even if the Jacobian were independent of Zs1, equality of the difference to (n2−n1) would not imply that Dec(Zs1+Zn2) itself is independent of Zs1. The argument, therefore, does not establish that Zs contains no background information or that Zn contains no speech information. This is load-bearing because the Abstract and Section 4.5 claim a provable disentanglement guarantee. Please either provide a valid formal argument or explicitly characterize the decoupling as an empirical property of the training objective.
  2. [§4.3, Eq. (6), and the 'angular matrix' condition] The claim that when the covariance matrix YY^T satisfies the 'angular matrix' condition we obtain PSPT_N = 0 is not substantiated. The term 'angular matrix' is not defined, no proof is supplied, and no experiment verifies that the trained encoder's embeddings satisfy this condition. The L⊥ loss in Eq. (5) only penalizes the sample inner product ⟨S,N⟩; it does not by itself constrain the projection matrices PS and PN to be globally orthogonal. Figure 4 reports cosine similarities between the weight matrices of PS and PN, but that is a different statement from orthogonality of the induced subspaces, let alone statistical independence of the encoded content. Please clarify what condition is actually needed and verify or replace this step.
  3. [Table 5 and §6.1.2, background sound extraction] The reported SDR-B values for the background stream directly undermine the claimed capability of background sound extraction. In Table 5, Ablation-3 achieves SDR-B = 0.49 dB, and the final DeCodec achieves −1.11 dB (causal) and −0.36 dB (non-causal). An SDR of 0 dB is the score obtained by outputting silence; negative values mean the extracted background is worse than silence. The paper does not discuss this, provides no baseline comparison for background extraction, and offers no alternative metric or listening test to offset these numbers. Since the abstract claims 'orthogonal subspaces dedicated to speech and background sound' and the paper lists 'background sound extraction' as an enabled capability, this is an empirical gap in the central claim. Please address this directly, either with a more appropriate evaluation, a comparison against separation baselines, or a more limited claim
minor comments (5)
  1. [§4.6, loss weights] The loss weights list contains a likely typo: '150.0 for LSG, 10.0 for LSG' should presumably read '150.0 for LSG, 10.0 for L⊥' or similar. Please correct.
  2. [Figure numbering] Section 4.1 refers to 'Figure 3' for the system overview, but the overview figure appears as Figure 2, while Figure 3 is the demo spectrograms. Please renumber figures consistently.
  3. [§6.1.4, Figure 6/Figure 5] The text says 'we plotted the semantic representation Zc and residual paralinguistic representation Zr ... with Figure 6(a) corresponding to Zc' but the displayed t-SNE figure is labeled Figure 5. Please correct the cross-reference.
  4. [Throughout] There are numerous typos and grammatical slips: 'nN' (Section 4.4), 'reprezentaions', 'Follwing', 'correlate', 'Distangled' (Index Terms), 'SOP vlock', 'Aound 700', 'denoing', 'tpyes', 'guidlines'. A careful proofread is needed before resubmission.
  5. [§5.2, Table 2] The table reports SDR on noisy speech but the meaning of SDR for a mixture target is not immediately obvious. Please state explicitly how SDR is computed for the noisy reconstruction condition (target is the full mixture y = s+n, presumably).

Circularity Check

0 steps flagged

No significant circularity: DeCodec's decoupling is supervised by explicit training losses and its headline results are produced with separately trained downstream models on external benchmarks.

full rationale

DeCodec's central decoupling claim is not circular. The speech/background decomposition is directly induced by the two proposed losses, L⊥ (Eq. 5) and LRST (Eq. 12), and is then evaluated on held-out data: DNS Challenge test sets for enhancement, LibriSpeech test-clean plus DNS noise for codec/VC/ASR, and subjective listening for TTS. The semantic/paralinguistic split is anchored to an external HuBERT-L9 representation through Eq. 7, and the downstream ASR and TTS models are trained separately on clean LibriTTS, so the reported WER/MOS improvements are not produced by parameters fitted to those same evaluation targets. The theoretical argument in Section 4.5 is mathematically unsound as written (the mean-value theorem step does not establish that Zs1 is independent of n1, and the 'angular matrix' condition in Section 4.3 is unproven), but an invalid proof is a correctness problem, not a circular reduction; the paper does not disguise a fitted parameter as a prediction. Self-citations appear (e.g., [23], [31], [48]) but only for standard background statements such as speech/background independence, and they are not load-bearing. The negative SDR-B values in Table 5 are an empirical weakness in the background-extraction claim, but that is a failure of the claimed capability, not a circularity in the derivation chain.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on a constructed decomposition assumption (Eqs. 1-3), an unverified covariance condition used to argue that L_orth yields true orthogonal projections (Section 4.3), and an invalid independence inference in the swap-training derivation (Section 4.5). Free parameters are hand-chosen loss weights, codebook sizes, and the SNR mixing range. No new physical entities are introduced; SOP, RST, and SG are mechanisms, not entities.

free parameters (5)
  • Loss weights = LRST=500, LSG=150, L_orth=? (LSG listed twice), Lrec=10, Ldisc=1, cb=1, commit=10
    Chosen by hand in Section 4.6; the balance between LRST and L_orth controls decoupling strength, and the typo prevents exact replication.
  • Number of RVQ quantizers = 8 speech + 8 noise (Ks=Kn=8)
    Design choice in Section 5.5; sets bitrate (4.0+4.0 kbps) and the semantic/paralinguistic boundary Zc=Qs1.
  • Codebook size and dimension = 1024 codes, dimension 1024
    Fixed to match baselines (Sections 5.3, 5.5); affects capacity and rate.
  • Training SNR mixing range = -5 to 40 dB
    Random mixing range chosen by hand (Section 5.1.1); generalization to SNR outside this range is untested.
  • Subspace output dimension = 1024 for both PS and PN
    Chosen in Section 5.5 to match encoder dimension D=1024.
axioms (5)
  • domain assumption Speech s and background sound n are mutually independent, so y = s + n (Eq. 1)
    Justified by physical generation mechanisms with citation [31]; needed for the orthogonal subspace decomposition to be well-posed.
  • domain assumption The mixture embedding space is the direct sum of speech and background subspaces, VY = VS ⊕ VN (Eq. 3)
    Assumes the encoder representation supports such a decomposition; it is constructed by the SOP module, not proven.
  • ad hoc to paper Encoder covariance YY^T 'satisfies the angular matrix' so that L_orth (Eq. 5) implies PSPT_N = 0
    The condition is named in Section 4.3 but never defined or verified; without it the orthogonality loss does not guarantee orthogonal projection matrices.
  • ad hoc to paper The mean value theorem step (Eqs. 15-16) implies content independence of Zs and Zn
    The inference is invalid as stated: the Jacobian's dependence on Zs1 through the evaluation point does not imply Zs1 carries noise information.
  • domain assumption HuBERT-L9 features of clean speech are a valid semantic supervision target for Zc
    Standard practice inherited from SpeechTokenizer [16]; external to DeCodec and well-grounded.

pith-pipeline@v1.3.0-alltime-deepseek · 16780 in / 22720 out tokens · 218392 ms · 2026-08-04T19:31:27.684900+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners." pith.science (2026). https://pith.science/paper/HTQCGNCQ

@misc{pith2026250909201,
  author       = {Pith},
  title        = {Pith review of: DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTQCGNCQ}},
  note         = {Machine review of arXiv:2509.09201}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Universal audio codecs learn entangled representations across audio types, whereas some specific codecs offer decoupled representations but are limited to speech. Real-world audio, however, often contains mixed speech and background sounds, and downstream tasks require selective access to these components. Therefore, we rethink the audio codec as a universal disentangled representation learner to enable controllable feature selection across different audio tasks. To this end, we introduce DeCodec, a novel neural codec that learns to decouple audio representations into orthogonal subspaces dedicated to speech and background sound, and within speech, representations are further decomposed into semantic and paralinguistic components. This hierarchical disentanglement allows flexible feature selection, making DeCodec a universal front-end for multiple audio applications. Technically, built upon a codec framework, DeCodec incorporates two key innovations: a subspace orthogonal projection module that factorizes the input into two decoupled orthogonal subspaces, and a representation swap training procedure that ensures these two subspaces are correlate to the speech and background sound, respectively. These allows parallel RVQs to quantize speech and background sound components independently. Furthermore, we employ semantic guidance to the speech RVQ to achieve semantic and paralinguistic decomposition. Experimental results show that DeCodec maintains advanced signal reconstruction while enabling new capabilities: superior speech enhancement and effective one-shot voice conversion on noisy speech via representation recombination, improved ASR robustness through clean semantic features, and controllable background sound preservation/suppression in TTS. Demo Page: https://luo404.github.io/DeCodecV2/

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 16 linked inside Pith

  1. [1]

    Speech conveys semantic Shilei zhang is the corresponding author

    INTRODUCTION In real-world scenarios, audio often consists of both speech and background sounds (BGS). Speech conveys semantic Shilei zhang is the corresponding author. content and speaker characteristics [1], while background sounds provide environmental cues that enhance realism [2]. Different audio tasks prioritize these elements differently: speech en...

    Pith/arXiv arXiv 2025

  2. [2]

    RELATED WORK 2.1. Audio codecs Audio codecs were originally used for signal compres- sion [20], and are now widely used as tokenizers for large model approaches due to discretizing and compressing audio features effectively [12, 13]. In 2021, [21] proposed Sound- Stream, which is a novel end-to-end neural audio codec that can efficiently compress speech, ...

    2021

  3. [3]

    Based on the differences in the physical generation mechanisms, it can be assumed that speech signal and background sound signal are mutually independent [31]

    SIGNAL MODEL AND PROBLEM FORMULATION Background sound is an additive interference to speech in the time domain, so the mixed signal can be written as y = s + n, (1) where y denotes the vector forms of mixed signal composed of clean speech s and background sound n. Based on the differences in the physical generation mechanisms, it can be assumed that speec...

  4. [4]

    METHODOLOGY 4.1. System Overview The proposed DeCodec consists of four main components: an encoder, a SOP block, a parallel residual vector quantizers (RVQs) with SG, and a decoder, as shown in Figure 3. The encoder is used to preliminarily convert the time-domain sig- nal y into an embedded representation Y based on the physi- cal properties of the audio...

  5. [5]

    front-end separation + back-end processing

    EXPERIMENTAL SETUP 5.1. Datasets 5.1.1. Training datasets DeCodec: Multilingual datasets were used at 16kHz sample rate, including Aishell3 [35], train-clean-100 and train-clean- 360 of LibriTTS [36], VCTK [37], WSJ1 and WSJ0 [38]. For background sound, the ESC-50 [39] and DNS-Noise [40] were used. Aound 700 of speech data were randomly selected and mixed...

  6. [6]

    RESULTS AND ANALYSIS In this section, we conduct two sets of experiments to verify the performance of the proposed DeCodec: 1) the audio pro- cessing tasks and their performance that DeCodec can achieve on its own, and 2) the performance of DeCodec as a feature extractor in downstream task models. 6.1. Performance on Decodec Since DeCodec is based on the ...

  7. [7]

    The experimental results confirm that the representations are sufficiently disentangled, enabling controllable feature selection tailored to diverse downstream tasks

    CONCLUSION This work presents DeCodec, reframing audio codecs as an universal disentangled representation learner to achieve hier- archical disentanglement for representing speech-background sound and semantic-paralinguistic. The experimental results confirm that the representations are sufficiently disentangled, enabling controllable feature selection ta...

  8. [8]

    Audiopalm: A large language model that can speak and listen,

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al., “Audiopalm: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Audit: Audio editing by follow- ing instructions with latent diffusion models,

    Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, et al., “Audit: Audio editing by follow- ing instructions with latent diffusion models,”Advances in Neural Information Processing Systems , vol. 36, pp. 71340–71357, 2023

    2023

  10. [10]

    Superm2m: Supervised and mixture-to-mixture co-learning for speech enhancement and noise-robust asr,

    Zhong-Qiu Wang, “Superm2m: Supervised and mixture-to-mixture co-learning for speech enhancement and noise-robust asr,” Neural Networks , vol. 188, pp. 107408, 2025

    2025

  11. [11]

    Preserving background sound in noise-robust voice conversion via multi-task learning,

    Jixun Yao, Yi Lei, Qing Wang, Pengcheng Guo, Ziqian Ning, Lei Xie, Hai Li, Junhui Liu, and Danming Xie, “Preserving background sound in noise-robust voice conversion via multi-task learning,” in ICASSP 2023-2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

    2023

  12. [12]

    Supervised speech separation based on deep learning: An overview,

    DeLiang Wang and Jitong Chen, “Supervised speech separation based on deep learning: An overview,” IEEE/ACM transactions on audio, speech, and lan- guage processing, vol. 26, no. 10, pp. 1702–1726, 2018

    2018

  13. [13]

    Complex spectral mapping for single-and multi-channel speech enhancement and robust asr,

    Zhong-Qiu Wang, Peidong Wang, and DeLiang Wang, “Complex spectral mapping for single-and multi-channel speech enhancement and robust asr,” IEEE/ACM transactions on audio, speech, and lan- guage processing, vol. 28, pp. 1778–1787, 2020

    2020

  14. [14]

    Speech enhancement with lstm recurrent neural networks and its application to noise- robust asr,

    Felix Weninger, Hakan Erdogan, Shinji Watanabe, Em- manuel Vincent, Jonathan Le Roux, John R Hershey, and Björn Schuller, “Speech enhancement with lstm recurrent neural networks and its application to noise- robust asr,” in International conference on latent vari- able analysis and signal separation . Springer, 2015, pp. 91–99

    2015

  15. [15]

    Deep neural networks for speech enhancement and speech recognition: A systematic review,

    Sureshkumar Natarajan, Syed Abdul Rahman Al- Haddad, Faisul Arif Ahmad, Raja Kamil, Mohd Khair Hassan, Syaril Azrad, June Francis Macleans, Sadiq H Abdulhussain, Basheera M Mahmmod, Nurbek Saparkhojayev, et al., “Deep neural networks for speech enhancement and speech recognition: A systematic review,” Ain Shams Engineering Journal, vol. 16, no. 7, pp. 103...

    2025

  16. [16]

    Attention-based latent features for jointly trained end-to-end automatic speech recognition with modified speech enhancement,

    Da-Hee Yang and Joon-Hyuk Chang, “Attention-based latent features for jointly trained end-to-end automatic speech recognition with modified speech enhancement,” Journal of King Saud University-Computer and Infor- mation Sciences, vol. 35, no. 3, pp. 202–210, 2023

    2023

  17. [17]

    Phonetic feature encoding in human superior temporal gyrus,

    Nima Mesgarani, Connie Cheung, Keith Johnson, and Edward F Chang, “Phonetic feature encoding in human superior temporal gyrus,” Science, vol. 343, no. 6174, pp. 1006–1010, 2014

    2014

  18. [18]

    Representation of temporal sound features in the human auditory cortex,

    Kirill V Nourski and John F Brugge, “Representation of temporal sound features in the human auditory cortex,” 2011

    2011

  19. [19]

    High fidelity neural audio compression,

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    High-fidelity au- dio compression with improved rvqgan,

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar, “High-fidelity au- dio compression with improved rvqgan,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  21. [21]

    Unicodec: Unified audio codec with single domain-adaptive codebook,

    Yidi Jiang, Qian Chen, Shengpeng Ji, Yu Xi, Wen Wang, Chong Zhang, Xianghu Yue, ShiLiang Zhang, and Haizhou Li, “Unicodec: Unified audio codec with single domain-adaptive codebook,” arXiv preprint arXiv:2502.20067, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, De- tai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al., “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” arXiv preprint arXiv:2403.03100 , 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Speechtokenizer: Unified speech tok- enizer for speech large language models,

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu, “Speechtokenizer: Unified speech tok- enizer for speech large language models,”arXiv preprint arXiv:2308.16692, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Moshi: a speech-text foun- dation model for real-time dialogue,

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour, “Moshi: a speech-text foun- dation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Dualcodec: A low-frame-rate, semantically-enhanced neural audio codec for speech generation,

    Jiaqi Li, Xiaolong Lin, Zhekai Li, Shixi Huang, Yuancheng Wang, Chaoren Wang, Zhenpeng Zhan, and Zhizheng Wu, “Dualcodec: A low-frame-rate, semantically-enhanced neural audio codec for speech generation,” arXiv preprint arXiv:2505.13000, 2025

    arXiv 2025

  26. [26]

    Brain plasticity under early auditory deprivation: evidence from congenital hearing-impaired people,

    Changxin Zhang, “Brain plasticity under early auditory deprivation: evidence from congenital hearing-impaired people,” Advances in Psychological Science , vol. 27, no. 2, pp. 278, 2019

    2019

  27. [27]

    A tutorial on mpeg/audio compression,

    Davis Pan, “A tutorial on mpeg/audio compression,” IEEE multimedia, vol. 2, no. 2, pp. 60–74, 1995

    1995

  28. [28]

    Soundstream: An end-to-end neural audio codec,

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transac- tions on Audio, Speech, and Language Processing , vol. 30, pp. 495–507, 2021

    2021

  29. [29]

    Hifi- codec: Group-residual vector quantization for high fi- delity audio codec,

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou, “Hifi- codec: Group-residual vector quantization for high fi- delity audio codec,” arXiv preprint arXiv:2305.02765 , 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Flow-vae vc: end-to-end flow framework with contrastive loss for zero-shot voice conversion,

    Le Xu, Rongxiu Zhong, Ying Liu, Huibao Yang, and Shilei Zhang, “Flow-vae vc: end-to-end flow framework with contrastive loss for zero-shot voice conversion,” in Proc. Interspeech 2023, 2023, pp. 2293–2297

    2023

  31. [31]

    Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrah- man Mohamed, “Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  32. [32]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Ad- vances in neural information processing systems , vol. 33, pp. 12449–12460, 2020

    2020

  33. [33]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  34. [34]

    Superb-sg: Enhanced speech processing universal per- formance benchmark for semantic and generative capa- bilities,

    Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy T Liu, Cheng-I Jeff Lai, Jiatong Shi, et al., “Superb-sg: Enhanced speech processing universal per- formance benchmark for semantic and generative capa- bilities,” arXiv preprint arXiv:2203.06849, 2022

    Pith/arXiv arXiv 2022

  35. [35]

    Rep- codec: A speech representation codec for speech tok- enization,

    Zhichao Huang, Chutong Meng, and Tom Ko, “Rep- codec: A speech representation codec for speech tok- enization,” arXiv preprint arXiv:2309.00169, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,

    Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng, “Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,” inICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 591–595

    2024

  37. [37]

    Semanticodec: An ul- tra low bitrate semantic audio codec for general sound,

    Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley, “Semanticodec: An ul- tra low bitrate semantic audio codec for general sound,” IEEE Journal of Selected Topics in Signal Processing , 2024

    2024

  38. [38]

    Sixty years of frequency-domain monaural speech en- hancement: From traditional to deep learning methods,

    Chengshi Zheng, Huiyong Zhang, Wenzhe Liu, Xiaoxue Luo, Andong Li, Xiaodong Li, and Brian CJ Moore, “Sixty years of frequency-domain monaural speech en- hancement: From traditional to deep learning methods,” Trends in Hearing , vol. 27, pp. 23312165231209913, 2023

    2023

  39. [39]

    A review of vector quantization techniques,

    A Vasuki and Ponnusamy Thangapandian Vanathi, “A review of vector quantization techniques,” IEEE Poten- tials, vol. 25, no. 4, pp. 39–47, 2006

    2006

  40. [40]

    Bertrand Russell, Principles of mathematics, Routledge, 2020

    2020

  41. [41]

    Neural dis- crete representation learning,

    Aaron Van Den Oord, Oriol Vinyals, et al., “Neural dis- crete representation learning,” Advances in neural infor- mation processing systems, vol. 30, 2017

    2017

  42. [42]

    Aishell-3: A multi-speaker mandarin tts corpus and the baselines,

    Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li, “Aishell-3: A multi-speaker mandarin tts corpus and the baselines,” arXiv preprint arXiv:2010.11567, 2020

    Pith/arXiv arXiv 2010

  43. [43]

    Lib- ritts: A corpus derived from librispeech for text-to- speech,

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu, “Lib- ritts: A corpus derived from librispeech for text-to- speech,” arXiv preprint arXiv:1904.02882, 2019

    Pith/arXiv arXiv 1904

  44. [44]

    The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,

    Christophe Veaux, Junichi Yamagishi, and Simon King, “The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,” in 2013 international conference orien- tal COCOSDA held jointly with 2013 conference on Asian spoken language research and evaluation (O- COCOSDA/CASLRE). IEEE, 2013, pp. 1–4

    2013

  45. [45]

    The design for the wall street journal- based,

    C Corpus, “The design for the wall street journal- based,” in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992 . Morgan Kaufmann Publishers, 1992, p. 357

    1992

  46. [46]

    Esc: Dataset for environmental sound classification,

    Karol J Piczak, “Esc: Dataset for environmental sound classification,” in Proceedings of the 23rd ACM interna- tional conference on Multimedia, 2015, pp. 1015–1018

    2015

  47. [47]

    The interspeech 2020 deep noise suppression challenge: Datasets, subjective test- ing framework, and challenge results,

    Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al., “The interspeech 2020 deep noise suppression challenge: Datasets, subjective test- ing framework, and challenge results,” arXiv preprint arXiv:2005.13981, 2020

    Pith/arXiv arXiv 2020

  48. [48]

    First stereo audio source separation evaluation campaign: data, algorithms and results,

    Emmanuel Vincent, Hiroshi Sawada, Pau Bofill, Shoji Makino, and Justinian P Rosca, “First stereo audio source separation evaluation campaign: data, algorithms and results,” in International Conference on Inde- pendent Component Analysis and Signal Separation . Springer, 2007, pp. 552–559

    2007

  49. [49]

    Robust speech recognition via large-scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28492–28518

    2023

  50. [50]

    Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler, “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” inICASSP 2021-2021 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 6493–6497

    2021

  51. [51]

    Inter-subnet: Speech enhancement with subband in- teraction,

    Jun Chen, Wei Rao, Zilin Wang, Jiuxin Lin, Zhiyong Wu, Yannan Wang, Shidong Shang, and Helen Meng, “Inter-subnet: Speech enhancement with subband in- teraction,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  52. [52]

    Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,

    Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann, “Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,” IEEE/ACM Transactions on Au- dio, Speech, and Language Processing , vol. 31, pp. 2724–2737, 2023

    2023

  53. [53]

    Selm: Speech enhancement using discrete tokens and language mod- els,

    Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, and Lei Xie, “Selm: Speech enhancement using discrete tokens and language mod- els,” in ICASSP 2024-2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11561–11565

    2024

  54. [54]

    Attention is all you need,

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Ad- vances in neural information processing systems , vol. 30, 2017

    2017

  55. [55]

    Polyspeech: Exploring unified multi- task speech models for competitiveness with single-task models,

    Runyan Yang, Huibao Yang, Xiqing Zhang, Tiantian Ye, Ying Liu, Yingying Gao, Shilei Zhang, Chao Deng, and Junlan Feng, “Polyspeech: Exploring unified multi- task speech models for competitiveness with single-task models,” arXiv preprint arXiv:2406.07801, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Neural codec lan- guage models are zero-shot text to speech synthesizers,

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al., “Neural codec lan- guage models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    Pith/arXiv arXiv 2023

  57. [57]

    Decoupled weight decay regularization,

    I Loshchilov, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017