Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

FreeCodec: A disentangled neural speech codec with fewer tokens

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a neural speech codec which splits speech into speaker, prosody, and content streams matches or beats far higher-bitrate codecs at roughly 57 tokens per second.

desk verdict A plausible three-stream speech codec with strong reported numbers, but the headline bitrate comparison omits the continuous speaker vector in the v1/v3 variants that carry the main results. read the letter →

arxiv 2412.01053 v3 pith:2M6XYWIR submitted 2024-12-02 cs.SD eess.AS

Youqiang Zheng , Weiping Tu , Yueteng Kang , Jie Chen , Yike Zhang , Li Xiao , Yuhong Yang , Long Ma This is my paper · ORCID
classification cs.SDeess.AS
keywords neuralspeechcodecdisentangledrepresentationlearninglow-bitratecodingvoiceconversionvectorquantizationspeakerembeddingprosodymodelingself-supervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FreeCodec claims that speech can be compressed far more efficiently if the codec first separates the signal into three distinct streams: a single global speaker vector, a low-frame-rate prosody stream, and a content stream. The paper reports that with roughly 57 tokens per second, about 0.45 kbps at 16 kHz, both its reconstruction models and its voice-conversion variant outperform existing codecs that use two to six times more bandwidth. The design is self-supervised: no phonetic, F0, or speaker labels are needed, though a pre-trained WavLM model supplies the semantic target for the content stream. If the claim holds, fewer tokens per second means shorter sequences for speech language models and practical ultra-low-bitrate speech transmission.

What carries the argument

The load-bearing mechanism is the three-branch encoder decomposition with dedicated quantizers. A pre-trained ECAPA-TDNN speaker encoder extracts one global timbre vector; a strided convolutional content encoder, following the SuperCodec encoder with strides (2, 4, 5, 8), outputs 256-dimensional content features at 50 Hz; and a prosody encoder built from two convolution stacks plus a stride-8 max pooling layer, taking the first 20 mel-spectrogram bins, outputs prosody features at about 7 Hz. Content and prosody are quantized with plain one-codebook vector quantizers of size 256, while speaker is continuous in FreeCodec-v1 and v3 and group vector quantized in FreeCodec-v2. The decoder adds a 4-layer Transformer content decoder and a ConvNeXt backbone before the mirrored upsampling decoder, and training combines reconstruction loss, VQ commitment loss, feature matching, adversarial loss, and a WavLM-based cosine similarity content loss.

What would settle it

Train a speaker-classification head on the quantized content tokens or quantized prosody tokens for held-out speakers and measure accuracy against chance. If accuracy stays well above chance, speaker information has leaked into streams that are supposed to carry only content or prosody, and the disentanglement claim fails; this is a direct, cheap test of the Section 2.2 factorization.

Watch

Extended reading notes

Core claim

The central discovery is that explicit disentanglement, rather than a single coupled latent, lets a codec spend its bit budget almost entirely on content and prosody while the speaker identity rides along as one cheap global vector. FreeCodec encodes speech with three parallel encoders: a pre-trained ECAPA-TDNN extracts one global timbre vector; a strided convolutional content encoder produces 50 Hz content features; and a prosody encoder with a stride-8 max pooling layer produces roughly 7 Hz prosody features. Only the content and prosody streams are vector-quantized, each with a single 256-entry codebook, while the speaker representation is kept continuous in the reconstruction and voice-conversion versions or group-quantized in the fully discrete version. The paper shows that this decomposition improves reconstruction quality at 0.45 kbps and, by swapping only the speaker vector, achieves the best speaker similarity in unseen-speaker voice conversion among all baselines tested.

Load-bearing premise

The load-bearing premise is that speech really separates into independent speaker, prosody, and content streams, and that the chosen encoders and losses keep speaker information from leaking into the quantized content and prosody tokens, as assumed in Section 2.2 and enforced only indirectly through the content loss in Section 2.5.

Editorial extensions

If this is right

  • At 0.45 kbps and 57 tokens per second, FreeCodec-v1 and v2 match or beat codecs operating at 0.9 to 3 kbps on UTMOS, STOI, WARP-Q, SECS, and MUSHRA scores, according to the paper's evaluations.
  • The disentangled representation enables zero-shot voice conversion by replacing only the speaker vector: FreeCodec-v3 at 0.45 kbps achieves the highest speaker similarity among all baselines in the LibriSpeech-to-VCTK conversion test.
  • The WavLM-based content loss is load-bearing: the paper's ablation shows that removing it drops all objective reconstruction metrics, especially UTMOS and STOI.
  • The fully discrete FreeCodec-v2 variant remains competitive with continuous-speaker systems, suggesting that a complete discrete token stream for speech is achievable at very low bitrate.
  • Because the content stream is supervised by a semantic target rather than phonetic labels, the same codec can be reused for reconstruction and generation tasks without retraining the tokenizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the token reduction from roughly 100 tokens per second in many codecs to 57 tokens per second would nearly halve the sequence length seen by speech language models, reducing autoregressive inference cost and context-window pressure.
  • Editorial inference: a direct stress test of the disentanglement claim would be a linear probe or speaker-classification head trained on the quantized content or prosody tokens; if speaker identity remains decodable there, the claimed separation is incomplete.
  • Editorial inference: because the content encoder is driven by a semantic target rather than phonetic labels, the approach points toward cross-lingual voice conversion, but the paper does not test that setting.
  • Editorial inference: the prosody stream's partial speaker-specific clustering shown in the t-SNE visualization suggests that a higher-capacity or explicitly speaker-invariant prosody encoder could further improve speaker transfer in voice conversion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. FreeCodec proposes a neural speech codec that factorizes speech into speaker, content, and prosody streams using separate encoders, dedicated quantizers, and an improved decoder. Three training variants are introduced: v1 (continuous speaker embedding) for reconstruction, v2 (GVQ-quantized speaker embedding) for low-bitrate coding, and v3 (continuous speaker embedding with content loss applied at the decoder) for voice conversion. The paper reports 57 tokens/s and 0.45 kbps and presents objective and MUSHRA evaluations on VCTK and LibriSpeech test-clean showing competitive or superior reconstruction versus TiCodec, DAC, SpeechTokenizer, FACodec, SemantiCodec, and WavTokenizer, plus voice-conversion results in which FreeCodec-v3 achieves the highest speaker similarity.

Significance. If the reported bitrate and disentanglement claims hold, FreeCodec is a useful contribution: it demonstrates that an explicit three-stream factorization of speech can be trained self-supervisedly, yields competitive reconstruction at ultra-low bitrate, and provides flexible variants for reconstruction and voice conversion. The paper deserves credit for including a MUSHRA subjective test, an ablation showing that the content loss matters, t-SNE visualizations of the learned representations, and comparisons against several strong baselines. The main caveats are the incomplete bitrate accounting for the continuous speaker embedding in v1/v3, the overbroad 'outperforms existing methods' phrasing relative to Table 1, and the incomplete evidence for full speaker disentanglement given the visible speaker clustering in the prosody representations.

major comments (3)
  1. [Section 2.3, Tables 1 and 3] The 0.45 kbps bandwidth figure accounts only for the quantized content and prosody codes in FreeCodec-v1/v3. The continuous ECAPA-TDNN speaker vector is fed directly to the decoder and must be transmitted or stored for reconstruction or conversion. A 192-dimensional float vector costs 6144 bits per utterance; for VCTK utterances of a few seconds this adds on the order of 1-2 kbps, placing the effective bitrate at or above the 1 kbps baselines. FreeCodec-v2, which quantizes the speaker embedding with GVQ, is not affected, but v1 and v3 carry the headline reconstruction and voice-conversion results. Please either include the speaker-vector bitrate in the reported bandwidth, or clearly state the side-information assumption and re-run the comparisons on an all-inclusive bitrate basis.
  2. [Section 4.1, Table 1] The statement that 'FreeCodec-v1 performs best or second-best in almost all objective metrics' is not supported on LibriSpeech Test-clean. There, FreeCodec-v1's WARP-Q (2.195) is worse than SpeechTokenizer (2.034), SemantiCodec (2.049), FACodec (2.170), and WavTokenizer (2.135), and its STOI (0.892) is below SpeechTokenizer (0.908). The abstract and Section 1 claim that the framework 'outperforms existing methods' in subjective and objective evaluation; this should be qualified to specific metrics and test sets, or the claim should be softened.
  3. [Section 4.2, Figure 2] The t-SNE for prosody representations shows visible speaker clustering, and the text concedes that 'prosodic information maintains partial speaker-specific characteristics.' This bears directly on the central disentanglement claim, since the speaker encoder is supposed to remove speaker information explicitly. Please quantify leakage (e.g., speaker-identification accuracy or mutual-information estimates on Zc and Zp) and state the implication for the voice-conversion and bitrate results, or revise the disentanglement claim accordingly.
minor comments (5)
  1. [Title and Abstract] The phrase 'fewer tokens' should be qualified: FreeCodec uses 57 tokens/s while TiCodec and DAC at 0.5 kbps use 50 tokens/s, so the advantage is lower bitrate per token and lower total bitrate, not universally fewer tokens per second.
  2. [Section 4.2] Please specify the voice-conversion inference procedure explicitly in Section 2 (source content and prosody plus target speaker embedding); the current text only describes it in one sentence in Section 4.2.
  3. [Section 3.1] The WavTokenizer baseline is a 24 kHz pretrained model and the bitrate is converted to an equivalent 16 kHz value; please clarify whether all evaluations were performed at 24 kHz or after resampling, and state the conversion formula.
  4. [Table 1] The ablation rows 'w/o Lcontent' are reported without variance or significance testing; adding multiple seeds or statistical tests would strengthen the claim that content loss causes the observed drops.
  5. [Section 2.2] The prosody encoder is described as producing a roughly 7 Hz feature with 256 dimensions; please state explicitly how the 57 tokens/s total is split between content (50 Hz) and prosody (7 Hz), and confirm that the prosody codebook operates at 7 tokens/s.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction: FreeCodec's central reconstruction and voice-conversion claims rest on external benchmarks; only minor non-load-bearing self-citations appear.

full rationale

FreeCodec is an empirical system paper, so its central claims are the benchmark numbers in Tables 1-3 and the MUSHRA results. These are evaluated against baselines that are retrained or run from official checkpoints, and the metrics (UTMOS, STOI, WARP-Q, SECS via Resemblyzer, Whisper WER/CER, F0-PCC) are external to the trained model. The WavLM-Large content loss and ECAPA-TDNN speaker encoder are training-side inputs, not evaluation metrics, so the content-loss objective does not define the reported reconstruction or conversion scores. The self/group citations (SuperCodec content encoder, FreeVC spectrogram-resize augmentation) provide architecture and training-procedure provenance; they are not invoked as evidence for a result, and no uniqueness theorem is imported to force the design. The continuous-speaker-vector omission from the 0.45 kbps bandwidth for v1/v3 is a fairness/correctness concern, but it is not a case where a predicted quantity is equivalent to an input by construction. No equation or training objective defines the claimed outcome in terms of itself. The score of 2 reflects only the presence of minor non-load-bearing self-citations, not any circular step.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result rests on borrowed pretrained models (ECAPA-TDNN, WavLM), architecture choices from prior work (SuperCodec encoder, Mega-TTS 2 prosody encoder, ConvNeXt decoder), and hand-set hyperparameters. No parameter-free derivation is attempted; the contribution is empirical. No new physical or conceptual entities are introduced.

free parameters (3)
  • Generator loss weights lambda_adv, lambda_feat, lambda_rec, lambda_vq, lambda_c = 3, 3, 1, 1, 10
    Hand-chosen hyperparameters balancing adversarial, feature, reconstruction, VQ commitment, and content losses; no sensitivity study is provided.
  • Content and prosody codebook size = 256
    Design choice for the plain vector quantizers; affects coding efficiency and reconstruction quality.
  • Speaker GVQ codebook size and number of groups = 1024 codes, 8 groups
    Design choice for quantizing the global speaker vector in FreeCodec-v2.
assumptions (4)
  • domain assumption ECAPA-TDNN speaker embedding provides a reliable global timbre representation
    Used as the speaker encoder in Section 2.2; if the embedding misses or confounds prosodic information, the speaker stream leaks.
  • domain assumption WavLM last-layer representation is a valid content or semantic learning target
    Used in Section 2.5 as the content loss target; if WavLM retains speaker information, the content encoder may inherit it.
  • domain assumption First 20 mel bins plus max pooling with stride 8 captures prosody with minimal speaker and content information
    Borrowed from Mega-TTS 2 in Section 2.2; underpins the 7 Hz prosody stream.
  • domain assumption Speech can be factorized into independent content, prosody, and speaker streams
    Core design premise in Section 2.1 and Figure 1; if real speech is more entangled, the disentanglement claims weaken.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FreeCodec: A disentangled neural speech codec with fewer tokens." pith.science (2026). https://pith.science/paper/2M6XYWIR

@misc{pith2026241201053,
  author       = {Pith},
  title        = {Pith review of: FreeCodec: A disentangled neural speech codec with fewer tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2M6XYWIR}},
  note         = {Machine review of arXiv:2412.01053}
}
read the original abstract

Neural speech codecs have gained great attention for their outstanding reconstruction with discrete token representations. It is a crucial component in generative tasks such as speech coding and large language models (LLM). However, most works based on residual vector quantization perform worse with fewer tokens due to low coding efficiency for modeling complex coupled information. In this paper, we propose a neural speech codec named FreeCodec which employs a more effective encoding framework by decomposing intrinsic properties of speech into different components: 1) a global vector is extracted as the timbre information, 2) a prosody encoder with a long stride level is used to model the prosody information, 3) the content information is from a content encoder. Using different training strategies, FreeCodec achieves state-of-the-art performance in reconstruction and disentanglement scenarios. Results from subjective and objective experiments demonstrate that our framework outperforms existing methods.

Figures

Figures reproduced from arXiv: 2412.01053 by the authors.

Figure 1
Figure 1. The architecture of FreeCodec and prosody encoder. In Fig.1(a), we show different versions of the FreeCodec training paradigm in different scenarios. In Fig.1(b), the prosody encoder removes the speaker information by a max pooling layer. respectively. Then the quantization layers produce compressed representations. Finally, the improved decoders, consisting of a content decoder, a backbone module, and an up-samplin… view at source ↗
Figure 2
Figure 2. (a)(b)(c) is the t-SNE visualization of speaker represen￾tations Zs, content representations Zc and prosody representa￾tions Zp after VQ. Different colors represent different speakers. 1https://github.com/resemble-ai/Resemblyzer 2https://huggingface.co/openai/whisper-large [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation

    eess.AS 2026-08 conditional novelty 7.0 of 10

    SemBridge supervises autoregressive states with discrete semantic tokens during training, improving content fidelity of continuous-latent speech generation without changing inference.

  2. MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation

    cs.SD 2025-05 conditional novelty 5.0 of 10

    A single-layer streaming Transformer codec with masked Gaussian noise injection during training reports state-of-the-art reconstruction and better downstream generation and understanding in 16 kHz English speech.

Reference graph

Works this paper leans on

45 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Compared to traditional parametric algorithms [1, 2], it has progressed significantly in medium- or low-bitrate scenarios

    Introduction Neural speech codecs are widely used to compress speech signals for a limited number of bits with minimal distortion. Compared to traditional parametric algorithms [1, 2], it has progressed significantly in medium- or low-bitrate scenarios. With the de- velopment of large language models (LLM), the discrete codes of neural speech codecs play ...

    work page

  2. [2]

    Overall As illustrated in Fig.1(a), our proposed method consists of three components: encoders, quantizers, and decoders

    Proposed Method 2.1. Overall As illustrated in Fig.1(a), our proposed method consists of three components: encoders, quantizers, and decoders. Unlike ex- isting works, our encoder proposes a more detailed modeling focus on different intrinsic properties in human speech. We in- troduce three types of encoders to encode content, speaker, and prosody (in add...

    arXiv 2025

  3. [3]

    Training Details and Baselines We trained our model on LibriSpeech [29], which consists of ap- proximately 1000 hours of speech at 16kHz

    Experimental Setup 3.1. Training Details and Baselines We trained our model on LibriSpeech [29], which consists of ap- proximately 1000 hours of speech at 16kHz. We use train-clean- 100, train-clean-360, and train-other-500 subsets for training. For a fair comparison, we adopt two recent neural codecs, TiCodec and Descript-audio-codec (DAC), which have de...

    work page

  4. [4]

    Reconstruction Quality Table 1 summarizes the results of objective reconstruction exper- iments

    Results 4.1. Reconstruction Quality Table 1 summarizes the results of objective reconstruction exper- iments. FreeCodec-v1 performs best or second-best in almost all objective metrics in test sets. Especially in out-of-domain envi- ronments, our proposed method achieves superior reconstruction performance using only approximately 57 tokens per second than...

    work page

  5. [5]

    We demonstrate that this framework can be applied to both re- construction and disentanglement tasks using different training strategies

    Conclusion In this paper, we propose a self-supervised disentanglement speech codec that factorizes speech into its intrinsic attributes. We demonstrate that this framework can be applied to both re- construction and disentanglement tasks using different training strategies. Compared to existing methods, our approach utilizes fewer tokens and lower bandwi...

    work page

  6. [6]

    62471343)

    Acknowledgement This paper is supported by the National Nature Science Founda- tion of China (No. 62471343)

    work page

  7. [7]

    Codec 2-open source speech coding at 2400 bits/s and below,

    D. Rowe, “Codec 2-open source speech coding at 2400 bits/s and below,” in TAPR and ARRL 30th Digital Communications Conference, 2011, pp. 80–84

    work page 2011

  8. [8]

    Melp: the new federal standard at 2400 bps,

    L. M. Supplee, R. P. Cohn et al., “Melp: the new federal standard at 2400 bps,” in 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2. IEEE, 1997, pp. 1591– 1594

    work page 1997

Show all 45 references
  1. [9]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  2. [10]

    Audiolm: a language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonovet al., “Audiolm: a language modeling approach to audio generation,” IEEE/ACM transactions on audio, speech, and language processing, vol. 31, pp. 2523–2533, 2023

    2023

  3. [11]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D´efossez, L. Mazar´e et al., “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  4. [12]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs et al. , “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2022

    2022

  5. [13]

    High fidelity neu- ral audio compression,

    A. D´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neu- ral audio compression,” arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  6. [14]

    Latent-domain predictive neural speech coding,

    X. Jiang, X. Peng, H. Xue, Y . Zhang, and Y . Lu, “Latent-domain predictive neural speech coding,” IEEE/ACM Transactions on Au- dio, Speech, and Language Processing, vol. 31, pp. 2111–2123, 2023

    2023

  7. [15]

    Supercodec: A neural speech codec with selective back-projection network,

    Y . Zheng, W. Tu et al. , “Supercodec: A neural speech codec with selective back-projection network,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 566–570

    2024

  8. [16]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,

    S. Ji, Z. Jiang et al., “Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,” in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https://openreview.net/forum?id=yBlVlS2Fd9

    2025

  9. [17]

    Neural discrete representation learning,

    A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 6309–6318

    2017

  10. [18]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang et al. , “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” in Forty-first International Conference on Machine Learning, 2024. [Online]. Available: https://openreview.net/forum?id=dVhrnjZJad

    2024

  11. [19]

    Speech Resynthesis from Discrete Dis- entangled Self-Supervised Representations,

    A. Polyak, Y . Adiet al., “Speech Resynthesis from Discrete Dis- entangled Self-Supervised Representations,” in Proc. Interspeech 2021, 2021, pp. 3615–3619

    2021

  12. [20]

    Fewer-token neural speech codec with time-invariant codes,

    Y . Ren, T. Wang, J. Yiet al., “Fewer-token neural speech codec with time-invariant codes,” in ICASSP 2024-2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 737–12 741

    2024

  13. [21]

    Speechtokenizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtokenizer: Unified speech tokenizer for speech language models,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  14. [22]

    Single-codec: Single-codebook speech codec towards high-performance speech generation,

    H. Li, L. Xue et al., “Single-codec: Single-codebook speech codec towards high-performance speech generation,” inInterspeech 2024, 2024, pp. 3390–3394

    2024

  15. [23]

    Semanticodec: An ultra low bitrate semantic audio codec for general sound,

    H. Liu, X. Xu et al., “Semanticodec: An ultra low bitrate semantic audio codec for general sound,” IEEE Journal of Selected Topics in Signal Processing, vol. 18, no. 8, pp. 1448–1461, 2024

    2024

  16. [24]

    Fireredtts: A foundation text-to- speech framework for industry-level generative speech applica- tions,

    H.-H. Guo, K. Liu et al. , “Fireredtts: A foundation text-to- speech framework for industry-level generative speech applica- tions,” arXiv preprint arXiv:2409.03283, 2024

    2024 arXiv

  17. [25]

    Lscodec: Low-bitrate and speaker-decoupled dis- crete speech codec,

    Y . Guoet al., “Lscodec: Low-bitrate and speaker-decoupled dis- crete speech codec,” arXiv preprint arXiv:2410.15764, 2024

    2024 arXiv

  18. [26]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharamanet al., “High-fidelity audio compression with improved rvqgan,” Advances in Neural Information Process- ing Systems, vol. 36, 2024

    2024

  19. [27]

    Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias,

    Z. Jiang, Y . Renet al., “Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias,” arXiv preprint arXiv:2306.03509, 2023

    2023 arXiv

  20. [28]

    Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” in Interspeech 2020, 2020, pp. 3830–3834

    2020

  21. [29]

    Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech,

    Y . Ren, M. Lei, Z. Huanget al., “Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7577–7581

    2022

  22. [30]

    Mega-TTS 2: Boosting prompting mechanisms for zero-shot speech synthesis,

    Z. Jiang, J. Liu, Y . Ren et al. , “Mega-TTS 2: Boosting prompting mechanisms for zero-shot speech synthesis,” in The Twelfth International Conference on Learning Representations ,

  23. [31]

    Yourtts: Towards zero-shot multi- speaker tts and zero-shot voice conversion for everyone,

    E. Casanova, J. Weber et al., “Yourtts: Towards zero-shot multi- speaker tts and zero-shot voice conversion for everyone,” inInter- national Conference on Machine Learning . PMLR, 2022, pp. 2709–2720

    2022

  24. [32]

    Promptcodec: High-fidelity neural speech codec using disentangled representation learning based adaptive feature-aware prompt encoders,

    Y . Pan, L. Ma, and J. Zhao, “Promptcodec: High-fidelity neural speech codec using disentangled representation learning based adaptive feature-aware prompt encoders,” arXiv preprint arXiv:2404.02702, 2024

    2024 arXiv

  25. [33]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 2022, pp. 11 976–11 986

    2022

  26. [34]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang et al., “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  27. [35]

    Freevc: Towards high-quality text-free one-shot voice conversion,

    J. Li, W. Tu, and L. Xiao, “Freevc: Towards high-quality text-free one-shot voice conversion,” inICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  28. [36]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal process- ing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  29. [37]

    Vqmivc: Vector quantization and mutual information-based unsupervised speech representation disentangle- ment for one-shot voice conversion,

    D. Wang, L. Denget al., “Vqmivc: Vector quantization and mutual information-based unsupervised speech representation disentangle- ment for one-shot voice conversion,” in Interspeech 2021, 2021, pp. 1344–1348

    2021

  30. [38]

    Warp-q: Quality prediction for generative neural speech codecs,

    W. A. Jassim, J. Skoglund et al., “Warp-q: Quality prediction for generative neural speech codecs,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE, 2021, pp. 401–405

    2021

  31. [39]

    Wav2vec-vc: V oice conversion via hidden representations of wav2vec 2.0,

    J. Lim and K. Kim, “Wav2vec-vc: V oice conversion via hidden representations of wav2vec 2.0,” inICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 326–10 330

    2024

  32. [40]

    Cstr vctk corpus: English multi- speaker corpus for cstr voice cloning toolkit (version 0.92),

    J. Yamagishi, C. Veaux et al., “Cstr vctk corpus: English multi- speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019

    2019

  33. [41]

    1534-1, method for the subjective assessment of inter- mediate quality levels of coding systems (mushra),

    R. ITU-R, “1534-1, method for the subjective assessment of inter- mediate quality levels of coding systems (mushra),”,”International Telecommunication Union, 2003

    2003

  34. [42]

    Speex: A free codec for free speech,

    J.-M. Valin, “Speex: A free codec for free speech,” arXiv preprint arXiv:1602.08668, 2016

    2016 arXiv

  35. [43]

    Utmos: Utokyo-sarulab system for voicemos chal- lenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos chal- lenge 2022,” in Interspeech 2022, 2022, pp. 4521–4525

    2022

  36. [44]

    A short-time objective intelligibil- ity measure for time-frequency weighted noisy speech,

    C. H. Taal, Hendriks et al., “A short-time objective intelligibil- ity measure for time-frequency weighted noisy speech,” in 2010 IEEE international conference on acoustics, speech and signal processing. IEEE, 2010, pp. 4214–4217

    2010

  37. [2024]

    Available: https://openreview.net/forum?id= mvMI3N4AvD

    [Online]. Available: https://openreview.net/forum?id= mvMI3N4AvD

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.