Pith. sign in

REVIEW 3 major objections 6 minor 36 references

DS-Codec: Dual-Stage Training with Mirror-to-NonMirror Architecture Switching for Speech Codec

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two-stage mirror switch tops prior 1 kbps speech codecs.

desk verdict Interesting mirror-to-nonmirror training recipe, but the headline comparison is contaminated by BigCodec checkpoint initialization. read the letter →

arxiv 2505.24314 v1 pith:2IV7YDOK submitted 2025-05-30 cs.SD eess.AS

Peijie Chen , Wenhao Guan , Kaidi Wang , Weijie Wu , Hukai Huang , Qingyang Hong , Lin Li This is my paper
classification cs.SDeess.AS
keywords neuralspeechcodecsinglecodebookmirror-to-nonmirrorarchitectureswitchingdual-stagetrainingproductquantizationvectorreconstructionlow-bitratecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DS-Codec is a neural speech codec that compresses speech to roughly 1 kbps using a single codebook, and the paper's central claim is that its two-stage training recipe reconstructs speech more faithfully than previous codecs at that bitrate. In the first stage the encoder, quantizer, and a mirrored decoder are trained together; in the second stage the mirror is broken, a transformer block is inserted, and the decoder is fine-tuned while the encoder and quantizer stay frozen. The paper argues that the mirrored first stage builds a more robust codebook, while the non-mirrored second stage lets a stronger decoder exploit that codebook for high-fidelity reconstruction. On LibriSpeech test-clean, the model reports UTMOS 4.218 and PESQ 2.882 for DS-Codec-VQ at 1.04 kbps, above the BigCodec baseline it builds on and above other single-codebook codecs. If the claim holds, the recipe cleanly separates codebook learning from decoder power and gives TTS systems a better single-stream tokenizer.

What carries the argument

The central mechanism is the two-stage training switch from a mirrored to a non-mirrored architecture. In stage 1, the encoder and a decoder that mirrors it are trained jointly with the quantizer, which the paper argues forces the codebook to capture information salient to reconstruction. In stage 2, the encoder and quantizer are frozen, a transformer block (residual attention and SwiGLU with RMSNorm) is added, and the decoder is fine-tuned under a non-mirrored, upsampling layout with reinitialized discriminators. The study treats the mirror as a codebook-robustness device and the stage-2 decoder as a reconstruction-fidelity device; the VQ module builds on L2-normalized low-dimensional vector quantization with 8,192 codes, and the PQ variant builds a 65,536-entry effective codebook from four 16-code sub-codebooks.

What would settle it

Train the same DS-Codec architecture with identical total compute but run stage 2 with the encoder and quantizer unfrozen, or train the whole model end-to-end in one stage. If the resulting UTMOS and PESQ on LibriSpeech test-clean are statistically indistinguishable from the reported 4.214 and 2.882, the mirror-to-nonmirror switch is not the cause of the gains. Alternatively, if codebook utilization or perplexity collapses once the transformer decoder is attached to the frozen codebook, the load-bearing assumption is violated.

Watch

Extended reading notes

Core claim

The paper's central claim is that a newly introduced training strategy, mirror-to-nonmirror architecture switching, lets a single-codebook speech codec outperform prior neural speech codecs on objective reconstruction metrics. The model, DS-Codec, starts with a mirrored encoder-decoder pair and jointly trains encoder, quantizer, and decoder; it then replaces the mirror by freezing encoder and quantizer and fine-tuning the decoder together with a newly added transformer block under a non-mirrored layout. The authors report that this two-stage scheme beats single-stage joint training on the same architecture, and that the mirrored first stage yields a smaller gap between the quantizer's input and output, which they read as evidence of a more robust codebook. With a vector-quantization variant (DS-Codec-VQ) and a product-quantization variant (DS-Codec-PQ), the codec reports top UTMOS, PESQ, STOI, and F1 scores on LibriSpeech test-clean and on 2,000 LJSpeech utterances, including higher PESQ than BigCodec (2.862 vs 2.681) at the same 1.04 kbps bitrate.

Load-bearing premise

The design assumes that the codebook learned jointly with a mirrored decoder remains the right quantization target after the architecture switches to a non-mirrored decoder with a transformer, so freezing the encoder and quantizer in stage 2 is the best choice.

Editorial extensions

If this is right

  • Single-codebook codecs at about 1 kbps can reach UTMOS and PESQ levels that previously required multi-codebook or higher-bitrate systems, simplifying TTS tokenization to one stream.
  • The stage-1 mirrored pretraining can be reused as a warm start: BigCodec's official checkpoint is used directly for DS-Codec-VQ's second stage, so the strategy is compatible with existing codec checkpoints.
  • The two-stage recipe is architecture-agnostic in principle: any mirrored codec can be converted to non-mirrored in stage 2, so the gains may transfer to other encoder and decoder families.
  • The transformer block contributes beyond the decoder alone, since DS-Codec-PQ with the stage-2 transformer (UTMOS 4.214) outperforms stage-2 without it (UTMOS 4.195), showing the non-mirrored decoder and transformer jointly matter.
  • On unseen LJSpeech data, the model retains its advantage, suggesting the training strategy generalizes across speakers and datasets rather than overfitting LibriSpeech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger test of the 'robust codebook' claim would be measuring codebook perplexity, utilization, or downstream TTS quality, which the paper does not report; the quantizer input-output MSE gap is indirect evidence.
  • The frozen-encoder assumption could be probed by unfreezing encoder and quantizer in stage 2; if equal-compute training without freezing matches the reported scores, the benefit would come from longer training rather than the mirror switch itself.
  • Stage 2 reinitializes discriminators and changes batch size and learning rate, so the reported gains bundle several hyperparameter changes with the architectural switch; a controlled ablation holding all else fixed would isolate the mirror contribution.
  • The codec is evaluated only on read English speech at 16 kHz; extending to noisy, multilingual, or higher-sample-rate speech would test whether the mirrored-codebook benefit survives domain shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DS-Codec, a single-codebook neural speech codec trained with a dual-stage strategy: stage 1 jointly trains a mirrored encoder, quantizer, and decoder, and stage 2 freezes the encoder and quantizer, adds a Transformer block, switches to a non-mirrored decoder, and fine-tunes the decoder and discriminator. Two variants are presented, DS-Codec-VQ (vector quantization) and DS-Codec-PQ (product quantization), and evaluated on LibriSpeech and LJSpeech against existing codecs such as BigCodec, WavTokenizer, and TiCodec. The paper claims that the mirrored stage produces more robust codebooks and that the overall strategy outperforms previous neural speech codecs at low bitrates.

Significance. If the claims are supported, the two-stage mirror-to-non-mirror training strategy is a useful contribution to single-codebook speech codec design, and the comparison of mirrored versus non-mirrored codebook learning in Section 3.5 is informative. The paper reports objective metrics on two corpora and includes an ablation against APCodec+'s staged training, which is a useful baseline. However, the manuscript provides no code, checkpoints, error bars, or significance tests, and one of the two headline comparisons in Table 1 uses a model initialized from BigCodec's official checkpoint, which prevents the reported gains from being attributed to the proposed training strategy. These issues are fixable but currently block acceptance.

major comments (3)
  1. [Section 3.2 and Table 1] The single-codebook variant matched to BigCodec's 1.04 kbps bitrate, DS-Codec-VQ, is initialized from BigCodec's official checkpoint and then only its decoder (and discriminator) are fine-tuned. The 0.110 UTMOS and 0.181 PESQ improvements over BigCodec could therefore come from continued training, the addition of the Transformer block, or the increased stage-2 training time, and cannot be attributed to the mirror-trained codebook. The independently trained DS-Codec-PQ variant avoids this issue but runs at 1.28 kbps, 23% higher than BigCodec. Consequently, the central claim that DS-Codec 'outperforms previous neural speech codecs' at roughly 1 kbps is not supported by the present evidence. Please provide a from-scratch matched-bitrate comparison or clearly restrict the claim to the higher-bitrate DS-Codec-PQ variant.
  2. [Section 3.5 and Figure 2] The claim that the mirrored structure 'significantly enhances the robustness of the learned codebooks' is based on a single training run without error bars, confidence intervals, or multiple seeds. The differences in VQ loss and MSE loss shown in Figure 2, and the metric improvements in Table 3 (e.g., UTMOS 4.123 vs 4.113 in stage 1), are small and may not be statistically significant. Please include variance across at least three random seeds, or otherwise provide a statistical test, for the central quantitative claims.
  3. [Section 3.6 and Table 3] The ablation in Table 3 is described as comparing training strategies 'under the same model architecture', but the paper does not specify how the APCodec+ staged strategy is implemented in the DS-Codec framework, particularly whether the Transformer block is present in the APCodec+ stage-1 setting and what hyperparameters are used. Without this detail or released code, the comparison is not reproducible and the claimed efficiency advantages (fewer epochs, lower cost) are not quantified. Please provide the experimental settings for the APCodec+ baseline and report the training epochs and compute used.
minor comments (6)
  1. [Section 2.2.2, Algorithm 1] The codebook index computation multiplies by S_{i-1} and then combines indices as a mixed-radix number, but the zero- versus one-based indexing of the code indices is not stated; please clarify to avoid ambiguity.
  2. [Table 1] The ground-truth row reports UTMOS 4.086 while DS-Codec scores exceed 4.2; since UTMOS is a learned proxy rather than a direct quality measure, a brief comment explaining why reconstructed speech can score above ground truth would help readers interpret the table.
  3. [Section 3.2] Training uses 1-second random crops, but evaluation is performed on full test utterances; please state whether the codec processes full utterances at inference time and whether this mismatch affects the reported metrics.
  4. [Section 3.4] The phrase 'outstanding performance across all objective metrics' is stronger than the data support; for example, the F1 and STOI differences between DS-Codec and BigCodec in Table 1 are 0.002 and 0.006, respectively. Please temper the language or provide statistical evidence.
  5. [Section 6, Reference [28]] The reference to the Vocos paper contains a formatting artifact ('V ocos'); please fix the author name spacing.
  6. [Throughout] The paper does not provide a link to code, model checkpoints, or the demo data; releasing these would substantially improve reproducibility, especially given the checkpoint-initialization issue noted in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DS-Codec's contributions are empirical design choices validated on held-out data, not derived from their own claims.

full rationale

The paper's central claims are empirical: it proposes a dual-stage training strategy and evaluates reconstruction quality on held-out LibriSpeech and LJSpeech subsets. There is no derivation loop in which a prediction is equivalent to its inputs by construction. No parameter is fitted to the evaluation metric and then renamed as a prediction; no uniqueness theorem is imported from prior work; no load-bearing argument rests on a self-citation. The only self-citation (SlimSpeech, reference [6]) is a contextual TTS reference and does not support the codec claim. The use of BigCodec's official checkpoint for DS-Codec-VQ (Section 3.2) is a stated initialization choice, not a circular reduction: the reported metrics are computed on test utterances not used for fitting, and continued fine-tuning of a baseline does not make the comparison logically circular, though it may weaken attribution of the gains to the proposed strategy. This is a comparison-design and attribution concern, not circularity under the rubric. The independently trained DS-Codec-PQ variant provides additional evidence for the training strategy, even if at a higher bitrate. Consequently, no specific circular step can be quoted and no score above 0 is warranted.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

All free parameters are hand-chosen hyperparameters or architecture constants. The main assumptions are domain-level: LibriSpeech training transfers to LJSpeech, the objective metrics measure reconstruction quality, and the frozen-encoder stage-2 design is beneficial. The mirror-robustness claim rests on an indirect loss gap rather than direct codebook utilization statistics.

free parameters (6)
  • VQ codebook size = 8,192
    Hand-chosen following Yu et al. [24]; not derived from any theory.
  • PQ codebook configuration = 4 x [16, 16, 16, 16] = 65,536
    Chosen by hand to create a large single codebook; number of split groups and per-group sizes are not derived.
  • Latent reduction dimension = 8
    Borrowed from DAC for quantization; not derived in the paper.
  • Downsampling factor = 200
    Architecture choice inherited from BigCodec; not justified by an optimization.
  • Stage 1 learning rate schedule = 1e-4 to 1e-5 over 1,000 warmup steps
    Hand-set optimizer hyperparameters; no sensitivity analysis.
  • Stage 2 learning rate schedule = 2e-5 to 1e-5
    Hand-set for decoder fine-tuning; no sensitivity analysis.
assumptions (4)
  • domain assumption LibriSpeech training subsets are sufficient to learn a codec that generalizes to LJSpeech.
    The model is trained on 960 hours of LibriSpeech and evaluated on LibriSpeech test-clean and LJSpeech; cross-corpus generalization is assumed from these results.
  • domain assumption UTMOS, PESQ, STOI, and F1 are adequate proxies for speech reconstruction quality.
    The paper uses only these objective metrics and does not report listening tests or confidence intervals.
  • ad hoc to paper The mirrored structure produces a more robust codebook.
    The paper asserts this based on Figure 2's quantizer input-output MSE gap, but 'robustness' is not formally defined or directly measured.
  • ad hoc to paper Freezing the encoder and quantizer in stage 2 preserves stage-1 benefits.
    Stage 2 freezes these modules to focus on decoder training; the paper does not test whether updating them would yield better results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DS-Codec: Dual-Stage Training with Mirror-to-NonMirror Architecture Switching for Speech Codec." pith.science (2026). https://pith.science/paper/2IV7YDOK

@misc{pith2026250524314,
  author       = {Pith},
  title        = {Pith review of: DS-Codec: Dual-Stage Training with Mirror-to-NonMirror Architecture Switching for Speech Codec},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2IV7YDOK}},
  note         = {Machine review of arXiv:2505.24314}
}
read the original abstract

Neural speech codecs are essential for advancing text-to-speech (TTS) systems. With the recent success of large language models in text generation, developing high-quality speech tokenizers has become increasingly important. This paper introduces DS-Codec, a novel neural speech codec featuring a dual-stage training framework with mirror and non-mirror architectures switching, designed to achieve superior speech reconstruction. We conduct extensive experiments and ablation studies to evaluate the effectiveness of our training strategy and compare the performance of the two architectures. Our results show that the mirrored structure significantly enhances the robustness of the learned codebooks, and the training strategy balances the advantages between mirrored and non-mirrored structures, leading to improved high-fidelity speech reconstruction.

Figures

Figures reproduced from arXiv: 2505.24314 by the authors.

Figure 1
Figure 1. Schematic diagram of DS-Codec and illustration of the dual-stage training strategy for DS-Codec. The codec with non-mirror architecture is composed of a mirrored Encoder, Quantization Module, Transformer Block, and mirrored Decoder. Short-Term Memory (LSTM) network. The CNN blocks are designed to downsample the input waveforms by a specific fac￾tor, utilizing multiple convolutional layers with varying dilation rates… view at source ↗
Figure 2
Figure 2. Training Loss Comparison. VQ loss represents the vector quantization loss, while MSE loss is defined as the mean squared error (MSE) loss between the input and output of the Quantization Module. der low-bitrate conditions. This performance can be attributed to the effectiveness of the two-stage training framework. In the first stage, the mirror architecture significantly enhances code￾book construction and represent… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 9 canonical work pages

  1. [1]

    A pivotal challenge is the transformation of continuous speech signals into interpretable representations suitable for inference and training within large language models

    Introduction In recent years, large language models (LLMs) [1, 2] have ex- hibited remarkable capabilities in the realm of text generation, garnering significant attention across various domains, includ- ing the field of text-to-speech synthesis (TTS) [3, 4, 5, 6]. A pivotal challenge is the transformation of continuous speech signals into interpretable r...

    work page

  2. [2]

    Model Architecture The overall framework of our model adopts a non-mirrored ar- chitecture, as illustrated in Figure 1

    Methods 2.1. Model Architecture The overall framework of our model adopts a non-mirrored ar- chitecture, as illustrated in Figure 1. The Encoder and Decoder architectures are inspired by BigCodec [17], with the Encoder comprising a series of residual Convolutional Neural Network (CNN) blocks. Each block incorporates snake activation func- tions [21] and i...

    arXiv 2025

  3. [3]

    Dataset and Metrics We use LibriSpeech [27] to train the speech codec we proposed

    Experiments 3.1. Dataset and Metrics We use LibriSpeech [27] to train the speech codec we proposed. The LibriSpeech is a corpus of approximately 1000 hours of 16kHz read English speech. Our training set was a combina- tion of the train-clean-100, train-clean-360, and train-other-500 subsets. And test-clean is used for evaluation. For the objective evaluat...

    work page 2000

  4. [4]

    Conclusion This paper introduces DS-Codec, a neural speech codec built on a novel two-stage training strategy. Our model demonstrates superior performance in speech reconstruction by leveraging a two-stage training framework that balances the advantages between mirrored and non-mirrored structures, outperforming previous neural speech codecs. Additionally...

    work page

  5. [5]

    Acknowledgements This work was supported in part by the National Natural Science Foundation of China under Grants 62276220 and 62371407 and the Innovation of Policing Science and Technol- ogy, Fujian province (Grant number: 2024Y0068)

  6. [6]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  7. [7]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  8. [8]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    arXiv 2023

Show all 36 references
  1. [9]

    Audiolm: a language modeling approach to audio gener- ation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: a language modeling approach to audio gener- ation,” IEEE/ACM transactions on audio, speech, and language processing, vol. 31, pp. 2...

    2023

  2. [10]

    Audiogen: Textu- ally guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textu- ally guided audio generation,” arXiv preprint arXiv:2209.15352, 2022

    2022 arXiv

  3. [11]

    Slim- speech: Lightweight and efficient text-to-speech with slim recti- fied flow,

    K. Wang, W. Guan, S. Lu, J. Yao, L. Li, and Q. Hong, “Slim- speech: Lightweight and efficient text-to-speech with slim recti- fied flow,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  4. [12]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020

    2020

  5. [13]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,”arXiv preprint arXiv:1904.05862, 2019

    1904 arXiv

  6. [14]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  7. [15]

    Neural discrete represen- tation learning,

    A. Van Den Oord, O. Vinyals et al., “Neural discrete represen- tation learning,” Advances in neural information processing sys- tems, vol. 30, 2017

    2017

  8. [16]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

  9. [17]

    A review of vector quantization tech- niques,

    A. Vasuki and P. Vanathi, “A review of vector quantization tech- niques,” IEEE Potentials, vol. 25, no. 4, pp. 39–47, 2006

    2006

  10. [18]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” Ad- vances in Neural Information Processing Systems, vol. 36, 2024

    2024

  11. [19]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022

    2022 arXiv

  12. [20]

    Hifi- codec: Group-residual vector quantization for high fidelity audio codec,

    D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y . Zou, “Hifi- codec: Group-residual vector quantization for high fidelity audio codec,” arXiv preprint arXiv:2305.02765, 2023

    2023 arXiv

  13. [21]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,

    S. Ji, Z. Jiang, W. Wang, Y . Chen, M. Fang, J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Li et al. , “Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,” arXiv preprint arXiv:2408.16532, 2024

    2024 arXiv

  14. [22]

    Bigcodec: Pushing the limits of low-bitrate neural speech codec,

    D. Xin, X. Tan, S. Takamichi, and H. Saruwatari, “Bigcodec: Pushing the limits of low-bitrate neural speech codec,” arXiv preprint arXiv:2409.05377, 2024

    2024 arXiv

  15. [23]

    Single-codec: Single-codebook speech codec towards high-performance speech generation,

    H. Li, L. Xue, H. Guo, X. Zhu, Y . Lv, L. Xie, Y . Chen, H. Yin, and Z. Li, “Single-codec: Single-codebook speech codec towards high-performance speech generation,” arXiv preprint arXiv:2406.07422, 2024

    2024 arXiv

  16. [24]

    Ad- dressing index collapse of large-codebook speech tokenizer with dual-decoding product-quantized variational auto-encoder,

    H. Guo, F. Xie, D. Yang, H. Lu, X. Wu, and H. Meng, “Ad- dressing index collapse of large-codebook speech tokenizer with dual-decoding product-quantized variational auto-encoder,”arXiv preprint arXiv:2406.02940, 2024

    2024 arXiv

  17. [25]

    Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tang et al., “Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,”arXiv preprint arXiv:2403.03100, 2024

    2024 arXiv

  18. [26]

    Neural networks fail to learn periodic functions and how to fix it,

    L. Ziyin, T. Hartwig, and M. Ueda, “Neural networks fail to learn periodic functions and how to fix it,” Advances in Neural Infor- mation Processing Systems, vol. 33, pp. 1583–1594, 2020

    2020

  19. [27]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  20. [28]

    Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022– 17 033, 2020

    2020

  21. [29]

    Vector-quantized image modeling with improved vqgan,

    J. Yu, X. Li, J. Y . Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y . Xu, J. Baldridge, and Y . Wu, “Vector-quantized image modeling with improved vqgan,” arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  22. [30]

    Product quantization for nearest neighbor search,

    H. Jegou, M. Douze, and C. Schmid, “Product quantization for nearest neighbor search,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 1, pp. 117–128, 2010

    2010

  23. [31]

    Apcodec+: A spectrum-coding-based high-fidelity and high-compression-rate neural audio codec with staged training paradigm,

    H.-P. Du, Y . Ai, R.-C. Zheng, and Z.-H. Ling, “Apcodec+: A spectrum-coding-based high-fidelity and high-compression-rate neural audio codec with staged training paradigm,” in 2024 IEEE 14th International Symposium on Chinese Spoken Language Pro- cessing (ISCSLP). IEEE, 2024, ...

    2024

  24. [32]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  25. [33]

    V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis,

    H. Siuzdak, “V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis,” arXiv preprint arXiv:2306.00814, 2023

    2023 arXiv

  26. [34]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,” arXiv preprint arXiv:2204.02152, 2022

    2022 arXiv

  27. [35]

    Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE international conference on acoustics, speech, and signal processing. Proceed...

    2001

  28. [36]

    Fewer-token neural speech codec with time-invariant codes,

    Y . Ren, T. Wang, J. Yi, L. Xu, J. Tao, C. Y . Zhang, and J. Zhou, “Fewer-token neural speech codec with time-invariant codes,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 12 737–12 741

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.