Pith. sign in

REVIEW 4 major objections 4 minor 43 references

Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MUFFIN claims psychoacoustic band quantization beats full-band neural codecs and reaches 12.5 Hz tokens.

desk verdict MUFFIN has a credible empirical package and a genuinely new multi-band RVQ idea, but the central band-splitting specification is physically impossible as written (37.5–75 Hz on a 75 Hz latent) and needs correction before the mechanism can be taken seriously. read the letter →

arxiv 2505.07235 v1 pith:XEIOQUSL submitted 2025-05-12 cs.SD eess.AS

Dianwen Ng , Kun Zhou , Yi-Wen Chao , Zhiwei Xiong , Bin Ma , Eng Siong Chng This is my paper · ORCID
classification cs.SDeess.AS
keywords neuralaudiocodingpsychoacousticcompressionmulti-bandspectralresidualvectorquantizationspeechcodeczero-shottext-to-speechsnakeactivationtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MUFFIN is a fully convolutional neural codec whose central idea is to perform quantization not on the full latent spectrum at once, but on three frequency bands selected from psychoacoustic knowledge. The paper claims this aligns compression effort with what the ear actually resolves: low bands carry intelligibility, mid bands carry articulation, and high bands carry speaker identity and timbre. On LibriTTS, IEMOCAP, GTZAN, and BBC benchmarks, it reports better reconstruction scores than Encodec, DAC, and a retrained HiFi-Codec at comparable or lower bitrates. A highly compressed variant runs at 12.5 Hz (100 tokens per second, 0.9 kB/s) with perceived quality close to the full-rate model, and MUFFIN tokens improve zero-shot TTS quality in a VALL-E-based system. The paper thus argues that psychoacoustic band structure is a usable inductive bias for neural audio coding, not just a legacy-codec trick.

What carries the argument

The load-bearing object is Multi-Band Spectral Residual Vector Quantization (MBS-RVQ), a quantizer that operates in the latent space of a convolutional autoencoder: the latent $z$ is transformed with FFT, filtered into three frequency bands (0–18.75, 18.75–37.5, 37.5–75 Hz) with scale factors 4, 2, and 1, converted back with inverse FFT, and each band is quantized with its own EMA codebook; quantization residuals feed the next band, so each codebook refines what the previous one missed. The second component is the modified snake activation $f(x) = x + \frac{\beta}{\alpha}\sin^2(\alpha x) + \gamma$, whose derivative is bounded by $|\beta|$, giving Lipschitz stability and, the paper argues, preserving psychoacoustic distances in latent space. The paper also invokes a perceptual-entropy bound $E_p \ge \sum_k H(B_k \mid x(t)) - \sum_k \Delta(B_k, x(t))$ to argue that multi-band modeling lowers the minimal bit rate needed for transparent coding.

What would settle it

Retrain the same MUFFIN architecture with vanilla RVQ while holding every non-quantizer component fixed, then also add each non-quantizer change one at a time to the retrained HiFi-Codec baseline; if any single change closes the LibriTTS test-clean gap in PESQ or UTMOS, the paper's central mechanism is not what drives the improvement.

Watch

Extended reading notes

Core claim

The central claim is that a neural audio codec can improve both compression efficiency and perceived fidelity by splitting the quantizer's latent representation into psychoacoustic frequency bands and allocating separate codebooks to each band with a sequence of residual refinements. This is realized by MBS-RVQ: an FFT of the latent, band-pass filtering into 0–18.75, 18.75–37.5, and 37.5–75 Hz, inverse FFT, then sequential quantization with scale factors 4, 2, and 1, where each stage quantizes the residual left by the previous stage. The architectural counterpart is the modified snake activation $f(x) = x + \frac{\beta}{\alpha}\sin^2(\alpha x) + \gamma$, claimed to preserve fine spectral detail while staying Lipschitz continuous. The paper reports that the resulting codebooks spontaneously separate speech content from speaker identity, with the high-band codebook forming well-separated speaker clusters in t-SNE, and that a 1920× downsampled variant reaches a 12.5 Hz token rate with UTMOS scores at or above the 75 Hz model. It also claims these tokens give state-of-the-art zero-shot TTS results when plugged into a VALL-E system.

Load-bearing premise

The comparison rests on the assumption that the reported gains come from the psychoacoustic multi-band quantizer and the modified snake activation, yet the ablation evidence removes those components without isolating the other architectural and training differences from HiFi-Codec, such as the ConvNeXt-style blocks, grouped convolutions, 64-bin mel loss, and added codebooks.

Editorial extensions

If this is right

  • At 3.0 kB/s-class rates, MUFFIN reports lower STFT and MEL distances and higher PESQ, STOI, UTMOS, and ViSQOL than Encodec, DAC, and a retrained HiFi-Codec on speech, emotional speech, music, and general audio.
  • The 12.5 Hz variant delivers 100 tokens per second at 0.9 kB/s with UTMOS at or above the default 75 Hz model, and it outperforms Mimi on naturalness and reconstruction fidelity.
  • MUFFIN tokens improve zero-shot TTS: VALL-E with MUFFIN achieves a WER of 12.20 versus 21.05 with Encodec and 32.35 with HiFi-Codec, with higher MOS and speaker similarity scores.
  • Codebook 3 (37.5–75 Hz) captures speaker identity without supervision, while codebooks 1 and 2 carry intelligibility and articulation; using only codebooks 1 and 2 yields 19.2 WER on LibriSpeech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The band-split design is tuned to speech; the paper's own music experiments show instrumental content does not separate cleanly across codebooks 2 and 3. Re-tuning the three-way split to a music-specific perceptual scale, such as ERB bands matched to instrument partials, is a direct testable extension the paper leaves open.
  • The codebook analysis suggests a label-free route to factorized speech representations, but the paper only shows speaker clustering, not verification accuracy. Computing a speaker equal-error rate from Codebook 3 embeddings alone would quantify how close this comes to supervised factorization.
  • The VALL-E difficulty at 12.5 Hz hints that ultra-low token rates change the LM's prompt-length balance. A systematic prompt-length sweep for 12.5 Hz tokens is a concrete next experiment, and it is an inference from the paper's reported failure, not a claim it makes.
  • If MBS-RVQ is the effective mechanism, the first codebook should carry less semantic load than in vanilla RVQ; the WER table supports this and predicts that swapping band ratios, for example 2:1:1 instead of 4:2:1, should visibly shift where intelligibility lives.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MUFFIN, a fully convolutional neural audio codec. Its central proposal is Multi-Band Spectral Residual Vector Quantization (MBS-RVQ), which applies an FFT-based frequency-band decomposition to the encoder's latent representation before quantization, using hand-selected band boundaries and 4:2:1 scale factors. The paper also proposes a modified snake activation with amplitude and bias terms, a transformer-inspired convolutional backbone, and reports reconstruction results on LibriTTS, IEMOCAP, GTZAN, and BBC, plus a zero-shot TTS evaluation with VALL-E. High-compression variants operating at 25 Hz and 12.5 Hz frame rates are presented, with the 12.5 Hz version claimed as state-of-the-art.

Significance. If the empirical results are reproducible, MUFFIN would be a competitive neural codec with an interesting factorized token representation for downstream generative tasks. The retrained HiFi-Codec baseline, the evaluation across multiple domains, and the downstream TTS experiment are notable strengths. However, the central mechanistic contribution as written is physically incoherent: the described three-band split on a 75 Hz real-valued latent places one band above the Nyquist frequency. This undercuts the reproducibility of the method and the interpretation of the codebook analyses. The theoretical support in Appendix B also contains a concrete error. The empirical findings may still hold, but the manuscript's description of the method needs major correction.

major comments (4)
  1. [Section 3.1 and Appendix E] The proposed three-band split of a 75 Hz latent into 0–18.75 Hz, 18.75–37.5 Hz, and 37.5–75 Hz is not realizable for a real-valued signal sampled at 75 Hz. The Nyquist limit is 37.5 Hz, so the third band is either empty or duplicates the lower bands in conjugate form. Consequently, Codebook 3 as described cannot encode the high-frequency/speaker cues that Section 4.2 and Appendix E attribute to it. Please clarify whether the latent representation is complex-valued, whether the code uses a different effective latent rate, or how the 37.5–75 Hz band is populated; as written, the method is physically impossible and not reproducible from the text.
  2. [Appendix B] The derivative of the modified snake activation is f'(x)=1+β sin(2αx), but the text writes |f'(x)|=|β sin(2αx)|≤|β|, which is algebraically incorrect. The correct bound is |f'(x)|≤1+|β|, so the activation is (1+|β|)-Lipschitz, not |β|-Lipschitz. In addition, the claim that convolutional and linear layers 'ensure' Lipschitz continuity is vacuous without bounds on the weight matrices; please correct the calculation and either provide norm control for the encoder or soften the claim.
  3. [Section 3.2, Theorem 3.1] The theorem is attributed to Cover (1999), but the stated inequality is not a standard theorem from that reference and no proof is supplied. Since the result is invoked to justify the bitrate allocation in MBS-RVQ, please provide a self-contained proof with precise definitions of H(B_k|x(t)) and Δ(B_k,x(t)), or recast the statement as a heuristic motivated by psychoacoustics rather than a theorem with external attribution.
  4. [Appendix C, Tables 7 and 8] The ablation of MBS-RVQ versus vanilla RVQ isolates that module, but the main paper claims that psychoacoustic multi-band coding is responsible for the gains. The comparison with the retrained HiFi-Codec baseline also includes unreported differences: ConvNeXt inverted bottlenecks, grouped convolutions, the 64-bin mel loss, and the number of codebooks. Without ablating these factors, the central attribution to MBS-RVQ is not established; please discuss this limitation and, if feasible, add at least a loss-function and architecture ablation.
minor comments (4)
  1. [Section 4.2, Table 5] WER values above 100 (e.g., 154, 191) are unusual; please clarify whether these are percentages, raw edit-distance rates, or device-independent scores, and define the maximum possible value for the Whisper-based evaluation.
  2. [Tables 1-4] The symbols '▽' and '▲' are not explained in the captions; please add a note that they denote the 25 Hz and 12.5 Hz variants, respectively.
  3. [Appendix A, Figure 3] The figure reproduces panels from French & Steinberg (1947); please specify whether the underlying book/copyright terms permit redistribution, or use original plots.
  4. [Section G] The column 'Encoding Rate' in the hyperparameter table is not defined; I assume it is the list of strides, but it differs from the 'Downsampling Rate' column. Please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; MUFFIN's reconstruction gains are benchmarked against external codecs and its ablations isolate MBS-RVQ and the snake modifications.

full rationale

I walked the paper's claimed derivation chain and found no step where a prediction reduces by construction to a fitted input or to a self-citation. The MBS-RVQ mechanism is defined directly on the latent spectrum with stated band boundaries and scale factors, and those parameters are hand-chosen psychoacoustic priors rather than fitted to the reported metrics. The central reconstruction claims are evaluated against external baselines (OPUS, Encodec, DAC, HiFi-Codec, Mimi), so the reported scores are not forced by the method's own definitions. Appendix C (Table 7) compares MBS-RVQ against vanilla RVQ under the same architecture, which isolates the proposed quantization mechanism, and Table 8 isolates the snake activation modifications. The 'Theorem 3.1' in Section 3.2 is attributed to Cover (1999) and is not derived from MUFFIN's own fitted parameters; even if the attribution is imprecise, it is not a self-referential reduction. Citations to the authors' prior work (Ng et al. 2023; Zhou et al. 2024a,b; Yip et al. 2024) appear only as context for downstream tasks and are not load-bearing for the codec-quality claims. The concern that the 37.5-75 Hz band exceeds the Nyquist rate of the 75 Hz latent is a correctness or reproducibility issue rather than a circularity issue, so it does not affect this circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The core mechanism rests on hand-chosen band definitions and on the assumption that latent-space frequency separation mirrors perceptual bands. The theoretical support for the latter is invalid as written, and alternative band splits are not tested.

free parameters (3)
  • Latent band boundaries and scale factors = 0-18.75 / 18.75-37.5 / 37.5-75 Hz, ratios 4:2:1
    Chosen by hand from psychoacoustic priors (Appendix A) and not ablated across alternative splits; the central MBS-RVQ design depends on them.
  • Number of bands and codebooks = 4 codebooks (3 band-specific plus residual)
    Architectural choice; the depth of the RVQ stack varies in high-compression variants but the band count stays 3.
  • Mel-spectrogram bins = 64 bins instead of HiFi-Codec's 80
    Selected based on perceptual evaluations as stated in Section 3.4; affects the training loss and may influence quality relative to the baseline.
assumptions (4)
  • domain assumption Different frequency bands carry distinct perceptual information: low bands for intelligibility, mid bands for articulation, high bands for speaker identity and timbre.
    Used to justify MBS-RVQ in Section 3.1 and Appendix A; supported by classical psychoacoustics but not specifically tied to the latent representation used.
  • domain assumption The encoder is Lipschitz continuous, so latent-space band splits preserve psychoacoustic structure.
    Appendix B attempts a proof, but the derivative bound is incorrect; the premise remains an assumption.
  • standard math Theorem 3.1 (perceptual entropy lower bound) as attributed to Cover (1999).
    Stated without proof and not a standard result from that source; if treated as an axiom, it is unverified.
  • domain assumption The retrained HiFi-Codec baseline is a fair control for the other architectures' differences.
    Other baselines (Encodec, DAC, Mimi) are used with pretrained models, so only HiFi-Codec is controlled for training data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding." pith.science (2026). https://pith.science/paper/XEIOQUSL

@misc{pith2026250507235,
  author       = {Pith},
  title        = {Pith review of: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEIOQUSL}},
  note         = {Machine review of arXiv:2505.07235}
}
read the original abstract

Achieving high-fidelity audio compression while preserving perceptual quality across diverse content remains a key challenge in Neural Audio Coding (NAC). We introduce MUFFIN, a fully convolutional Neural Psychoacoustic Coding (NPC) framework that leverages psychoacoustically guided multi-band frequency reconstruction. At its core is a Multi-Band Spectral Residual Vector Quantization (MBS-RVQ) module that allocates bitrate across frequency bands based on perceptual salience. This design enables efficient compression while disentangling speaker identity from content using distinct codebooks. MUFFIN incorporates a transformer-inspired convolutional backbone and a modified snake activation to enhance resolution in fine-grained spectral regions. Experimental results on multiple benchmarks demonstrate that MUFFIN consistently outperforms existing approaches in reconstruction quality. A high-compression variant achieves a state-of-the-art 12.5 Hz rate with minimal loss. MUFFIN also proves effective in downstream generative tasks, highlighting its promise as a token representation for integration with language models. Audio samples and code are available.

Figures

Figures reproduced from arXiv: 2505.07235 by the authors.

Figure 1
Figure 1. Illustration of the MBS-RVQ process: Fast Fourier Trans￾form (FFT) is applied to the encoded latent representation to isolate specific frequency bands, capturing targeted spectral information for each codebook. The filtered representation is reconstructed us￾ing inverse FFT before undergoing quantization. The quantization residuals are then passed to the next codebook Therefore, rather than splitting the bands at th… view at source ↗
Figure 2
Figure 2. Architecture of MUFFIN incorporating a fully convolu￾tional structure. learn richer, more detailed representations, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The figures have been sourced from French & Steinberg (1947), which discusses how speech sounds are recognized by the ear. The data were collected from microphones based on human speech and then analyzed with computational tools to derive the intensity and sound pressure levels. (1) Comparison of Speech Spectra. (2) Idealized Long Average Speech Spectrum at one meter from lips. (3) R.m.s. pressure of speech at 30cm … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of our proposed modifications to the vanilla snake activation and its behavior in actual modeling for different sequential data. E. Illustrations of the auditory feature across various codebook representations. In this section, we conduct a comprehensive e…
Figure 5
Figure 5. Figure 5: A t-SNE plot showcasing each codebook, with speech randomly sampled from VoxCeleb, effectively represents six distinct speakers of the color code. Furthermore, we stress that the reconstructed audio from the demos presented in Section (F) offers compelling evidence tha…
Figure 6
Figure 6. Figure 6: An illustration depicts a randomly sampled speech utterance alongside its reconstruction using incremental codebooks [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: The elbow plot of the word error rate from whisper-large model, utilizing the same setup of incremental codebooks. contextual speech content; using Codebook 3 alone results in high recognition errors, suggesting its limited contribution to 18 [PITH_FULL_IMAGE:figures/…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 18 canonical work pages

  1. [1]

    Apcodec: A neural audio codec with parallel ampli- tude and phase spectrum encoding and decoding

    Ai, Y ., Jiang, X.-H., Lu, Y .-X., Du, H.-P., and Ling, Z.- H. Apcodec: A neural audio codec with parallel ampli- tude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402.10533,

    arXiv

  2. [3]

    The data were collected from microphones based on human speech and then analyzed with computational tools to derive the intensity and sound pressure levels

    The figures have been sourced from French & Steinberg (1947), which discusses how speech sounds are recognized by the ear. The data were collected from microphones based on human speech and then analyzed with computational tools to derive the intensity and sound pressure levels. (1) Comparison of Speech Spectra. (2) Idealized Long Average Speech Spectrum ...

    work page 1947

  3. [5]

    A t-SNE plot showcasing each codebook, with speech randomly sampled from V oxCeleb, effectively represents six distinct speakers of the color code. Furthermore, we stress that the reconstructed audio from the demos presented in Section (F) offers compelling evidence that Codebook 3 effectively quantizes speaker attributes from the phonetic content. This i...

    work page 2022

  4. [7]

    Moshi: a speech-text foundation model for real-time dialogue

    D´efossez, A., Mazar ´e, L., Orsini, M., Royer, A., P ´erez, P., J ´egou, H., Grave, E., and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037,

    arXiv

  5. [8]

    Cosyvoice: A scalable multilingual zero-shot text-to- speech synthesizer based on supervised semantic tokens, 2024a

    Du, Z., Chen, Q., Zhang, S., Hu, K., Lu, H., Yang, Y ., Hu, H., Zheng, S., Gu, Y ., Ma, Z., Gao, Z., and Yan, Z. Cosyvoice: A scalable multilingual zero-shot text-to- speech synthesizer based on supervised semantic tokens, 2024a. Du, Z., Zhang, S., Hu, K., and Zheng, S. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural sp...

    work page 2024

  6. [13]

    Langman, R., Juki´c, A., Dhawan, K., Koluguri, N

    Accessed: [October, 7, 2024]. Langman, R., Juki´c, A., Dhawan, K., Koluguri, N. R., and Ginsburg, B. Spectral codecs: Spectrogram-based audio codecs for high quality speech synthesis. arXiv preprint arXiv:2406.05298,

    arXiv 2024

  7. [14]

    Lim, J. H. and Ye, J. C. Geometric gan. arXiv preprint arXiv:1705.02894,

    arXiv

  8. [16]

    and Yu, J

    Luo, Y . and Yu, J. Music source separation with band-split rnn. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1893–1901,

    work page 1901

Show all 43 references
  1. [17]

    Gull: A generative multifunctional audio codec

    Luo, Y ., Yu, J., Chen, H., Gu, R., and Weng, C. Gull: A generative multifunctional audio codec. arXiv preprint arXiv:2404.04947,

    arXiv

  2. [19]

    Q., Yang, Z., Ni, J., Zhang, C., Ma, Y ., Ni, C., Chng, E

    Ng, D., Zhang, R., Yip, J. Q., Yang, Z., Ni, J., Zhang, C., Ma, Y ., Ni, C., Chng, E. S., and Ma, B. De’hubert: Disentan- gling noise in a self-supervised model for robust speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Pro...

    2023

  3. [20]

    Hall-e: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis

    Nishimura, Y ., Hirose, T., Ohi, M., Nakayama, H., and Inoue, N. Hall-e: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis. arXiv preprint arXiv:2410.04380,

    arXiv

  4. [21]

    Librispeech: an asr corpus based on public domain au- dio books

    Panayotov, V ., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain au- dio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pp. 5206–5210. IEEE,

    2015

  5. [23]

    Ears: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation

    Richter, J., Wu, Y .-C., Krenn, S., Welker, S., Lay, B., Watanabe, S., Richard, A., and Gerkmann, T. Ears: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation. arXiv preprint arXiv:2406.06185,

    arXiv

  6. [24]

    W., Beerends, J

    Rix, A. W., Beerends, J. G., Hollier, M. P., and Hekstra, A. P. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international con- ference on acoustics, speech, and signal processing. Pro-...

    2001

  7. [25]

    arXiv preprint arXiv:2204.02152,

    arXiv

  8. [26]

    Santana, I. A. P., Pinhelli, F., Donini, J., Catharin, L., Man- golin, R. B., Feltrim, V . D., Domingues, M. A., et al. Music4all: A new music database and its applications. In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 399–404. IEEE,

    2020

  9. [27]

    Sturm, B

    URL https://arxiv.org/abs/ 2304.09116. Sturm, B. L. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv:1306.1461,

    arXiv

  10. [31]

    H., and Lee, H.-y

    Wu, H., Chung, H.-L., Lin, Y .-C., Wu, Y .-K., Chen, X., Pai, Y .-C., Wang, H.-H., Chang, K.-W., Liu, A. H., and Lee, H.-y. Codec-superb: An in-depth analysis of sound codec models. arXiv preprint arXiv:2402.13071,

    arXiv

  11. [32]

    D., Markovi ´c, D., and Richard, A

    Wu, Y .-C., Gebru, I. D., Markovi ´c, D., and Richard, A. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  12. [33]

    Hifi-codec: Group-residual vector quantization for high fidelity audio codec

    Yang, D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Y . Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765,

    arXiv

  13. [34]

    Q., Zhao, S., Ng, D., Chng, E

    Yip, J. Q., Zhao, S., Ng, D., Chng, E. S., and Ma, B. To- wards audio codec-based speech separation. In Proc. Interspeech 2024, pp. 2190–2194,

    2024

  14. [35]

    and Wang, X

    Yu, W. and Wang, X. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    arXiv

  15. [36]

    J., Jia, Y ., Chen, Z., and Wu, Y

    Zen, H., Dang, V ., Clark, R., Zhang, Y ., Weiss, R. J., Jia, Y ., Chen, Z., and Wu, Y . Libritts: A corpus de- rived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882,

    1904 arXiv

  16. [37]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

    Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y ., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000, 2023a. Zhang, X., Zhang, D., Li, S., Zhou, Y ., and Qiu, X. Speech- tokeni...

    arXiv

  17. [40]

    Then, Fig

    These figures extend the observations to longer ranges (1 meter) and include measurements of RMS pressure, following the experimental setup described by Dunn & White (1940) for six male subjects. Then, Fig. 4 and 5 present the results from articulation tests employing low-pass...

    1940

  18. [42]

    The elbow plot of the word error rate from whisper-large model, utilizing the same setup of incremental codebooks. contextual speech content; using Codebook 3 alone results in high recognition errors, suggesting its limited contribution to 18 MUFFIN: Multi-band Frequency Recon...

    2024

  19. [43]

    1920 12.5 0.9 100 Encodec 7.43 7.43 14.9 1.51 4.10 (2, 4, 5,

    1920

  20. [1940]

    D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J

    Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301,

    arXiv

  21. [1979]

    Demystify mamba in vision: A linear attention perspective

    Han, D., Wang, Z., Xia, Z., Han, Y ., Pu, Y ., Ge, C., Song, J., Song, S., Zheng, B., and Huang, G. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605,

    arXiv

  22. [1983]

    S., and Zisserman, A

    Nagrani, A., Chung, J. S., and Zisserman, A. V oxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612,

    arXiv

  23. [1988]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and dif- fusion models

    Ju, Z., Wang, Y ., Shen, K., Tan, X., Xin, D., Yang, D., Liu, Y ., Leng, Y ., Song, K., Tang, S., et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and dif- fusion models. arXiv preprint arXiv:2403.03100,

    arXiv

  24. [1999]

    and Gu, A

    Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060,

    arXiv

  25. [2003]

    Estimating or propagating gradients through stochastic neurons for con- ditional computation

    Bengio, Y ., L´eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv:1308.3432,

    arXiv

  26. [2008]

    and Bojar, O

    Variˇs, D. and Bojar, O. Sequence length is a domain: Length- based overfitting in transformer models. arXiv preprint arXiv:2109.07276,

    arXiv

  27. [2013]

    H., Hendriks, R

    Taal, C. H., Hendriks, R. C., Heusdens, R., and Jensen, J. A short-time objective intelligibility measure for time- frequency weighted noisy speech. In 2010 IEEE inter- national conference on acoustics, speech and signal pro- cessing, pp. 4214–4217. IEEE,

    2010

  28. [2015]

    Native multi-band audio coding within hyper-autoencoded reconstruction propagation networks

    Petermann, D., Jang, I., and Kim, M. Native multi-band audio coding within hyper-autoencoded reconstruction propagation networks. In ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  29. [2017]

    J., and Tang, Y

    10 MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding Liu, Q., Wang, W., Jackson, P. J., and Tang, Y . A perceptually-weighted deep neural network for monaural speech enhancement in various background noise condi- tions. In 2017 25th European Signal P...

    2017

  30. [2019]

    C., and Harte, N

    Hines, A., Skoglund, J., Kokaram, A. C., and Harte, N. Visqol: an objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing, 2015: 1–18,

    2015

  31. [2020]

    H., et al

    Zhou, K., Zhang, Y ., Zhao, S., Wang, H., Pan, Z., Ng, D., Zhang, C., Ni, C., Ma, Y ., Nguyen, T. H., et al. Emo- tional dimension control in language model-based text-to- speech: Spanning a broad spectrum of human emotions. arXiv preprint arXiv:2409.16681, 2024a. Zhou, K., Zh...

  32. [2021]

    Neural codec language models are zero-shot text to speech synthesizers

    Wang, C., Chen, S., Wu, Y ., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y ., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111,

    arXiv

  33. [2022]

    Once-for-all se- quence compression for self-supervised speech models

    Chen, H.-J., Meng, Y ., and Lee, H.-y. Once-for-all se- quence compression for self-supervised speech models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  34. [2023]

    Pyramidcodec: Hierarchical codec for long- form music generation in audio domain

    Chen, J., Dai, Z., Ye, Z., Tan, X., Liu, Q., Guo, Y ., and Xue, W. Pyramidcodec: Hierarchical codec for long- form music generation in audio domain. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4253–4263,

    2024

  35. [2024]

    High fidelity neural audio compression

    9 MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding D´efossez, A., Copet, J., Synnaeve, G., and Adi, Y . High fidelity neural audio compression. arXiv preprint arXiv:2210.13438,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.