REVIEW
NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference
Audio and Speech Processing · Submitted 7 Aug 2025 · 40 references
Pith's verdict: NanoCodec is a solid, incremental codec paper with real ablations and a downstream TTS test, but the headline accuracy claims rest on point estimates without variance and the 12.5 FPS trade-off is more muted than the 'high-quality ultra fast' framing suggests.
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
A codec running at 12.5 frames per second can match or beat higher-frame-rate codecs on reconstruction quality while sharply cutting the autoregressive steps a speech LLM must take.
The reading
What carries the argument
The load-bearing component is a partially causal, fully convolutional encoder-decoder with FSQ (Finite Scalar Quantization) vector quantization running at 12.5 frames per second, paired with a causal decoder. At 12.5 FPS, one second of speech becomes 12.5 code sequences rather than 21.5, so an autoregressive speech LLM needs about 42% fewer steps; the causal decoder means those steps can be decoded streamingly without a five-frame lookahead cache. The ablation design isolates frame rate, causality, and bitrate, and the speaker-consistency loss plus the discriminators carry the quality improvements.
Extended reading notes
Core claim
The central discovery is that a codec designed for 12.5 FPS can preserve the reconstruction quality of a 21.5 FPS codec while enabling streaming-friendly decoding, provided the right architectural choices are made. NanoCodec keeps LFSC's FSQ-based quantization and HiFi-GAN-style generator, but adds dilated residual blocks with rates 1, 3, and 5, switches decoder activations from Leaky ReLU to Snake, uses a multi-band multi-scale STFT discriminator, and adds a speaker-consistency loss that pushes reconstructed speech toward the same speaker embedding as the ground truth. These modifications let the 1.78 kbps, 12.5 FPS, partially causal model match or beat LFSC on SQMOS, mel distance, speaker
Load-bearing premise
The headline comparisons assume the five automatic metrics (SQMOS, PESQ, mel distance, SECS, and CER) capture what human listeners care about, since the codec evaluation reports no human listening test and no confidence intervals.
Editorial extensions
If this is right
- A speech LLM trained on NanoCodec tokens at 12.5 FPS reached CER 0.49 on the LibriTTS test-clean set, beating the LFSC-trained TTS's CER 0.85, while real-time factor improved 2.33x and time-to-first-audio dropped by roughly 2.5x compared with the 21.5 FPS LFSC model.
- At 1.1 kbps, NanoCodec reports better SQMOS, PESQ, mel distance, SECS, and CER than Mimi, with CER 3.617 vs 7.221, so the 12.5 FPS format does not force an intelligibility penalty at the lowest bitrate the paper compares.
- Frame-rate reduction from 21.5 to 12.5 FPS at nearly the same bitrate leaves SQMOS, speaker similarity, and CER comparable, suggesting the quality bottleneck is not the frame rate itself but the amount of speech generated per autoregressive step.
- Lower bitrates hurt intelligibility far more than other metrics: CER rises from 2.423 at 1.78 kbps to 3.617 at 1.1 kbps and 6.519 at 0.8 kbps, so naive bitrate reduction for simpler LLM token distributions carries a measurable intelligibility cost.
- Lengthening the TTS context from 5 to 10 seconds lifted speaker similarity from 0.635 to 0.691 with the 12.5 FPS codec, indicating that longer context can partially compensate for the reduced number of tokens available per utterance.
Reading between the lines
- If the automatic-metric results hold up in human listening, the main remaining bottleneck for fast speech LLMs is not codec quality but the autoregressive model's ability to work with fewer, longer tokens; architectural changes like explicit duration modeling or longer conditioning context may be more productive than pushing frame rate even lower.
- The success of a noncausal encoder plus causal decoder suggests that other codecs could adopt the same split to gain streaming capability at low latency, since only the decoder is used during TTS or speech-to-speech inference; this is a testable recipe for future codec designs.
- The sharp CER degradation below 1.1 kbps points to a practical floor for pure reconstruction-based codecs; hybrid approaches that inject semantic or phonetic information into the quantized tokens might push intelligible bitrates lower than 1 kbps, which the paper does not explore.
- Because the paper reports point estimates without confidence intervals for the codec metrics, a direct re-run with multiple seeds or significance testing would clarify which small SQMOS gaps (for example 4.441 vs 4.432) are robust; this is a check available to any group with the public checkpoints.
Original abstract
Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens, enabling the application of language modeling techniques to speech data. However, existing audio codecs often operate at high frame rates, leading to slow training and inference, particularly for autoregressive models. To address this, there is growing interest in low frame-rate audio codecs, which reduce the number of autoregressive steps required to generate one second of audio. In this paper, we conduct ablation studies to examine the impact of frame rate, bitrate, and causality on codec reconstruction quality. Based on our findings, we introduce NanoCodec, a state-of-the-art audio codec that achieves high-quality compression at just 12.5 frames per second (FPS). NanoCodec outperforms related works across various bitrate ranges, establishing a new benchmark for low-latency and efficient Speech LLM training and inference.
References
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Uses its method or tool 1 paper
-
For speech, we adopt NanoCodec [ref] at 0.6 kbps by default.
IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems · Experimental Setup · 1.00
What it cites · 23 sentences without a confident label
Uses the method or tool of 7 papers
-
Speaker similarity is assessed using the Speaker Encoder Cosine Similarity (SECS) [ref] , computed with the SOTA ECAPA2 speaker encoder [ref] .
ECAPA2: A Hybrid Neural Network Architecture and Training Strategy for Robust Speaker Embeddings · Results and Discussion · 1.00
-
To evaluate the performance of the Koel-TTS model with various codecs, we followed the methodology proposed by [ref] .
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance · Results and Discussion · 1.00
-
We retained the WavLM-based discriminator and the multi-period discriminator [ref] from the original work [ref] , but replaced the multi-scale complex STFT discriminator [ref] with the multi-band multi-scale STFT discriminator [ref] .
High Fidelity Neural Audio Compression · NanoCodec · 0.91
Cites as background 2 papers
-
In recent years, Neural Audio Codecs (NACs), which process raw waveforms as both input and output, have emerged as a promising solution that balances audio quality and bitrate [ref] .
Spectral Codecs: Improving Non-Autoregressive Speech Synthesis with Spectrogram-Based Audio Codecs · Introduction · 0.87
-
In recent years, Neural Audio Codecs (NACs), which process raw waveforms as both input and output, have emerged as a promising solution that balances audio quality and bitrate [ref] .
High Fidelity Neural Audio Compression · Introduction · 0.87
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.