Pith. sign in

REVIEW 2 major objections 6 minor 58 references

Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Speech-token compression cuts TTS word errors by 56 percent.

desk verdict A promising compression trick for speech token LMs with large reported gains, but the train/inference mismatch on W-tokens and an overstated redundancy claim need answers before I'd trust the numbers. read the letter →

arxiv 2505.24496 v1 pith:RC7AVQLN submitted 2025-05-30 eess.AS

Wenrui Liu , Qian Chen , Wen Wang , Yafeng Chen , Jin Xu , Zhifang Guo , Guanrou Yang , Weiqin Li
show 6 more authors
Xiaoda Yang Tao Jin Minghui Fang Jialong Zuo Bai Jionghao Zemin Liu
This is my paper
classification eess.AS
keywords speechgenerationneuralaudiocodectext-to-speechtokencompressionlong-contextmodelingshort-rangedependencylanguagemodelzero-shotTTS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that in neural-codec text-to-speech, the language model does not need to attend to every previous speech token: prompt tokens and a short local window carry the information needed to predict the next token, while distant tokens mostly add redundancy and dilute attention. It proposes compressed-to-fine language modeling, which keeps prompt and local tokens intact and compresses each long-range span of speech tokens into a single summary token that the model attends to instead. Across several codecs (EnCodec, WavTokenizer, HuBERT, BigCodec) and two language-model architectures (VALL-E and a decoder-only transformer), the scheme consistently lowers word error rate and raises speaker similarity and naturalness. The largest reported gain is a 56.24 percent relative reduction in WER for VALL-E trained on 20,000 hours, with the summary tokens also enabling faster inference by evicting the KV cache of already-compressed tokens.

What carries the argument

The load-bearing object is the compressed token $W$, inserted after every $G$ speech tokens and masked to attend only to the $G$ tokens in its own span. During training $W$ is computed from ground-truth span tokens; during generation it is computed from the model's own predictions and retained in the KV cache while the raw span tokens are discarded. It acts as an information bottleneck that filters long-range redundancy while keeping coarse semantics, and the paper sets $G$ so that one second of speech becomes about five summary tokens, matching the syllable rate.

What would settle it

Take the trained compressed-to-fine model and decode long utterances twice: once with $W$ computed from the model's predicted span tokens, as in the paper, and once with $W$ computed from ground-truth span tokens, keeping everything else fixed. If predicted-token summaries give no better WER than the prompt-local baseline while ground-truth summaries keep the reported gains, the compression mechanism depends on a signal that real decoding does not provide.

Watch

Extended reading notes

Core claim

The central claim is that letting a speech-token language model attend to all previously generated tokens is not just wasteful but actively harmful: long-range speech tokens carry redundant acoustic information and dilute the attention that should glue text to speech. The paper's model therefore splits the token history into fine-grained initial and short-range information (prompt tokens plus a sliding window of recent tokens) and compressed long-range information (spanwise summary tokens $W$). This makes the text-to-speech alignment path continuous and monotonic and improves intelligibility, speaker similarity, and perceived quality across codecs and architectures, while a faster-inference variant drops the hidden states of already-compressed tokens.

Load-bearing premise

The whole gain rests on the summary tokens being computed from ground-truth speech tokens during training but from the model's own possibly flawed predictions during generation, and nothing in the training loop prepares the model for that mismatch.

Editorial extensions

If this is right

  • If the method is right, existing transformer-based codec language models can gain accuracy and speed without changing the codec or the LM architecture, only the attention mask and the inserted summary tokens.
  • The reported optimal setting—about five summary tokens per second and a local window of about one second—means the method is adapting to syllabic speech rhythm rather than to a particular codec's frame rate.
  • The 56.24 percent relative WER reduction on VALL-E and the 42.76 percent relative reduction with WavTokenizer imply that a large share of token-level prediction errors comes from long-range context, not from local acoustic detail.
  • Faster inference follows directly: once a span is compressed, its KV cache can be evicted, lowering per-step complexity from $O(N_p+T)$ to roughly $O(N_p+T/G+N_{AR})$.
  • Because the scheme leaves the codec untouched, it can be layered onto already-trained neural codec language models as a training-time masking scheme rather than requiring a new tokenizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: the same span-compression idea could be applied to the non-autoregressive stage, which the paper keeps on prompt-local windows only, so a bidirectional summary token for acoustic codebooks is a natural testable extension.
  • Not in the paper: because the scheme is codec-agnostic and its main speed gain is KV-cache eviction, it should transfer to streaming or voice-interactive settings, where a causal compressor that summarizes a span as soon as it closes would be the needed adaptation.
  • Not in the paper: the train/inference mismatch for $W$ is the main risk; adding scheduled sampling or a separately trained compressor that reads predicted tokens would directly test whether the gains survive when summaries are built from the model's own output.
  • Not in the paper: the short-range-dependency analysis suggests attention masking itself, rather than compression per se, may explain part of the gain; an ablation comparing summary tokens against a learned 'ignore long-range' embedding would separate those effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper proposes compressed-to-fine language modeling for zero-shot TTS with neural codec language models. For each predicted speech-token span of length G, a compressed token W is inserted that attends only to that span; the current token attends to the text/speech prompt, previously inserted W tokens, and a local sliding window, but not to raw long-range tokens. Training uses teacher-forced ground-truth spans (Eq. 6), while inference computes W from the already predicted tokens of each span (Eq. 7 and Algorithm 1). Experiments on LibriSpeech with HuBERT, BigCodec, WavTokenizer, and EnCodec, paired with decoder-only or VALL-E-style LMs, report consistent WER, SIM, and UTMOS gains, with WER reductions such as 5.94 to 3.40 for WavTokenizer and 5.05 to 2.21 for VALL-E* at 20k hours. Ablations vary G and N, and a KV-cache eviction scheme yields about 20% faster AR decoding.

Significance. The method is attractive for its simplicity and generality: it is codec- and architecture-agnostic, requires only inserting W tokens and modifying attention masks, and the paper provides consistent empirical gains across four codecs and two LM families, plus ablations and pseudo-code. If the mechanism is robust, the compressed-to-fine idea is a useful addition to speech LM design. However, the motivating claim of universal short-range dependency is contradicted by the WavTokenizer row of Table 4, and the W token's train/inference distribution shift is unexamined; both need to be addressed before the paper's general claims can be accepted.

major comments (2)
  1. [§5.2, Table 4] The claim that 'prompt tokens and local tokens already encompass the necessary semantic and acoustic information' is not supported for WavTokenizer. In Table 4, the unrestricted decoder-only baseline achieves WER 5.94, but restricting LMAR to a causal local window of size N_AR=75 raises WER to 7.85, and N_AR=150 gives only 4.93 (Table 4) or 4.96 (Table 5), a marginal improvement over full context. The strong short-range-dependency conclusion is therefore driven by the EnCodec/VALL-E configuration and should be restated as codec-dependent, with an analysis of why WavTokenizer behaves differently. This is load-bearing because short-range dependency is the paper's motivating observation.
  2. [§3.4, Eq. (6)–(7), Algorithm 1] The compressed token W is trained under teacher forcing: in Eq. (6) each W_k attends only to the ground-truth tokens of its span. At inference, Algorithm 1 computes W_k by attending to the G previously predicted tokens of that span, and the paper provides no scheduled sampling, no consistency regularization, and no analysis of the effect of early decoding errors on W. Because W is the only long-range information carrier once the KV cache is evicted, and because Table 4 shows that long-range context is not redundant for WavTokenizer (dropping it at N_AR=75 worsens WER from 5.94 to 7.85), this train/inference discrepancy is a load-bearing robustness gap. Please add an experiment or analysis that quantifies sensitivity to span-level prediction errors, for example comparing W computed from ground-truth spans versus predicted spans, or adding scheduled sampling during training, and report whether the headline gains survive.
minor comments (6)
  1. [Table 2] The Ground Truth row shows '69.72% 4.15' but no WER value; if 69.72% is SIM, the WER column is empty, and if it is WER, the value is implausible for LibriSpeech ground truth. Please correct the table.
  2. [Tables 4 and 5] The WavTokenizer N_AR=150 configuration is reported inconsistently: Table 4 gives WER 4.93, SIM 47.72%, UTMOS 4.15, while Table 5 gives WER 4.96, SIM 48.04%, UTMOS 4.16 for the same configuration. Clarify which numbers are correct.
  3. [§3.3] The definition 'CR = frame_rate/G' is a compressed-token rate in Hz, not a compression ratio; Section 5.5 then describes compression 'by a factor of 5'. Please use consistent terminology so readers do not confuse the compression ratio G:1 with the output token rate.
  4. [§1 and §5.2] The contribution bullet claiming that relying solely on prompt and local tokens gives a 2% absolute WER reduction and a 2% absolute SIM gain is not directly supported by Table 4: the N_AR=50 row for EnCodec/VALL-E gives a 1.16-point WER reduction, while the N_NAR=50 row gives a 2.01-point SIM gain. State which configuration the 2% figures refer to.
  5. [§4 and Table 2] The text says training data is expanded to 44,000 hours using MLS to assess data scaling, but Table 2 only reports 960h and '20Kh' scale rows; clarify what '20Kh' means and whether the 44k-hour experiment was performed.
  6. [§5.1] The claim of 'consistently and substantially improves speech quality (UTMOS)' is stronger than the evidence for HuBERT, where UTMOS moves only from 4.07 to 4.08; consider softening 'substantially' or reporting significance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central gains are empirical comparisons against independent baselines, and no predicted quantity is fitted from the data it is evaluated on.

full rationale

The paper's central claim is that adding compressed long-range spans W plus prompt-local sliding windows to LMAR/LMNAR improves WER, SIM, and UTMOS. This is established by training and evaluation against independent baselines, not derived from any fitted parameter. Eq. 6 and Eq. 7 do exhibit a train/inference distribution shift: W attends to ground-truth span tokens during training but to previously predicted tokens during inference. That is a genuine exposure-bias robustness concern, not a definitional equivalence, and it does not make the reported improvements forced by construction. The compression rate CR=5 is taken from external studies of syllabic rate, and the G/N choices in Table 5 are post-hoc ablations rather than quantities fitted to the headline test metrics. The one overlapping-author citation, WavTokenizer [23], is used as a pretrained external codec, and the reported gains also appear with EnCodec, HuBERT, and BigCodec, so the result does not reduce to that self-citation. No equation in the paper is equivalent to its input by construction, and no prediction is a fitted value renamed as a result. The derivation chain is therefore self-contained; the main risks are empirical robustness and hyperparameter selection, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method's claims rest on several domain assumptions about speech-token locality and compressibility, plus the ad hoc W-token bottleneck. Hyperparameters G, N_AR, N_NAR, and CR are tuned on the task, so the headline numbers reflect favorable configurations rather than parameter-free derivations.

free parameters (4)
  • G (compressed span size) = 10 (EnCodec), 15 (WavTokenizer), 10 (HuBERT), 16 (BigCodec)
    Chosen so that the compression rate CR = frame_rate / G is about 5 Hz. Ablation in Table 5 shows G=10 is best for EnCodec, G=15 for WavTokenizer.
  • N_AR (causal sliding window size for LMAR) = 50 (EnCodec/VALL-E), 75 (WavTokenizer), 50 (HuBERT), 80 (BigCodec)
    Tuned in Tables 4 and 5. For WavTokenizer, N=75 alone degrades WER versus dense attention, while N=150 improves it; the final method pairs N with G.
  • N_NAR (bidirectional sliding window size for LMNAR) = 50 (EnCodec/VALL-E)
    Selected from ablations in Tables 4 and 5; values of 30, 50, 120 were tested, with 50 performing best.
  • CR (compression rate) = 5 Hz
    Set from prior syllabic-rate evidence [3,11,20]. It is a design hyperparameter, and the ablation over G shows sensitivity (WER ranges from 2.87 to 3.84 across G=5,10,15 for VALL-E).
assumptions (4)
  • domain assumption Monotonic alignment between text and speech tokens implies that prediction of the current token relies primarily on local context.
    Motivates the entire approach in Sections 1 and 3.2. The paper provides supporting experiments, but the premise is not proven beyond the tested configurations and is contradicted by the WavTokenizer N=75 row in Table 4.
  • domain assumption Long-range speech tokens contain redundant information that can be compressed without losing essential semantics.
    Core premise of the compression mechanism. The paper offers indirect evidence through improved metrics but does not directly measure redundancy or information preservation.
  • ad hoc to paper A single W token per span, attending only to that span, can serve as an adequate information bottleneck.
    Introduced by this paper. No independent evidence outside the paper's own experiments; the training/inference mismatch of W is an unaddressed assumption.
  • domain assumption 5 Hz discrete units align with syllables and carry sufficient semantic information for TTS.
    Borrowed from prior work [3,11,20] to justify setting CR=5. The paper does not independently validate this for each codec.
invented entities (1)
  • W compressed token (pseudo-token)
    purpose: Inserted into the input sequence to summarize a span of G speech tokens via masked self-attention; provides coarse long-range context while being cheaper than attending to all past tokens.
    The W token is a new architectural construct introduced by the paper. Its adequacy is evidenced only by improvements on the paper's own test sets; it makes no externally falsifiable predictions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation." pith.science (2026). https://pith.science/paper/RC7AVQLN

@misc{pith2026250524496,
  author       = {Pith},
  title        = {Pith review of: Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RC7AVQLN}},
  note         = {Machine review of arXiv:2505.24496}
}
read the original abstract

Neural audio codecs, used as speech tokenizers, have demonstrated remarkable potential in the field of speech generation. However, to ensure high-fidelity audio reconstruction, neural audio codecs typically encode audio into long sequences of speech tokens, posing a significant challenge for downstream language models in long-context modeling. We observe that speech token sequences exhibit short-range dependency: due to the monotonic alignment between text and speech in text-to-speech (TTS) tasks, the prediction of the current token primarily relies on its local context, while long-range tokens contribute less to the current token prediction and often contain redundant information. Inspired by this observation, we propose a \textbf{compressed-to-fine language modeling} approach to address the challenge of long sequence speech tokens within neural codec language models: (1) \textbf{Fine-grained Initial and Short-range Information}: Our approach retains the prompt and local tokens during prediction to ensure text alignment and the integrity of paralinguistic information; (2) \textbf{Compressed Long-range Context}: Our approach compresses long-range token spans into compact representations to reduce redundant information while preserving essential semantics. Extensive experiments on various neural audio codecs and downstream language models validate the effectiveness and generalizability of the proposed approach, highlighting the importance of token compression in improving speech generation within neural codec language models. The demo of audio samples will be available at https://anonymous.4open.science/r/SpeechTokenPredictionViaCompressedToFinedLM.

Figures

Figures reproduced from arXiv: 2505.24496 by the authors.

Figure 1
Figure 1. Illustration of the proposed Compressed-to-fine Language Modeling versus existing methods in autoregressive [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Training process of the proposed compressed-to-fine language modeling for speech generation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of attention weight between text [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of the Dense Attention and Prompt-Local Sliding Window Attention in Non-Autogressive Language [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 19 canonical work pages

  1. [1]

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. 2024. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051 (2024)

    arXiv 2024

  2. [2]

    Matthew Aylett and Alice Turk. 2006. Language redundancy predicts syllabic duration and the spectral characteristics of vocalic syllable nuclei. The Journal of the Acoustical Society of America 119, 5 (2006), 3048–3058

    work page 2006

  3. [3]

    Alan Baade, Puyuan Peng, and David Harwath. 2024. Syllablelm: Learning coarse semantic units for speech language models. arXiv preprint arXiv:2410.04029 (2024)

    arXiv 2024

  4. [4]

    Matthew Baas, Pieter Scholtz, Arnav Mehta, Elliott Dyson, Akshat Prakash, and Herman Kamper. 2025. MARS6: A Small and Robust Hierarchical-Codec Text-to- Speech Model. arXiv preprint arXiv:2501.05787 (2025)

    work page Pith review arXiv 2025

  5. [5]

    Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636 (2023)

    arXiv 2023

  6. [6]

    Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. 2024. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04904 (2024)

    arXiv 2024

  7. [7]

    Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. 2025. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282 (2025)

    arXiv 2025

  8. [8]

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. 2024. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370 (2024)

    arXiv 2024

Show all 58 references
  1. [9]

    Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2025. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. IEEE Transactions on Audio, Speech and Langu...

    2025

  2. [10]

    Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, et al. 2024. 3D-Speaker-Toolkit: An Open Source Toolkit for Multi-modal Speaker Verification and Diarization. (2024). https://arxiv.org/pdf/2403.19971

    2024 arXiv

  3. [11]

    Cheol Jun Cho, Nicholas Lee, Akshat Gupta, Dhruv Agarwal, Ethan Chen, Alan W Black, and Gopala K Anumanchipalli. 2024. Sylber: Syllabic Embedding Repre- sentation of Speech from Raw Audio. arXiv preprint arXiv:2410.07168 (2024)

    2024 arXiv

  4. [12]

    Min Chu and Hu Peng. 2006. Objective measure for estimating mean opinion score of synthesized speech. US Patent 7,024,362

    2006

  5. [13]

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438 (2022)

    2022 arXiv

  6. [14]

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037 (2024)

    2024 arXiv

  7. [15]

    P Kingma Diederik. 2014. Adam: A method for stochastic optimization. (No Title) (2014)

    2014

  8. [16]

    Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. 2021. Variable- rate discrete representation learning. arXiv preprint arXiv:2103.06089 (2021)

    2021 arXiv

  9. [17]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407 (2024)

    2024 arXiv

  10. [18]

    Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. 2024. Funcodec: A funda- mental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 591–595

    2024

  11. [19]

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957 (2025)

    2025 arXiv

  12. [20]

    Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. 2025. Recent Advances in Discrete Speech Tokens: A Review. arXiv preprint arXiv:2502.06490 (2025)

    2025

  13. [21]

    Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, and Furu Wei. 2024. VALL-E R: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment. arXiv preprint arXiv:2406.07855 (2024)

    2024 arXiv

  14. [22]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Rus- lan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processi...

    2021

  15. [23]

    Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al . 2024. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532 (2024)

    2024 arXiv

  16. [24]

    Yidi Jiang, Qian Chen, Shengpeng Ji, Yu Xi, Wen Wang, Chong Zhang, Xianghu Yue, ShiLiang Zhang, and Haizhou Li. 2025. UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook. arXiv preprint arXiv:2502.20067 (2025)

    2025 arXiv

  17. [25]

    Daniel Jurafsky, Alan Bell, Michelle Gregory, and William D Raymond. 2008. Probabilistic relations between words: Evidence from reduction in lexical produc- tion. In Frequency and the emergence of linguistic structure. John Benjamins Publishing Company, 229–254

    2008

  18. [26]

    Hassan Salami Kavaki and Michael I Mandel. 2025. Audio Sparse-Transformer for Speech Classification. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2025

  19. [27]

    Jaeyoung Kim, Mostafa El-Khamy, and Jungwon Lee. 2020. T-gsa: Trans- former with gaussian-weighted self-attention for speech enhancement. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6649–6653

    2020

  20. [28]

    Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems 33 (2020), 8067–8077

    2020

  21. [29]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems 33 (2020), 17022–17033

    2020

  22. [30]

    Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. 2025. Baichuan-Omni-1.5 Techni- cal Report. arXiv preprint arXiv:2501.15368 (2025)

    2025

  23. [31]

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. 2025. MoBA: Mixture of Block Attention for Long-Context LLMs. arXiv preprint arXiv:2502.13189 (2025)

    2025 arXiv

  24. [32]

    Zofia Malisz, Erika Brandt, Bernd Möbius, Yoon Mi Oh, and Bistra Andreeva

  25. [33]

    Yuto Nishimura, Takumi Hirose, Masanari Ohi, Hideki Nakayama, and Nakamasa Inoue. 2024. HALL-E: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis. arXiv preprint arXiv:2410.04380 (2024)

    2024 arXiv

  26. [34]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5206–5210. doi:10.1109/ICASSP.2015.7178964

    2015

  27. [35]

    Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, and David Harwath. 2024. Voicecraft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16973 (2024)

    2024 arXiv

  28. [36]

    Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411 (2020)

    2020 arXiv

  29. [37]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning. PMLR, 28492–28518

    2023

  30. [38]

    Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. 2024. Fewer-token neural speech codec with time-invariant codes. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 12737–12741

    2024

  31. [39]

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152 (2022)

    2022 arXiv

  32. [40]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227

    2023

  33. [41]

    Kevin J Shih, Rafael Valle, Rohan Badlani, Adrian Lancucki, Wei Ping, and Bryan Catanzaro. 2021. RAD-TTS: Parallel flow-based TTS with robust alignment learn- ing and diverse synthesis. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models

    2021

  34. [42]

    Amitay Sicherman and Yossi Adi. 2023. Analysing discrete self supervised speech representation for spoken language modeling. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2023

  35. [43]

    Yakun Song, Zhuo Chen, Xiaofei Wang, Ziyang Ma, and Xie Chen. 2024. Ella-v: Stable neural codec language modeling with alignment-guided sequence reorder- ing. arXiv preprint arXiv:2401.07333 (2024)

    2024 arXiv

  36. [44]

    Refael Tikochinski, Ariel Goldstein, Yoav Meiri, Uri Hasson, and Roi Reichart

  37. [45]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  38. [46]

    Yuhao Wang, Heyang Liu, Ziyang Cheng, Ronghua Wu, Qunshan Gu, Yanfeng Wang, and Yu Wang. 2025. VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation. arXiv:2504.04060 [cs.CL] https://arxiv. Speech Token Prediction via Compressed-to-fine Langua...

    2025 arXiv

  39. [47]

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. 2024. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750 (2024)

    2024 arXiv

  40. [48]

    Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2024. Big- codec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377 (2024)

    2024 arXiv

  41. [49]

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. 2023. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765 (2023)

    2023 arXiv

  42. [50]

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xuankai Chang, Jiatong Shi, Jiang Bian, Zhou Zhao, et al. 2024. Uniaudio: Towards universal audio generation with large language models. In Forty-first International Conference on Machine Learning

    2024

  43. [51]

    Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. 2024. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408.17175 (2024)

    2024 arXiv

  44. [52]

    Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems 36 (2023), 78808–78823

    2023

  45. [53]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2021), 495–507

    2021

  46. [54]

    Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. LibriTTS: A Corpus Derived from LibriSpeech for Text- to-Speech. arXiv:1904.02882 [cs.SD] https://arxiv.org/abs/1904.02882

    2019 arXiv

  47. [55]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000 (2023)

    2023 arXiv

  48. [56]

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023. Speech- tokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692 (2023). Conference’17, July 2017, Washington, DC, USA Wenrui Liu, Qian Chen, Wen Wang, Yafeng Chen...

    2023 arXiv

  49. [2018]

    Frontiers in Communication 3 (2018), 25

    Dimensions of segmental variability: Interaction of prosody and surprisal in six languages. Frontiers in Communication 3 (2018), 25

    2018

  50. [2025]

    Nature Communications 16, 1 (2025), 803

    Incremental accumulation of linguistic context in artificial and biological neural networks. Nature Communications 16, 1 (2025), 803

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.