pith. sign in

arxiv: 2606.09019 · v1 · pith:KIXGXHKRnew · submitted 2026-06-08 · 💻 cs.SD · cs.AI

TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech

Yejin Lee , Junwon Moon , Hyoeun Kim , Hyunjin Choi , Heeseung Kim , Kyuhong Shim This is my paper

Pith reviewed 2026-06-27 15:25 UTC · model grok-4.3

classification 💻 cs.SD cs.AI
keywords autoregressive TTSaudio token compressionpatch-based modelingefficient inferencecodec-based speechLoRA adaptationKV cachespeech language models
0
0 comments X

The pith

Grouping speech tokens into patches lets pretrained AR-TTS models run 1.8 times faster with 75 percent less cache memory.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a framework that compresses long sequences of discrete audio tokens into shorter patches for autoregressive modeling. A compressor creates these patches, the main model processes the reduced sequence, and an extractor rebuilds the detailed tokens using speaker information. This keeps the original high-quality model unchanged while addressing the efficiency problem caused by very long token sequences in speech. Sympathetic readers would care because current TTS systems are slow at inference due to processing one token at a time. The approach demonstrates that shifting to patch-level causal modeling can deliver speed and memory gains without major redesigns.

Core claim

TLDR performs causal modeling on patch-level sequences rather than token-level sequences in codec-based autoregressive text-to-speech. Consecutive tokens are grouped into compact latent patches by a lightweight compressor. These patches are modeled by a frozen pretrained AR backbone that is adapted using LoRA. Fine-grained speech tokens are then reconstructed within each patch by a speaker-conditioned extractor. With a patch size of 4 this yields an inference speedup of 1.8 times and reduces global KV-cache memory by up to 75 percent.

What carries the argument

The patch compressor, frozen AR backbone with LoRA, and speaker-conditioned token extractor that together enable modeling at patch level instead of token level.

If this is right

  • Shorter sequences mean the autoregressive backbone performs fewer causal steps during generation.
  • The KV cache only needs to store hidden states for the reduced number of patches rather than all tokens.
  • Existing pretrained modules can be reused without replacement or full retraining.
  • Reconstruction of tokens from patches preserves overall TTS quality and naturalness.
  • Patch-level global causal modeling provides a practical route to lower inference costs in current systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the compression works well across different speakers and styles, the method could extend to other long-sequence generation tasks.
  • The separation of global patch modeling from local token reconstruction might allow independent optimization of each part.
  • Reducing sequence length this way could make real-time TTS more feasible on resource-limited devices.

Load-bearing premise

The patches created by the compressor contain enough information for the extractor to recover the original speech tokens without significant degradation in audio quality.

What would settle it

A side-by-side comparison on standard TTS benchmarks where TLDR shows substantially lower mean opinion scores or higher word error rates than the baseline despite the reported speed gains.

Figures

Figures reproduced from arXiv: 2606.09019 by Heeseung Kim, Hyoeun Kim, Hyunjin Choi, Junwon Moon, Kyuhong Shim, Yejin Lee.

Figure 1
Figure 1. Figure 1: Overview of TLDR, a patch-based autoregressive framework for speech generation. A token-to-patch compressor maps consecutive codec tokens into patch representations, a patch-level global sequence model predicts the patch sequence, and a patch-to-token extractor generates fine￾grained codec tokens within each patch. TLDR partitions consecutive codec tokens into patches and feeds the resulting patch sequence… view at source ↗
Figure 2
Figure 2. Figure 2: WER–RTF trade-off of TLDR with different patch sizes, compared with existing speech generation models. 4 8 12 16 20 Output duration (s) 100 200 300 400 500 600 700 800 KV cache size (MB) KV cache memory CosyVoice3 (Baseline) TLDR (k=4) TLDR (k=6) TLDR (k=8) 4 8 12 16 20 Output duration (s) 2500 5000 7500 10000 12500 15000 17500 20000 Peak GPU memory (MB) End-to-end peak memory CosyVoice3 (Baseline) TLDR (k… view at source ↗
Figure 3
Figure 3. Figure 3: Inference memory comparison between [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Text–speech cross-attention maps for CosyVoice3 and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
read the original abstract

Codec-based autoregressive (AR) speech language models have achieved strong text-to-speech (TTS) quality by modeling speech as sequences of discrete audio tokens with large pretrained backbones. However, this token-level formulation creates a structural efficiency bottleneck: speech-token sequences are much longer than text sequences, requiring the AR backbone to perform causal computation at every token position and maintain a KV cache that grows with the sequence length. We introduce TLDR, a patch-based autoregressive framework that accelerates codec-based AR-TTS by shifting the causal modeling from token-level speech sequences to patch-level sequences. TLDR groups consecutive codec tokens into compact latent patches using a lightweight compressor, models the resulting shorter patch sequence with a frozen pretrained AR-TTS backbone adapted by LoRA, and reconstructs fine-grained speech tokens within each patch using a speaker-conditioned extractor. With a patch size of 4, TLDR achieves a 1.8x inference speedup over the baseline AR-TTS model and reduces global KV-cache memory by up to 75%. Experimental results indicate that patch-level global causal modeling can be a practical way to reduce the inference cost of pretrained codec-based AR-TTS systems without replacing the existing modules.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper introduces TLDR, a patch-based autoregressive framework for codec-based TTS. Consecutive codec tokens are grouped into compact latent patches via a lightweight compressor; the shorter patch sequence is modeled by a frozen pretrained AR-TTS backbone with LoRA adaptation; fine-grained tokens are reconstructed inside each patch by a speaker-conditioned extractor. At patch size 4 the method reports a 1.8 imes inference speedup and up to 75 % reduction in global KV-cache memory, arguing that patch-level causal modeling is a practical route to lower inference cost without replacing existing modules.

Significance. If the reconstruction step preserves naturalness and intelligibility, the approach offers a lightweight way to shorten the effective sequence length seen by large AR backbones while retaining their pretrained weights. The reuse of an existing model with only LoRA plus a small extractor is a practical strength that could be adopted by other codec-based TTS systems.

major comments (1)
  1. [Abstract] Abstract (and §4 experimental results): the practicality claim rests on the assertion that the speaker-conditioned extractor recovers fine-grained codec tokens “with negligible loss in overall TTS quality and naturalness.” No token-level reconstruction accuracy, mel-distance, WER, or perceptual (MOS) scores are reported to bound this loss; without such quantification the 1.8 imes speedup and 75 % KV reduction cannot be judged practically useful.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed comment on the need for explicit quantification of reconstruction quality. We agree that the practicality of the reported efficiency gains requires supporting measurements and will revise the manuscript to include them.

read point-by-point responses

  1. Referee: [Abstract] Abstract (and §4 experimental results): the practicality claim rests on the assertion that the speaker-conditioned extractor recovers fine-grained codec tokens “with negligible loss in overall TTS quality and naturalness.” No token-level reconstruction accuracy, mel-distance, WER, or perceptual (MOS) scores are reported to bound this loss; without such quantification the 1.8× speedup and 75 % KV reduction cannot be judged practically useful.

    Authors: We agree that the manuscript should provide direct quantitative bounds on any quality degradation from the speaker-conditioned extractor to substantiate the claim of negligible loss. The current version supports the overall practicality through end-to-end TTS results, but does not isolate token-level reconstruction accuracy, mel-distance, WER, or MOS for the extractor step. In the revised manuscript we will add these metrics in §4 (and update the abstract), reporting reconstruction accuracy, mel-spectrogram distances, WER on synthesized speech, and MOS comparisons against both the original codec tokens and the baseline AR-TTS model. This will allow readers to evaluate the efficiency-quality trade-off explicitly. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical efficiency gains from sequence shortening

full rationale

The paper describes an engineering framework that groups tokens into patches via a compressor, applies LoRA to a frozen AR backbone, and uses an extractor for reconstruction. Speedup and KV-cache reduction follow directly from shorter patch sequences by construction of the method, with no equations, fitted parameters renamed as predictions, or self-citation chains presented as load-bearing derivations. The abstract and description contain no mathematical reductions or uniqueness theorems; results are framed as experimental outcomes on a reused pretrained model. This is a standard non-circular practical adaptation.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The framework rests on the unverified assumption that token grouping into patches preserves enough information for high-quality reconstruction; patch size 4 appears chosen experimentally.

free parameters (1)
  • patch size = 4
    Experimental choice reported to achieve the stated speedup and memory reduction.
axioms (1)
  • domain assumption Consecutive codec tokens contain redundant information that a lightweight compressor can capture in compact patches while allowing faithful reconstruction by a speaker-conditioned extractor.
    This premise is required for the patch-level modeling to maintain TTS quality.

pith-pipeline@v0.9.1-grok · 5759 in / 1239 out tokens · 25882 ms · 2026-06-27T15:25:01.056859+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 20 canonical work pages · 11 internal anchors

  1. [1]

    Seed-TTS: A Family of High-Quality Versatile Speech Generation Models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    AudioLM: A language modeling approach to audio generation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: A language modeling approach to audio generation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023

    2023

  3. [3]

    Accelerating Large Language Model Decoding with Speculative Sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    work page arXiv 2024

  5. [5]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    2022

  6. [6]

    F5- tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5- tts: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025

    2025

  7. [7]

    High Fidelity Neural Audio Compression

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  8. [8]

    Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143, 2020

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143, 2020

    work page arXiv 2005

  9. [9]

    CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training

    Zhihao Du et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In2024 IEEE spoken language technology workshop (SLT), pages 682–689. IEEE, 2024

    2024

  13. [13]

    Better & Faster Large Language Models via Multi-token Prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Fireredtts: A foundation text-to- speech framework for industry-level generative speech applica- tions,

    Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024. 10

    work page arXiv 2024

  15. [15]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  16. [16]

    Dynamic chunking for end-to-end hierarchical sequence modeling

    Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling. InInternational Conference on Learning Representations, 2026

    2026

  17. [17]

    Ditar: Diffusion transformer autoregressive modeling for speech generation

    Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation. InInternational Conference on Machine Learning (ICML), 2025

    2025

  18. [18]

    Glow-TTS: A generative flow for text-to- speech via monotonic alignment search

    Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: A generative flow for text-to- speech via monotonic alignment search. InAdvances in Neural Information Processing Systems, volume 33, pages 8067–8077, 2020

    2020

  19. [19]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pages 611–626, 2023

    2023

  20. [20]

    V oicebox: Text-guided multilingual universal speech generation at scale.Advances in neural information processing systems, 36:14005–14034, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oicebox: Text-guided multilingual universal speech generation at scale.Advances in neural information processing systems, 36:14005–14034, 2023

    2023

  21. [21]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning (ICML), 2023

    2023

  22. [22]

    Fast and high-quality auto-regressive speech synthesis via speculative decoding

    Bohan Li, Hankun Wang, Situo Zhang, Yiwei Guo, and Kai Yu. Fast and high-quality auto-regressive speech synthesis via speculative decoding. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  23. [23]

    Accelerating autoregressive speech synthesis inference with speech speculative decoding.arXiv preprint arXiv:2505.15380, 2025

    Zijian Lin, Yang Zhang, Yougen Yuan, Yuming Yan, Jinjiang Liu, Zhiyong Wu, Pengfei Hu, and Qun Yu. Accelerating autoregressive speech synthesis inference with speech speculative decoding.arXiv preprint arXiv:2505.15380, 2025

    work page arXiv 2025

  24. [24]

    Matcha-tts: A fast tts architecture with conditional flow matching

    Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter. Matcha-tts: A fast tts architecture with conditional flow matching. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11341–11345. IEEE, 2024

    2024

  25. [25]

    Hierarchical transformers are more efficient language models

    Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Łukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. InFindings of the Association for Computational Linguistics: NAACL 2022, pages 1559–1571, 2022

    2022

  26. [26]

    Accelerating codec-based speech synthesis with multi-token prediction and speculative decoding

    Tan Dat Nguyen, Ji-Hoon Kim, Jeongsoo Choi, Shukjae Choi, Jinseok Park, Younglo Lee, and Joon Son Chung. Accelerating codec-based speech synthesis with multi-token prediction and speculative decoding. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  27. [27]

    Byte latent transformer: Patches scale better than tokens

    Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, et al. Byte latent transformer: Patches scale better than tokens. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9238–9258, 2025

    2025

  28. [28]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015

    2015

  29. [29]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  30. [30]

    FastSpeech 2: Fast and high-quality end-to-end text to speech

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech 2: Fast and high-quality end-to-end text to speech. InInternational Conference on Learning Representations, 2021

    2021

  31. [31]

    Fastspeech: Fast, robust and controllable text to speech.Advances in neural information processing systems, 32, 2019

    Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech.Advances in neural information processing systems, 32, 2019

    2019

  32. [32]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022.arXiv preprint arXiv:2204.02152, 2022

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022.arXiv preprint arXiv:2204.02152, 2022. 11

    work page arXiv 2022

  33. [33]

    Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Wespeaker: A research and production oriented speaker embedding learning toolkit

    Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  35. [35]

    Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens

    Xinsheng Wang et al. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens.arXiv preprint arXiv:2503.01710, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Maskgct: Zero-shot text-to- speech with masked generative codec transformer,

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskgct: Zero-shot text-to-speech with masked genera- tive codec transformer.arXiv preprint arXiv:2409.00750, 2024

    work page arXiv 2024

  37. [37]

    V ocalnet: Speech llms with multi-token prediction for faster and high-quality generation

    Yuhao Wang, Heyang Liu, Ziyang Cheng, Ronghua Wu, Qunshan Gu, Yanfeng Wang, and Yu Wang. V ocalnet: Speech llms with multi-token prediction for faster and high-quality generation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19595–19612, 2025

    2025

  38. [38]

    Fireredtts-2: Towards long conversational speech generation for podcast and chatbot,

    Kun Xie et al. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.arXiv preprint arXiv:2509.02020, 2025

    work page arXiv 2025

  39. [39]

    Qwen2 Technical Report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis,

    Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, et al. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.04128, 2025

    work page arXiv 2025

  41. [41]

    Megabyte: Predicting million-byte sequences with multiscale transformers.Advances in Neural Information Process- ing Systems, 36:78808–78823, 2023

    Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers.Advances in Neural Information Process- ing Systems, 36:78808–78823, 2023

    2023

  42. [42]

    Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    2021

  43. [43]

    LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech.arXiv preprint arXiv:1904.02882, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  44. [44]

    Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech.arXiv preprint arXiv:2506.21619, 2025

    Siyi Zhou et al. Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech.arXiv preprint arXiv:2506.21619, 2025. 12 A Implementation Details Table 7: Model configuration of TLDR. Component Hyperparameter Value Token-to-Patch Compressor Layers 1 Hidden dim / Heads 896 / 8 FFN dim 3584 Cross-attn h...

    work page arXiv 2025