Pith. sign in

REVIEW 3 major objections 5 minor 66 references

A single unified model can generate a complete mixed audio scene - dialogue, ambience, and timed events - from one text prompt.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:12 UTC pith:5TLMBXSK

load-bearing objection A serious, honestly hedged system report whose strongest claim (temporal localization) rests on the weakest evidence; worth refereeing, but the in-house benchmarks need human validation, error bars, and more baselines before the headline result is credible. the 3 major comments →

arxiv 2607.27011 v2 pith:5TLMBXSK submitted 2026-07-29 eess.AS

Qwen-Audio-3.0-Gen-Preview Technical Report

Junyu Dai , Xiaoyue Duan , Xinyue Fan , Yihan Feng , Jingbei Li , Xiangang Li , Yunjia Li , Lejun Min
show 9 more authors
Yufei Shi Xingchen Song Yiran Wang Cheng Wen Menglin Wu Bajian Xiang Huaicheng Zhang Han Zhao Ruichen Zheng
This is my paper
classification eess.AS
keywords unified audio generationdiffusion transformervariational autoencodertemporal localizationmulti-speaker dialoguetext-to-audio generationmixed-scene audioclassifier-free guidance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Qwen-Audio-3.0-Gen-Preview is a unified audio-generation framework that aims to prove that one model, with one text interface and one generation path, can produce the entire mixed waveform of a scene: character dialogue, persistent ambience, foreground events, and long-range background structure. Instead of separate models per sound type plus manual mixing, the paper proposes a non-autoregressive Diffusion Transformer - generating the whole waveform in one diffusion pass rather than token-by-token - operating in the shared continuous latent space of a variational autoencoder, with free-form prompts converted into structured temporal records that the model learns to follow. The authors' evidence is comparative: highest speaker similarity on a public reference-conditioned benchmark, best cross-turn speaker consistency on a multi-speaker benchmark, and better temporal localization (higher overlap between requested and judged audible event intervals) on a rich-timeline benchmark, alongside advantages on audio-language-model and aesthetic-quality assessments. If the central claim holds, complex long-form audio scenes - film, game, or podcast - could be authored by describing them, with character voices and event timing preserved across tens of seconds. The paper's stated goal is to show that unified generation can handle temporally structured audio without task-specific branches.

Core claim

The central claim, stated on the paper's own terms, is that heterogeneous audio - standalone speech, music, sound effects, and mixtures containing all of them - can share a single non-autoregressive generation path without losing quality or control. A shared continuous VAE compresses 48 kHz stereo waveforms into 25 Hz latent sequences and is given semantic supervision from a frozen language model; a Diffusion Transformer denoises noise into those latents conditioned on text tokens. Free-form user requests are first rewritten by a prompt-enhancement module into a structured record (global scene, speaker profiles, time-ordered events, primary audible content), rendered as a textual condition,

What carries the argument

The load-bearing components are (1) a shared continuous variational autoencoder that compresses 48 kHz stereo waveforms into 128-dimensional latents at 25 Hz and is trained with an auxiliary semantic objective against a frozen language model, giving one representation for speech, music, and sound effects; (2) a Diffusion Transformer that denoises noise latents conditioned jointly on caption and text tokens; and (3) the structured record R=(G,P,E,U) - global scene and soundscape, role profiles, time-ordered events, primary audible content - which is rendered by a deterministic template into the textual condition. Semantic conditional views (full, dialogue, scene, empty) underpin classifier-fr

Load-bearing premise

The temporal-localization claim stands or falls on the assumption that the two large-language-model judges correctly identify which requested events are audible and when they occur; the paper reports no human validation and one judge belongs to the same model family as the proposed system.

What would settle it

Take a set of generated rich-timeline scenes, have human annotators mark the audible onset and offset of each requested event, and compute the mean intersection over union between those human intervals and the requested times. If human mIoU is near chance while the LLM judges' mIoU remains high, the reported temporal control is an artifact of the judge rather than the audio.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A complete mixed scene - dialogue, ambience, localized events, and background beds - can be generated as one waveform from a single text prompt, eliminating the separate-model-then-mix pipeline.
  • Character voices can be kept consistent across many dialogue turns in both English and Chinese without a task-specific multi-speaker branch.
  • Requested sound events can be placed near their specified start and end times in tens-of-second scenes, as measured by overlap between requested and judged event intervals.
  • A 25 Hz continuous latent with semantic supervision is sufficient for high-fidelity 48 kHz stereo generation across speech, music, and sound effects.
  • The same architecture can also perform zero-shot speaker similarity at a level comparable to dedicated TTS systems, so unification need not sacrifice voice fidelity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the temporal-localization result holds up under human listening, the structured-record format points toward a practical editing loop: change one event's timing or level and regenerate the scene while other components stay fixed, since each event is conditioned as an independent row on the timeline.
  • The paper's evaluation of timing uses two LLM judges and no human listeners; a natural next step is a human onset/offset annotation study or an acoustic event detector to confirm that the reported mIoU reflects audible timing rather than textual alignment.
  • The coverage-localization trade-off (the baseline recalls more events, the proposed model localizes them better) suggests that users may want a precision/recall control - for example through the classifier-free guidance scale - rather than a single fixed behavior.
  • Because the synthetic corpus is built from counterfactual recipes that change exactly one factor, the same data machinery could support controlled inference-time interventions; the paper trains on this structure but does not explicitly demonstrate an editing interface.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive audio generation system in which a Diffusion Transformer (DiT) generates continuous latents in a shared 25 Hz variational autoencoder (VAE) space, and the VAE decodes the complete 48 kHz stereo waveform. Structured textual conditions are produced from real annotations, synthetic recipe-generated scenes, and prompt-enhanced user requests. A two-stage data curriculum first trains broad standalone generation, then rich-timeline mixed-scene generation. The system is evaluated on Seed-TTS-Eval, an in-house reference-conditioned multi-speaker benchmark, an in-house rich-timeline benchmark, AudioCaps, SongBench, and VAE reconstruction/probe experiments. The reported strengths are speaker similarity on the public reference-conditioned benchmark, cross-turn speaker consistency and temporal localization on the in-house benchmarks, and LALM/AudioBox scores on AudioCaps. The paper claims that a single generation path can handle standalone audio, multi-speaker dialogue, and temporally structured mixed scenes without task-specific branches.

Significance. If the central claims hold, the system would be a meaningful step toward a single non-autoregressive model for heterogeneous long-form audio, a direction of substantial interest to the speech and audio generation community. The paper has real strengths: it anchors the core TTS-style evaluation on the public Seed-TTS-Eval benchmark; it reports a controlled downstream VAE probe in which the generator architecture, data, and optimization budget are held fixed; and its claims are carefully hedged, explicitly acknowledging where the model does not lead. However, the paper releases no code, checkpoints, or evaluation scripts, and the in-house benchmarks are small and cannot be independently inspected. The most important unresolved issue is that the temporal-localization claim rests on unvalidated LLM judge outputs and construction-time labels that the paper itself concedes may not correspond to perceptually audible events. With additional human validation and uncertainty quantification, this could become a solid technical report; without it, the paper's headline comparative strength is not yet established.

major comments (3)
  1. [§6, Table 6; §4.3] Table 6 is the sole direct evidence for the paper's headline temporal-localization claim. The mIoU gap (43.73 vs 38.48 under Gemini; 43.12 vs 37.36 under Qwen3.5) rests entirely on two LLM judges' estimates of which events are audible and when they occur. There is no human validation, no measure of judge reliability beyond rank agreement, and no confidence intervals or per-item analysis. Section 4.3 explicitly states that construction-time strong labels 'need not coincide perfectly with the perceptually audible activity'; if the reference intervals are not grounded in audibility, or if both judges share systematic bias (one judge is from the same Qwen family as the proposed system), the reported mIoU advantage would not reflect real temporal control. The proposed model also has substantially lower event recall (88.58 vs 98.73 under Gemini), so its superiority is only in localization metr
  2. [§6, Table 3; §4.2] The cross-turn consistency claim is based entirely on the in-house multi-speaker benchmark with a single baseline (Seed-Audio-1.0) and no uncertainty estimates. The proposed model wins CONS by 0.043 (EN) and 0.036 (ZH) while losing EN SIM by 0.061 and EN WER by 0.12; with roughly a hundred reference speakers per language and no per-item variance, these differences may be within speaker/sample variability. Reporting confidence intervals or pairwise significance tests over speakers/instances is necessary. Adding a second strong baseline (for example VibeVoice or Qwen-Audio-3.0-TTS) and describing the evaluation prompts would also make the claim more robust.
  3. [§6, Tables 4–5] The AudioCaps LALM advantage is a supporting strength but shares the validation gap: one of the three LALM judges (Qwen3.5-Omni-Plus) belongs to the same model family as the proposed system; the LALM scores are not calibrated against human judgment; and the subset is on the order of a hundred prompts with no confidence intervals. The Spearman correlations with CLAP are low (0.037–0.254), which the paper interprets as complementarity, but it also means the LALM measurements have no external anchor. To support the claim that the model's clearest advantages are in LALM/AudioBox assessments, the authors should report per-judge confidence intervals, agreement between judges on individual items, and at least a small human-rated alignment sample.
minor comments (5)
  1. [Throughout] There are several typographical artifacts from formatting, e.g., 'V AE' for 'VAE' and 'V oice' for 'Voice' (Sections 2.1 and 4.1). These should be cleaned up.
  2. [Table 3 caption/text] The sentence 'Compared with Seed-Audio-1.0 1' has a stray footnote marker; the footnote should be placed after the model name in the table or integrated cleanly.
  3. [Table 7] The parenthetical values from the reward-based in-house model are correctly marked as not directly comparable, but presenting them in the same table may confuse readers; consider moving them to a separate footnote or appendix.
  4. [§4.3] The 'limitations' paragraph is welcome, but it should be more explicit that the construction-time strong labels are used as reference intervals in Table 6; this connection is currently only implicit and is central to the temporal-localization evaluation.
  5. [§6.1, Tables 8–11] The VAE component evaluation is carefully designed with a fixed downstream generator, but no training/evaluation hyperparameters or code release are described. A brief reproducibility note would strengthen the report.

Circularity Check

0 steps flagged

No significant circularity: public benchmarks anchor the main claims; in-house benchmark limitations are evaluation risks, not circular reductions.

full rationale

I walked the derivation chain from the structured-record conditioning (Eq. 1, renderer Eq. 2) through semantic views and CFG (Eqs. 3-4) to the evaluations. The architecture claims (shared VAE, DiT, non-autoregressive path) are implemented and then evaluated rather than derived from the benchmarks. Seed-TTS-Eval, AudioCaps, SongBench, ViSQOL, and the LibriSpeech-PC-200 probe are external or controlled, so speaker-similarity, LALM, and reconstruction claims have independent grounding. The in-house multi-speaker and rich-timeline benchmarks are team-authored, and the rich-timeline metric relies on LLM judges with no human validation or uncertainty estimates; the paper itself concedes construction-time labels 'need not coincide perfectly with the perceptually audible activity' and 'no uncertainty estimates are reported.' This is a validity/robustness risk, but the reported mIoU values (~38-44%) show the judges are not simply copying the prompt's stated times, so the metric is not identical to the input by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from self-citations, and the self-citations ([7], [53], [39]) are baselines or judges, not load-bearing justifications. Thus no step reduces to its own inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The system's central claim rests mainly on engineering choices and evaluation assumptions rather than mathematical derivation. The key uncharged costs are the undisclosed data mixture, the hand-set loss weights/CFG scale, and the assumption that LLM judges and WavLM similarity faithfully measure temporal and identity fidelity.

free parameters (5)
  • CFG guidance scale s = not reported
    Eq. 4; the reported speaker-similarity and temporal-localization results are generated with a specific s, which is not given.
  • VAE loss weights lambda_KL, lambda_adv, lambda_fm, lambda_sem = not reported
    Eq. 6; these balance reconstruction, KL, adversarial, feature-matching, and semantic terms and are chosen by hand.
  • Post-training data mixture ratios = not reported
    Table 1 gives only qualitative 'relative scales' (Largest/Substantial/Small); the actual mixture is undisclosed but central to the claimed curriculum effect.
  • Semantic-view dropout distribution = not reported
    Eq. 3: mixture over views {full, dialogue, scene, empty} is 'configurable'; attribute-missing probabilities described in Sec. 5.2 are not quantified.
  • Prompt-enhancement token budget = not reported
    Sec. 4.4: renderer 'enforces a token budget, omitting low-priority detail by rule'; budget size is not given but determines which conditions reach the model.
axioms (5)
  • domain assumption WavLM speaker similarity (SIM) and cross-turn consistency (CONS) are valid proxies for voice identity preservation.
    Tables 2-3: the headline 'clearest strength' claims rely on these scalar similarities; no listening test or alternative identity metric is provided.
  • domain assumption LLM judges' event audibility and temporal-interval estimates are accurate measurements of event recall and mIoU.
    Table 6: Gemini-3.1-Pro-Preview and Qwen3.5-Omni-Plus determine audibility and intervals; no human validation of these judge outputs is reported.
  • domain assumption Frozen Qwen2.5-3B next-token prediction provides useful semantic supervision that transfers to downstream audio generation.
    Sec. 5.4: semantic continuation is claimed to improve downstream generation; the controlled probe (Table 11) is the only support and uses a task-specific F5-style generator, not the full system.
  • domain assumption Scaper-style synthetic mixtures with recipe-defined strong labels are a reliable training distribution for real mixed-scene audio.
    Sec. 4.3: counterfactual recipes are used to teach temporal/role organization; the paper concedes 'any downstream benefit of synthetic data must be established empirically.'
  • standard math Standard diffusion/classifier-free-guidance formalism (Eq. 4) is valid as used.
    Sec. 5.3: CFG combination vcfg = vu + s(vc - vu) is standard; no novel math is asserted.

pith-pipeline@v1.3.0-daily-deepseek · 20653 in / 16962 out tokens · 138660 ms · 2026-08-01T10:12:49.819211+00:00 · methodology

0 comments
read the original abstract

Existing single-domain and multi-task audio systems remain limited in directly organizing heterogeneous audio components, ambience, and multiple roles into long-form temporal scenes. We present Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework that uses a Diffusion Transformer (DiT) and a shared variational autoencoder (VAE) to generate the complete mixed waveform. Prompt enhancement converts free-form requests into structured temporal records that are rendered as textual conditions, while a two-stage data curriculum and semantic conditional views train the proposed model to use these conditions across standalone and mixed-scene audio. A shared continuous VAE compresses 48kHz stereo waveforms into 25Hz latent sequences and incorporates semantic supervision, providing one representation for heterogeneous audio. On the public reference-conditioned benchmark, speaker similarity is the proposed model's clearest strength across all three subsets. Across the multi-speaker and rich-timeline benchmarks, its clearest comparative strengths are cross-turn consistency in both languages and temporal localization, respectively. On AudioCaps, its advantages are concentrated in evaluations using large audio-language models and AudioBox. These results demonstrate the potential of unified generation for temporally structured audio without task-specific branches.

Figures

Figures reproduced from arXiv: 2607.27011 by Bajian Xiang, Cheng Wen, Han Zhao, Huaicheng Zhang, Jingbei Li, Junyu Dai, Lejun Min, Menglin Wu, Ruichen Zheng, Xiangang Li, Xiaoyue Duan, Xingchen Song, Xinyue Fan, Yihan Feng, Yiran Wang, Yufei Shi, Yunjia Li.

Figure 1
Figure 1. Figure 1: Overview of the unified non-autoregressive audio generation system. Caption and text [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the two-stage data curriculum. Single-domain speech, music, and sound-effect [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multi-level music annotation pipeline. Full-mix features describe rhythmic, tonal, and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 18 linked inside Pith

  1. [1]

    MusicLM: Generating music from text

    Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Seed-TTS: A family of high-quality versatile speech generation models, 2024

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, J...

    2024

  3. [3]

    Neural codec language models are zero-shot text to speech synthesizers

    Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33:705–718, 2025

    2025

  4. [4]

    F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025

    2025

  5. [5]

    MMAudio: Taming multimodal joint training for high-quality video-to-audio synthesis, 2025

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. MMAudio: Taming multimodal joint training for high-quality video-to-audio synthesis, 2025

    2025

  6. [6]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. InProceedings of the 37th International Conference on Neural Information Processing Systems, pages 47704–47720, 2023. 16

    2023

  7. [7]

    Pushing the frontier of full-song generation: Hierarchical autoregressive planning meets flow-matching rendering, 2026

    Junyu Dai, Xinyue Fan, Weiqin Li, Xiangang Li, Yunjia Li, Bin Ma, Yukun Ma, Chongjia Ni, Yufei Shi, Biao Tian, Haoxu Wang, Menglin Wu, Jianwei Yu, Huaicheng Zhang, Han Zhao, Shengkui Zhao, and Haina Zhu. Pushing the frontier of full-song generation: Hierarchical autoregressive planning meets flow-matching rendering, 2026

    2026

  8. [8]

    CosyV oice 3: Towards in-the-wild speech generation via scaling-up and post-training, 2025

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang, Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, Shiliang Zhang, Wen Wang, and Jieping Ye. CosyV oice 3: Towards in-the-wild speech generation via scaling-up and post-training, 2025

    2025

  9. [9]

    Audio-Oscar: A multi-agent system for complex audio scene generation, orchestration, and refinement.arXiv preprint arXiv:2606.07397, 2026

    Yifan Duan, Qixiang Xu, Hengtao Wu, Zhanxun Liu, Wenhao Guan, Junxi Liu, Ziyang Ma, Kelu Xu, and Xie Chen. Audio-Oscar: A multi-agent system for complex audio scene generation, orchestration, and refinement.arXiv preprint arXiv:2606.07397, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Stable Audio Open

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable Audio Open. InICASSP 2025–2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  11. [11]

    Text-to-audio generation using instruction guided latent diffusion model

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction guided latent diffusion model. InProceedings of the 31st ACM International Conference on Multimedia, pages 3590–3598, 2023

    2023

  12. [12]

    Qwen3-TTS technical report, 2026

    Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, Xinyu Zhang, Pei Zhang, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-TTS technical report, 2026

    2026

  13. [13]

    Make-An-Audio 2: Temporal-enhanced text-to-audio generation.arXiv preprint arXiv:2305.18474, 2023

    Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-An-Audio 2: Temporal-enhanced text-to-audio generation.arXiv preprint arXiv:2305.18474, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Algorithms to measure audio programme loudness and true-peak audio level

    International Telecommunication Union. Algorithms to measure audio programme loudness and true-peak audio level. Technical Report Recommendation ITU-R BS.1770-5, International Telecommunication Union, November 2023

    2023

  15. [15]

    ControlAudio: Tackling text-guided, timing-indicated and intelligible audio generation via progressive diffusion modeling

    Yuxuan Jiang, Zehua Chen, Zeqian Ju, Yusheng Dai, Weibei Dou, and Jun Zhu. ControlAudio: Tackling text-guided, timing-indicated and intelligible audio generation via progressive diffusion modeling. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1394–1413, 2026

    2026

  16. [16]

    AudioCaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)...

    2019

  17. [17]

    High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    2023

  18. [18]

    V oicebox: Text-guided multilingual universal speech generation at scale.Advances in Neural Information Processing Systems, 36:14005–14034, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oicebox: Text-guided multilingual universal speech generation at scale.Advances in Neural Information Processing Systems, 36:14005–14034, 2023

    2023

  19. [19]

    HoliTok: A coutinuous holistic tokenization with robust dual capabilities of speech generation and understanding.arXiv preprint arXiv:2605.29948, 2026

    Bohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi, Zhihan Li, Da Zheng, Colin Zhang, and Kai Yu. HoliTok: A coutinuous holistic tokenization with robust dual capabilities of speech generation and understanding.arXiv preprint arXiv:2605.29948, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    UNISON: A unified sound generation and editing framework via deep LLM fusion.arXiv preprint arXiv:2605.31530, 2026

    Zhaoqing Li, Haoning Xu, Jingran Su, Yaofang Liu, Zhefan Rao, Huimeng Wang, Jiajun Deng, Tianzi Wang, Zengrui Jin, Rui Liu, Haoxuan Che, and Xunying Liu. UNISON: A unified sound generation and editing framework via deep LLM fusion.arXiv preprint arXiv:2605.31530, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    AudioLDM: text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: text-to-audio generation with latent diffusion models. InProceedings of the 40th International Conference on Machine Learning, pages 21450–21474, 2023

    2023

  22. [22]

    AudioLDM 2: Learning holistic audio generation with self- supervised pretraining.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2871– 2883, 2024

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. AudioLDM 2: Learning holistic audio generation with self- supervised pretraining.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2871– 2883, 2024. 17

    2024

  23. [23]

    WavJourney: Compositional audio creation with large language models

    Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Qiushi Huang, Meng Cui, Jinhua Liang, Yin Cao, Qiuqiang Kong, Mark D Plumbley, et al. WavJourney: Compositional audio creation with large language models. IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  24. [24]

    UniMoE-Audio: Unified speech and music generation with dynamic-capacity MoE, 2025

    Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Xinyu Chen, Haoyuan Shi, Jinchao Li, Qi Wang, Haolan Chen, Fanbo Meng, Mingjun Zhao, Yu Xu, Yancheng He, Baotian Hu, and Min Zhang. UniMoE-Audio: Unified speech and music generation with dynamic-capacity MoE, 2025

    2025

  25. [25]

    The Song Describer Dataset: A corpus of audio captions for music-and-language evaluation.arXiv preprint arXiv:2311.10057, 2023

    Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. The Song Describer Dataset: A corpus of audio captions for music-and-language evaluation.arXiv preprint arXiv:2311.10057, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Dasheng AudioGen: A unified model for generating coherent audio scenes from text.arXiv preprint arXiv:2605.27838, 2026

    Jiahao Mei, Heinrich Dinkel, Yadong Niu, Xingwei Sun, Gang Li, Yifan Liao, Jiahao Zhou, Junbo Zhang, Jian Luan, and Mengyue Wu. Dasheng AudioGen: A unified model for generating coherent audio scenes from text.arXiv preprint arXiv:2605.27838, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Semantic-V AE: Semantic-alignment latent representation for better speech synthesis.arXiv preprint arXiv:2509.22167, 2025

    Zhikang Niu, Shujie Hu, Jeongsoo Choi, Yushen Chen, Peining Chen, Pengcheng Zhu, Yunting Yang, Bowen Zhang, Jian Zhao, Chunhui Wang, et al. Semantic-V AE: Semantic-alignment latent representation for better speech synthesis.arXiv preprint arXiv:2509.22167, 2025

    arXiv 2025

  28. [28]

    LibriSpeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: An ASR corpus based on public domain audio books. In2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE, 2015

    2015

  29. [29]

    SAME: A semantically-aligned music autoencoder.arXiv preprint arXiv:2605.18613, 2026

    Julian D Parker, Zach Evans, CJ Carr, Zachary Zukowski, Josiah Taylor, Matthew Rice, and Jordi Pons. SAME: A semantically-aligned music autoencoder.arXiv preprint arXiv:2605.18613, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    VibeV oice: Expressive podcast generation with next-token diffusion

    Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, Shaohan Huang, Yan Xia, and Furu Wei. VibeV oice: Expressive podcast generation with next-token diffusion. InICLR 2026, February 2026

    2026

  31. [31]

    UniSonate: A unified model for speech, music, and sound effect generation with text instructions

    Chunyu Qiang, Xiaopeng Wang, Kang Yin, Yuzhe Liang, Yuxin Guo, Teng Ma, Ziyu Zhang, Tianrui Wang, Cheng Gong, Yushen Chen, et al. UniSonate: A unified model for speech, music, and sound effect generation with text instructions. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28043–28...

    2026

  32. [32]

    Roman, Christopher Ick, Sivan Ding, Adrian S

    Iran R. Roman, Christopher Ick, Sivan Ding, Adrian S. Roman, Brian McFee, and Juan P. Bello. Spatial Scaper: A library to simulate and augment soundscapes for sound event localization and detection in realistic rooms. InICASSP 2024–2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1221–1225. IEEE, 2024

    2024

  33. [33]

    Scaper: A library for soundscape synthesis and augmentation

    Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello. Scaper: A library for soundscape synthesis and augmentation. In2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 344–348. IEEE, 2017

    2017

  34. [34]

    NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Jiang Bian, et al. NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. InInternational Conference on Learning Representations, volume 2024, pages 698–722, 2024

    2024

  35. [35]

    Borderless long speech synthesis.arXiv preprint arXiv:2603.19798, 2026

    Xingchen Song, Di Wu, Dinghao Zhou, Pengyu Cheng, Hongwu Ding, Yunchao He, Jie Wang, Shengfan Shen, Sixiang Lv, Lichun Fan, Hang Su, Yifeng Wang, Shuai Wang, Meng Meng, and Jian Luan. Borderless long speech synthesis.arXiv preprint arXiv:2603.19798, 2026

    Pith/arXiv arXiv 2026

  36. [36]

    F5R-TTS: Improving flow-matching based text-to-speech with group relative policy optimization, 2025

    Xiaohui Sun, Ruitong Xiao, Jianye Mo, Bowen Wu, Qun Yu, and Baoxun Wang. F5R-TTS: Improving flow-matching based text-to-speech with group relative policy optimization, 2025

    2025

  37. [37]

    Foley-Omni: A unified multimodal generation model from task-level audio synthesis to complete video soundtrack generation.arXiv preprint arXiv:2606.03672, 2026

    Ye Tao, Lupeng Liu, Xuenan Xu, Jiasun Feng, Jiarui Wang, Ying Qin, Shuiyang Mao, Wei Liu, and Shuai Wang. Foley-Omni: A unified multimodal generation model from task-level audio synthesis to complete video soundtrack generation.arXiv preprint arXiv:2606.03672, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    MiMo-Audio: Audio language models are few-shot learners, 2025

    Core Team, Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhuai Ren, Shuo Liu, Tao Guo, Weiji Zhuang, Xin Zhang, Xingchen Song, Yihan Yan, Yongzhe He, Cici, Bowen Shen, Chengxuan Zhu, Chong Ma, Chun Chen, Heyu Chen, Jiawei Li, Lei Li, Menghang Zhu, Peidian Li, Qiying Wang, Sirui Deng, Weimin Xiong, Wenshan Huang, Wenyu Yang, Yilin Jia...

    2025

  39. [39]

    Qwen3.5-Omni technical report, 2026

    Qwen Team. Qwen3.5-Omni technical report, 2026

    2026

  40. [40]

    Bagpiper: Solving open-ended audio tasks via rich captions

    Jinchuan Tian, Haoran Wang, Bo-Hao Su, Chien-yu Huang, Qingzheng Wang, Jiatong Shi, William Chen, Xun Gong, Siddhant Arora, Chin-Jou Li, et al. Bagpiper: Solving open-ended audio tasks via rich captions. arXiv preprint arXiv:2602.05220, 2026

    Pith/arXiv arXiv 2026

  41. [41]

    AudioX: Diffusion transformer for anything-to-audio generation.arXiv preprint arXiv:2503.10522, 2025

    Zeyue Tian, Yizhu Jin, Zhaoyang Liu, Ruibin Yuan, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. AudioX: Diffusion transformer for anything-to-audio generation.arXiv preprint arXiv:2503.10522, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Audio-Omni: Extending multi-modal understanding to versatile audio generation and editing

    Zeyue Tian, Binxin Yang, Zhaoyang Liu, Jiexuan Zhang, Ruibin Yuan, Hubery Yin, Qifeng Chen, Chen Li, Jing Lyu, Wei Xue, et al. Audio-Omni: Extending multi-modal understanding to versatile audio generation and editing. InProceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pages 1–10, 2026

    2026

  43. [43]

    Meta Au- diobox Aesthetics: Unified automatic quality assessment for speech, music, and sound.arXiv preprint arXiv:2502.05139, 2025

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, Carleigh Wood, Ann Lee, and Wei-Ning Hsu. Meta Au- diobox Aesthetics: Unified automatic quality assessment for speech, music, and sound.arXiv preprint arXiv:2502.05139, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Sound event detection in domestic environments with weakly labeled data and soundscape synthesis

    Nicolas Turpault, Romain Serizel, Ankit Parag Shah, and Justin Salamon. Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. InProceedings of the Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), pages 253–257, New York, NY , USA, 2019

    2019

  45. [45]

    Audiobox: Unified audio generation with natural language prompts.arXiv preprint arXiv:2312.15821, 2023

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts.arXiv preprint arXiv:2312.15821, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Back to ear: Perceptually driven high fidelity music reconstruction.arXiv preprint arXiv:2509.14912, 2025

    Kangdi Wang, Zhiyue Wu, Dinghao Zhou, Rui Lin, Junyu Dai, and Tao Jiang. Back to ear: Perceptually driven high fidelity music reconstruction.arXiv preprint arXiv:2509.14912, 2025

    arXiv 2025

  47. [47]

    MaskGCT: Zero-shot text-to-speech with masked generative codec transformer

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. MaskGCT: Zero-shot text-to-speech with masked generative codec transformer. InInternational Conference on Learning Representations, volume 2025, pages 47127–47150, 2025

    2025

  48. [48]

    T2A-Feedback: Improving basic capabilities of text-to-audio generation via fine-grained AI feedback

    Zehan Wang, Ke Lei, Chen Zhu, Jiawei Huang, Sashuai Zhou, Luping Liu, Xize Cheng, Shengpeng Ji, Zhenhui Ye, Tao Jin, et al. T2A-Feedback: Improving basic capabilities of text-to-audio generation via fine-grained AI feedback. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23535–23547, 2025

    2025

  49. [49]

    MuChin: A Chinese colloquial description benchmark for evaluating language models in the field of music.arXiv preprint arXiv:2402.09871, 2024

    Zihao Wang, Shuyu Li, Tao Zhang, Qi Wang, Pengfei Yu, Jinyang Luo, Yan Liu, Ming Xi, and Kejun Zhang. MuChin: A Chinese colloquial description benchmark for evaluating language models in the field of music.arXiv preprint arXiv:2402.09871, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Scott Wisdom, Hakan Erdogan, Daniel P. W. Ellis, Romain Serizel, Nicolas Turpault, Eduardo Fonseca, Justin Salamon, Prem Seetharaman, and John R. Hershey. What’s all the FUSS about free universal sound separation data? InICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 186–190. IEEE, 2021

    2021

  51. [51]

    Step-Audio 2 technical report, 2025

    Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zho...

    2025

  52. [52]

    SongBench: A fine-grained multi-aspect benchmark for song quality assessment.arXiv preprint arXiv:2604.25937, 2026

    Dapeng Wu, Shun Lei, Wei Tan, Guangzheng Li, Yunzhe Wang, Huaicheng Zhang, Lishi Zuo, and Zhiyong Wu. SongBench: A fine-grained multi-aspect benchmark for song quality assessment.arXiv preprint arXiv:2604.25937, 2026

    Pith/arXiv arXiv 2026

  53. [53]

    Qwen-Audio-3.0-TTS: Freely controllable and highly robust speech synthesis with multi-stage training paradigm, 2026

    Bajian Xiang, Cheng Wen, Han Zhao, Hao Wang, Haoxu Wang, Jiawei Jin, Jiayan Cui, Jie Chen, Mengxi Nie, Tianyu Zhao, Weiqin Li, Xiang Lv, Xiangang Li, Yang Xiang, and Yang Zhou. Qwen-Audio-3.0-TTS: Freely controllable and highly robust speech synthesis with multi-stage training paradigm, 2026

    2026

  54. [54]

    FireRedTTS-2: Towards long conversational speech generation for podcast and chatbot, 2025

    Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. FireRedTTS-2: Towards long conversational speech generation for podcast and chatbot, 2025

    2025

  55. [55]

    AudioTime: A temporally-aligned audio-text benchmark dataset

    Zeyu Xie, Xuenan Xu, Zhizheng Wu, and Mengyue Wu. AudioTime: A temporally-aligned audio-text benchmark dataset. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  56. [56]

    PicoAudio: Enabling precise temporal control- lability in text-to-audio generation

    Zeyu Xie, Xuenan Xu, Zhizheng Wu, and Mengyue Wu. PicoAudio: Enabling precise temporal control- lability in text-to-audio generation. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  57. [57]

    LongCat-AudioDiT: High-fidelity diffusion text-to-speech in the waveform latent space, 2026

    Detai Xin, Shujie Hu, Chengzuo Yang, Chen Huang, Guoqiao Yu, Guanglu Wan, and Xunliang Cai. LongCat-AudioDiT: High-fidelity diffusion text-to-speech in the waveform latent space, 2026

    2026

  58. [58]

    Qwen3-Omni technical report, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo ...

    2025

  59. [59]

    UniFlow-Audio: Unified flow matching for audio generation from omni-modalities

    Xuenan Xu, Jiahao Mei, Zihao Zheng, Ye Tao, Zeyu Xie, Yaoyun Zhang, Haohe Liu, Yuning Wu, Ming Yan, Wen Wu, et al. UniFlow-Audio: Unified flow matching for audio generation from omni-modalities. arXiv preprint arXiv:2509.24391, 2025

    arXiv 2025

  60. [60]

    Ming-UniAudio: Speech LLM for joint understanding, generation and editing with unified representation, 2025

    Canxiang Yan, Chunxiang Jin, Dawei Huang, Haibing Yu, Han Peng, Hui Zhan, Jie Gao, Jing Peng, Jingdong Chen, Jun Zhou, Kaimeng Ren, Ming Yang, Mingxue Yang, Qiang Xu, Qin Zhao, Ruijie Xiong, Shaoxiong Lin, Xuezhi Wang, Yi Yuan, Yifei Wu, Yongjie Lyu, Zhengyu He, Zhihao Qiu, Zhiqiang Fang, and Ziyuan Huang. Ming-UniAudio: Speech LLM for joint understanding...

    2025

  61. [61]

    UniAudio: An audio foundation model toward universal audio generation.arXiv preprint arXiv:2310.00704, 2023

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Zhou Zhao, Xixin Wu, and Helen Meng. UniAudio: An audio foundation model toward universal audio generation.arXiv preprint arXiv:2310.00704, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    MiniMax-Speech: Intrinsic zero-shot text-to-speech with a learnable speaker encoder, 2025

    Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, Mingqi Yang, Peikai Huang, Ruiyang Jin, Sitan Jiang, Weihua Cheng, Yawei Li, Yichen Xiao, Yiying Zhou, Yongmao Zhang, Yuan Lu, and Yucen He. MiniMax-Speech: Intrinsic zero-shot text-to-speech with a learnable speaker encoder, 2025

    2025

  63. [63]

    LoSATok: Low- dimensional semantic-acoustic tokenizer for cross-domain audio understanding and generation.arXiv preprint arXiv:2605.27840, 2026

    Zhisheng Zhang, Xiang Li, Yixuan Zhou, Jing Peng, Guoyang Zeng, and Zhiyong Wu. LoSATok: Low- dimensional semantic-acoustic tokenizer for cross-domain audio understanding and generation.arXiv preprint arXiv:2605.27840, 2026

    Pith/arXiv arXiv 2026

  64. [64]

    PicoAudio2: Temporal controllable text-to-audio generation with natural language description

    Zihao Zheng, Zeyu Xie, Xuenan Xu, Wen Wu, Chao Zhang, and Mengyue Wu. PicoAudio2: Temporal controllable text-to-audio generation with natural language description. InICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 14507–14511. IEEE, 2026

    2026

  65. [65]

    V oxCPM2 technical report, 2026

    Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Jiancheng Gui, Jiaheng Wu, Ziyang Wang, Xudong Shen, Runchuan Ye, Zhisheng Zhang, Jiuyang Zhou, Bingsong Bai, Weiyue Sun, Mengyuan Deng, Qundong Shi, Zhiyong Wu, and Zhiyuan Liu. V oxCPM2 technical report, 2026. 20

    2026

  66. [2019]

    Association for Computational Linguistics