Pith. sign in

REVIEW 4 major objections 6 minor 58 references

SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One flow-matching model generates speech, music, singing, and sound effects together by routing contiguous audio chunks through a conflict-gated prior–evidence mixture-of-experts.

desk verdict CPE-MoE is a real new routing mechanism, but the paper's central claim about chunk granularity needs the missing token-level control before it fully holds. read the letter →

arxiv 2608.09571 v1 pith:CMV73VJR submitted 2026-08-10 cs.SD

Yunrui Cai , Xu Li , Yucheng Zhou , Jinchao Li , Dingdong Wang , Dongchao Yang , Xixin Wu , Chen Zhang
show 3 more authors
Zhiyong Wu Pengfei Wan Helen Meng
This is my paper · ORCID
classification cs.SD
keywords unifiedaudiogenerationmixture-of-expertschunk-levelroutingconflict-gatedflowmatchingtext-to-audiocomplexscenecompositionalfidelity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SonicWeave sets out to show that a single flow-matching generator can compose speech, singing, music, sound effects, and their mixtures, and that the right conditional-computation design—routing contiguous chunks of audio through experts under a learned balance between a global text-and-phase prior and local acoustic evidence—is what makes unified generation work. The paper claims that this chunk-routed, conflict-gated mixture-of-experts (CPE-MoE) consistently beats matched dense and token-routed MoE baselines on TTS, TTA, and TTM benchmarks, and that its largest gains appear in complex scenes where components overlap or alternate. A sympathetic reader would care because unified audio generation has been blocked by the conflicting structural demands of speech versus music versus effects, and this is a concrete recipe for allocating computation flexibly within a single scene without sacrificing quality.

What carries the argument

The central object is CPE-MoE (Conflict-gated Prior–Evidence Mixture-of-Experts), which replaces the dense feed-forward network in the final four transformer layers. For each contiguous chunk of acoustic frames it fuses two routing signals—a global prior from the structured caption and diffusion-time embedding, and a local evidence vector pooled from the emerging audio state—through a learned conflict gate, then performs top-2 expert dispatch at chunk granularity while a shared expert handles text and time tokens. The chunk is the routing unit, not the representation unit: expert selection is shared within a chunk, but experts transform the original frame states, preserving frame-level detail while enforcing local computational continuity. A Switch-Transformer-style auxiliary loss prevents expert collapse, and the gate is trained by the generation objective rather than as a calibrated probability of evidence correctness.

What would settle it

Retrain Base-MoE and Dense with the same per-model hyperparameter sweep and several random seeds; if the reported WER, CER, FAD, KL, CLAP, and MOS-R differences collapse to within seed-to-seed noise on the same benchmarks, then the central claim that CPE-MoE routing causes the improvement is refuted.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that prior–evidence chunk routing is a useful intermediate granularity for unified audio generation. Concretely, CPE-MoE partitions audio latents into chunks of four frames, computes a global prior logit vector from the structured text summary and diffusion phase, computes a local evidence logit vector from the pooled chunk state, and lets a learned conflict gate interpolate between them: $\ell_j = (1-g_j)\ell_{\text{prior}} + g_j\ell_{\text{evid}}$. Text and time tokens bypass the routing gate through a shared expert, and each selected expert still transforms the original frame states, so pooling controls selection without discarding frame-level detail. The paper reports content-dependent expert specialization across layers and diffusion phases, and credits this mechanism for improved compositional quality—higher semantic adherence and request realization—while keeping perceptual quality on par with the token-MoE control.

Load-bearing premise

The Dense and Base-MoE controls are exactly matched to SonicWeave—same data, backbone, and training budget—so the measured gaps must be attributable to the chunk-routed gating design itself.

Editorial extensions

If this is right

  • A single set of weights can generate speech, music, singing, sound effects, and mixtures from structured captions, so model count need not scale with task count.
  • Chunk size four emerges as the trade-off point: fine enough to react to short events and speaker turns, coarse enough to keep locally coherent audio on one expert path.
  • The conflict gate yields a phase-adaptive prior-to-evidence trajectory, meaning early denoising leans on the text prior and later refinement leans on acoustic evidence.
  • Compositional fidelity and perceptual quality are separable: the routing module improves the former while leaving the latter close to a matched token-MoE.
  • Structured captions that separate foreground, background, and texture provide a stable conditioning pathway that the routing prior can exploit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the matched-baseline comparison is taken at face value, the same chunking and gate design could transfer to other conditional diffusion transformers, including image and video generation, wherever a global prompt and a local state coexist.
  • The paper's routing analyses suggest that gate values could double as a per-region confidence signal; using them to switch between prior-driven and evidence-driven generation at inference is a testable extension the paper does not pursue.
  • The reported gains on complex scenes, if robust, imply that evaluation of unified audio models should include compositional metrics, since single-task benchmarks understate the benefit of this routing design.
  • Because the paper reports no variance or significance testing, a multi-seed re-run with per-model hyperparameter tuning is the direct check on whether chunk-routed gating causes the reported improvements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SonicWeave, a unified flow-matching model for audio scene generation, with a core CPE-MoE module that routes contiguous audio chunks through a mixture of experts using a global text-and-diffusion-phase prior, local acoustic evidence, and a learned conflict gate. The method is evaluated on public TTS, TTA, and TTM benchmarks, on a curated 100-prompt complex-scene suite with both a reference-free LLM judge and human listening, and through routing analyses. The central claim is that chunk-level prior-evidence routing offers a useful intermediate granularity between domain-level and token-level conditional computation for unified audio generation.

Significance. If the central claim is established, SonicWeave would be a useful contribution to unified audio generation: the chunking idea is well motivated by acoustic continuity, the structured-caption interface is practical, and the controlled comparisons against matched Dense and Base-MoE baselines on public tasks are a genuine strength. The paper is also unusually transparent about protocols in the appendix, including sampling settings, the structured-prompt conversion rules, and the routing analysis machinery. However, the specific claim of an 'intermediate granularity' is not directly tested by the reported experiments, and the evaluation lacks variance or significance information on the objective metrics. The contribution is therefore promising but currently conditional on additional controls.

major comments (4)
  1. [§4.1, §E.3, §5] The central claim that chunk-level routing is a 'useful intermediate granularity' is not tested by the reported controls. The Base-MoE baseline changes two factors at once relative to CPE-MoE: it routes individual tokens rather than chunks, and it omits the conflict-gated prior-evidence fusion. The ablations in Appendix E hold the router design fixed at C=4 and C=8, so no comparison isolates granularity. A token-level CPE-MoE variant with C=1 that retains the same prior, evidence, and conflict gate, and ideally a clip-level variant with C=T, are needed to attribute the observed gains to chunking. Without them, the improvements could come from the prior-evidence router or the learned gate, and C=4 could outperform C=8 simply because finer routing is better. The paper itself concedes 'non-exhaustive routing ablations' in the conclusion, but this is the ablation that the paper's title and conclusion depend on.
  2. [Tables 2–5, §C.1] The controlled comparisons are reported as point estimates from a single training run with no variance or significance information. The TTS gains over Base-MoE are 0.4, 0.3, and 0.8 percentage points in Table 2, and MOS-Q is tied at 4.57 in Table 5, so the statement that SonicWeave 'consistently improves' over Base-MoE would be much more convincing with multiple seeds, confidence intervals, or significance tests. Section C.1 describes the shared data, backbone, and training budget but does not state that the controlled models were trained with multiple seeds, nor does it document the hyperparameter search budget for each baseline, so the claim that the baselines are exactly matched controls is not fully verifiable.
  3. [§4.3, §D.3, §D.4] The Complex-Scene headline metrics, AI-Tech and AI-Sem in Table 5, come from a reference-free Gemini judge, but the paper reports no validation of that judge against the human ratings and no inter-rater agreement statistics, despite the detailed seven-dimensional rubric in Appendix D.3. The human study covers only 25 of the 100 prompts with 25 listeners, and the MOS numbers are reported as means with standard deviations across listeners only; the MOS-R difference between SonicWeave and Base-MoE is 4.49 versus 4.31 with overlapping standard deviations. Given the central role of complex-scene compositional fidelity in the conclusions, the evaluation needs either judge-human correlation and agreement statistics, or an explicit statement that the reference-free scores are exploratory.
  4. [§D.2, Table 5] The comparison against external systems in the Complex-Scene suite is confounded by the prompt interface. Public systems receive natural-language prompts while SonicWeave, Dense, and Base-MoE receive structured captions; since structured conditioning is itself a contribution of the paper (Section 3.2), the MOS-R gains over Higgs Audio V2 and Dasheng AudioGen in Table 5 cannot be attributed to CPE-MoE. The conclusion's statement of improvement 'over the strongest external baseline' should be framed as a full-system comparison, not as evidence for the routing contribution, and the text should acknowledge that the external systems were not given the structured interface.
minor comments (6)
  1. [§4.1] Please report the exact parameter counts for Dense, Base-MoE, and SonicWeave; the term 'parameter-matched' is load-bearing for the controlled comparison and should be quantified.
  2. [§C.3] The APG guidance uses eta=0.85 for speech-only generation and eta=0.5 for scenes with sound effects or music; state whether these values were selected per task or benchmark and consider a sensitivity analysis over w and eta, since inference-time hyperparameters can affect the comparison.
  3. [Equation (5)] The chunk-pooling operation is simple mean pooling; a brief discussion of why mean pooling is preferred over max or attention pooling for the routing decision would help the reader assess the design.
  4. [Figure 5, §F.4] The mixed-scene routing trace is clearly labeled as illustrative, but the caption's claim that speech regions 'show stronger reliance on the text prior' should be accompanied by an uncertainty or cross-example summary rather than a single demo prompt.
  5. [References] Several references carry 2026 dates and some are arXiv preprints; please verify all bibliographic details at the final submission stage, especially for Dasheng AudioGen and UniMoE-Audio, since these are moving targets.
  6. [Appendix C.4] The paper states that all reported results should use the fixed sampling protocol 'rather than selecting a schedule per example'; it would be helpful to also state explicitly that no per-benchmark selection of the guidance scale w was performed, or to list the w values used if they varied.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CPE-MoE is trained end-to-end and evaluated on held-out public benchmarks; no reported result is a fitted parameter renamed as a prediction.

full rationale

The central claims are empirical: SonicWeave is trained with a composite flow-matching, contrastive, and load-balancing objective, and its quality is measured on held-out public benchmarks (TTS, TTA, TTM) and a complex-scene suite against matched Dense and Base-MoE controls. There is no derivation chain in which an output quantity is equivalent by construction to an input quantity. The router and conflict gate are learned parameters, not fitted to the evaluation outcomes, and the routing analyses in Figures 3-6 are descriptive post-hoc observations rather than predictions that reduce to the model's own fitting procedure. The paper does cite prior work with overlapping authors, notably UniAudio [6] and SegTune [25], but these are contextual related-work citations and are not load-bearing for the paper's main empirical conclusions. The strongest methodological concern is a control gap: Base-MoE differs from CPE-MoE in both routing granularity and router inputs, and no C=1 token-level CPE-MoE variant is reported, so the specific benefit of chunk-level routing is not perfectly isolated. That is an experimental-design limitation, not circularity. The paper itself acknowledges non-exhaustive routing ablations in its conclusion, which further confirms that the claim is offered as an empirical finding rather than a definitional tautology. The reader's observation that the model's own text and audio representations feed both routing and output is a standard property of conditional computation and does not constitute a circular reduction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several hand-chosen hyperparameters (chunk size, expert count, loss weights, guidance scales) and on assumptions about the pretrained encoders and the structured-caption interface. No new physical or conceptual entities are introduced beyond the CPE-MoE module.

free parameters (5)
  • Chunk size C = 4
    Routing chunk length; C=4 was chosen after showing C=8 degrades metrics (Appendix E.3, Table 9). Not derived from data.
  • Routed experts and top-K = 4 experts, top-2
    Design choice for the MoE layer; no sensitivity study reported (Section 3.4, Table 6).
  • Loss weights alpha_CL, alpha_MoE = 0.1, 0.05
    Weights for contrastive and load-balancing losses in Eq. (15); chosen by hand (Table 6).
  • Guidance scale w and APG dampening eta = w=4.5, eta=0.85/0.5
    Inference hyperparameters for APG; selected for best benchmark results, no validation protocol described (Sections 3.6, C.4).
  • Reference masking span fraction = U[0.7,1.0]
    Training choice for continuation/inpainting mask length (Appendix C.2).
assumptions (6)
  • standard math Flow-matching objective and linear interpolation path are standard and unbiased.
    Used in Section 3.5; unproved background for the training objective.
  • domain assumption Pretrained stereo VAE and frozen Qwen3-VL-8B text encoder provide reliable representations.
    Assumed to be adequate for audio and text; their limitations are not studied.
  • domain assumption Chunk-pooled state is a sufficient summary for routing decisions.
    Eq. (5) pools chunk states for routing; no analysis of information loss.
  • ad hoc to paper Conflict gate trained by generation objective learns meaningful reliability.
    The gate is not calibrated to any external ground truth; its behavior is interpreted post hoc in Section 4.4.
  • domain assumption Structured caption conversion is lossless with respect to the source annotation.
    Appendices B.2-B.3 assert the converter adds no information; this is a design claim that could fail.
  • ad hoc to paper Fixed 1/2,1/2 shared-sparse composition prevents gate collapse.
    Design choice in Eq. (12) with no empirical comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation." pith.science (2026). https://pith.science/paper/CMV73VJR

@misc{pith2026260809571,
  author       = {Pith},
  title        = {Pith review of: SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMV73VJR}},
  note         = {Machine review of arXiv:2608.09571}
}
read the original abstract

Text-conditioned general audio generation is moving beyond isolated speech, music, and sound-effect synthesis toward a single model that can compose them into controllable, coherent audio scenes. This unified setting is particularly challenging: heterogeneous components impose conflicting structural requirements on a shared backbone, while a complex mixed scene may contain locally distinct or overlapping content that demands fine-grained adaptation within the same clip. Existing audio mixture-of-experts (MoEs) mainly route at the domain level, while token-wise routing overlooks the local continuity inherent to acoustic signals. We propose SonicWeave, a flow-matching model for unified audio scene generation. At its core is a chunk-routed MoE with a conflict-gated prior-evidence routing mechanism (CPE-MoE). CPE-MoE routes contiguous acoustic chunks by combining a global prior that encodes the structured text condition and diffusion phase with local evidence from the evolving acoustic state. A learned conflict gate favors the prior when local states are unreliable, while allowing local evidence to influence routing when a region departs from the global scene context. SonicWeave supports speech, music, sound effects, singing, and their fine-grained mixtures with a single set of weights. Across TTS, TTA, and TTM benchmarks, SonicWeave consistently improves over controlled Dense and Base-MoE baselines. Complex-scene evaluation further demonstrates improved compositional quality, while routing analyses reveal content-dependent expert specialization across diffusion phases. These results suggest that temporally coherent, prior-evidence routing is an effective conditional-computation strategy for unified audio generation. Project page: https://caiyunrui.github.io/SonicWeave.

Figures

Figures reproduced from arXiv: 2608.09571 by the authors.

Figure 1
Figure 1. Overview of SonicWeave for unified audio scene generation. Natural-language descriptions are converted into structured captions that separate speech content, vocal attributes, music, sound effects, and ambience. A single model then synthesizes these heterogeneous components jointly, while its CPE-MoE module provides conflict-gated, content-aware routing across acoustic chunks. expert, preserving a stable conditionin… view at source ↗
Figure 2
Figure 2. SonicWeave framework. Left: a stereo VAE encodes audio into continuous latents, which are jointly modeled with a structured-caption stream in a flow-matching DiT. Right: the CPE-MoE module groups audio tokens into chunks and fuses a global text-and-phase prior with local acoustic evidence for sparse expert routing. 3 Method 3.1 Overview Our dual-stream conditional flow-matching generator, Son￾icWeave, jointly models… view at source ↗
Figure 3
Figure 3. Routing analysis on the balanced routing benchmark. (a) Content-conditioned expert dispatch across routed layers, showing non-uniform and content-dependent specialization. (b) Mean conflict gate E[gj |t] across diffusion phases, showing content-dependent and phase-dependent prior–evidence balancing. and SFX favor different expert combinations from music and singing, while mixed scenes retain a distinct distribution … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Phase- and content-conditioned expert usage. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Conflict-gate dynamics and mixed-scene routing. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Gate-controlled prior–evidence routing. Valid chunks are grouped by their conflict-gate values. At low gj , the fused posterior is close to the global prior; at high gj , it is close to local evidence. JSD denotes Jensen–Shannon divergence. The upper trace shows the co…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 42 canonical work pages

  1. [1]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxi- milian Nickel, and Matthew Le. Flow matching for gener- ative modeling. InThe Eleventh International Conference on Learning Representations, pages 1–28, 2023

    work page 2023

  2. [2]

    V oicebox: Text-guided multilingual universal speech generation at 8 scale.Advances in neural information processing systems, 36:14005–14034, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oicebox: Text-guided multilingual universal speech generation at 8 scale.Advances in neural information processing systems, 36:14005–14034, 2023

    work page 2023

  3. [3]

    F5- TTS: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5- TTS: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 6255–6271, 2025

    work page 2025

  4. [4]

    Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization

    Chia-Yu Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Zadeh, Chuan Li, Rafael Valle, Bryan Catanzaro, and Soujanya Poria. Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization. InThe F ourteenth International Conference on Learning Representations, pages 1–24, 2026

    work page 2026

  5. [5]

    Plumbley

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. AudioLDM 2: Learning holistic audio generation with self-supervised pretrain- ing.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2871–2883, 2024

    work page 2024

  6. [6]

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Zhou Zhao, Xixin Wu, and Helen M. Meng. UniAudio: Towards universal audio generation with large language models. InProceedings of the 41st International Conference on Machine Learning, pages 56422–56447, 2024

    work page 2024

  7. [7]

    Audiobox: Unified audio generation with natural language prompts.CoRR, abs/2312.15821,

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Sum- mers, Carleigh Wood, Joshua Lane, Mary Williamson, and Wei-Ning Hsu. Audiobox: Unifi...

    arXiv

  8. [8]

    Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

    Jiahao Mei, Heinrich Dinkel, Yadong Niu, Xingwei Sun, Gang Li, Yifan Liao, Jiahao Zhou, Junbo Zhang, Jian Luan, and Mengyue Wu. Dasheng AudioGen: A unified model for generating coherent audio scenes from text. CoRR, abs/2605.27838, 2026. doi: 10.48550/arXiv.2605. 27838

    work page Pith review arXiv doi:10.48550/arxiv.2605.27838 2026

Show all 58 references
  1. [9]

    UniMoE-Audio: Unified speech and music generation with dynamic- capacity mixture-of-experts

    Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Xinyu Chen, Haoyuan Shi, Haolan Chen, Fanbo Meng, Mingjun Zhao, Yu Xu, Yancheng He, Bao- tian Hu, Haizhou Li, and Min Zhang. UniMoE-Audio: Unified speech and music generation with dynamic- capacity mixture-of-exp...

    2026

  2. [10]

    Mandic, Wenwu Wang, and Mark D

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Text-to-audio generation with latent diffu- sion models. InProceedings of the 40th International Conference on Machine Learning, pages 21450–21474, 2023

    2023

  3. [11]

    UniSonate: A unified model for speech, music, and sound effect generation with text instructions

    Chunyu Qiang, Xiaopeng Wang, Kang Yin, Yuzhe Liang, Yuxin Guo, Teng Ma, Ziyu Zhang, Tianrui Wang, Cheng Gong, Yushen Chen, Ruibo Fu, Longbiao Wang, and Jianwu Dang. UniSonate: A unified model for speech, music, and sound effect generation with text instructions. InProceedings ...

    2026

  4. [12]

    Make-An-Audio 2: Temporal-enhanced text-to-audio generation.CoRR, abs/2305.18474, 2023

    Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-An-Audio 2: Temporal-enhanced text-to-audio generation.CoRR, abs/2305.18474, 2023. doi: 10.48550/arXiv.2305.18474

    2023 doi arXiv

  5. [13]

    Text-to-audio generation using in- struction guided latent diffusion model

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using in- struction guided latent diffusion model. InProceedings of the 31st ACM international conference on multimedia, pages 3590–3598, 2023

    2023

  6. [14]

    Freeaudio: Training-free timing plan- ning for controllable long-form text-to-audio generation

    Yuxuan Jiang, Zehua Chen, Zeqian Ju, Chang Li, Weibei Dou, and Jun Zhu. Freeaudio: Training-free timing plan- ning for controllable long-form text-to-audio generation. InProceedings of the 33rd ACM International Confer- ence on Multimedia, pages 9871–9880, 2025

    2025

  7. [15]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learn- ing Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learn- ing Research, 23(120):1–39, 2022

    2022

  8. [16]

    Scaling diffusion transformers to 16 billion parameters.CoRR, abs/2407.11633, 2024

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Scaling diffusion transformers to 16 billion parameters.CoRR, abs/2407.11633, 2024. doi: 10.48550/arXiv.2407.11633

    2024 doi arXiv

  9. [17]

    Switch diffu- sion transformer: Synergizing denoising tasks with sparse mixture-of-experts

    Byeongjun Park, Hyojun Go, Jin-Young Kim, Sangmin Woo, Seokil Ham, and Changick Kim. Switch diffu- sion transformer: Synergizing denoising tasks with sparse mixture-of-experts. InEuropean Conference on Com- puter Vision, pages 461–477. Springer, 2024

    2024

  10. [18]

    EC-DIT: Scaling diffusion transformers with adaptive expert-choice routing

    Haotian Sun, Tao Lei, Bowen Zhang, Yanghao Li, Haoshuo Huang, Ruoming Pang, Bo Dai, and Nan Du. EC-DIT: Scaling diffusion transformers with adaptive expert-choice routing. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [19]

    Qwen2-VL: Enhanc- ing vision-language model’s perception of the world at any resolution.CoRR, abs/2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, 9 Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhanc- ing vision-language mode...

    2024 doi arXiv

  12. [20]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 4195– 4205, 2023

    2023

  13. [21]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceed- ings of t...

    2021

  14. [22]

    CosyV oice 2: Scalable streaming speech synthesis with large language models.CoRR, abs/2412.10117, 2024

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. CosyV oice 2: Scalable streaming speech synthesis...

    2024 doi arXiv

  15. [23]

    Step-Audio: Unified understanding and generation in intelligent speech interaction.CoRR, abs/2502.11946, 2025

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Min- grui Chen, et al. Step-Audio: Unified understanding and generation in intelligent speech interaction.CoRR, abs/2502.11946, 2025. doi: 10.48550/arXiv.2502.11946

    2025 doi arXiv

  16. [24]

    Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023

    2023

  17. [25]

    SegTune: Structured and fine-grained con- trol for song generation

    Yuejiao Wang, Zihao Ji, Pengfei Cai, Xu Li, Haorui Zheng, Zewen Song, Zhongliang Liu, Chen Zhang, and Pengfei Wan. SegTune: Structured and fine-grained con- trol for song generation. InProceedings of the 64th An- nual Meeting of the Association for Computational Lin- guistics ...

    2026 doi

  18. [26]

    Qwen2.5-Omni technical report.CoRR, abs/2503.20215,

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni technical report.CoRR, abs/2503.20215,

    arXiv

  19. [27]

    Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons

    Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1–5, 2025

    2025

  20. [28]

    Audiox: A unified framework for anything-to-audio generation

    Zeyue Tian, Zhaoyang Liu, Yizhu Jin, Ruibin Yuan, Li- umeng Xue, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. Audiox: A unified framework for anything-to-audio generation. InThe F ourteenth International Conference on Learning Representations, pages 1–27, 2026

    2026

  21. [29]

    Higgs Audio V2: Redefining expressiveness in audio generation

    Boson AI. Higgs Audio V2: Redefining expressiveness in audio generation. GitHub repository, 2025. Boson AI Higgs Audio project

    2025

  22. [30]

    UniFlow- Audio: Unified flow matching for audio generation from omni-modalities.CoRR, abs/2509.24391, 2025

    Xuenan Xu, Jiahao Mei, Zihao Zheng, Ye Tao, Zeyu Xie, Yaoyun Zhang, Haohe Liu, Yuning Wu, Ming Yan, Wen Wu, Chao Zhang, and Mengyue Wu. UniFlow- Audio: Unified flow matching for audio generation from omni-modalities.CoRR, abs/2509.24391, 2025. doi: 10.48550/arXiv.2509.24391

    2025 doi

  23. [31]

    Supervised learning of universal sentence representations from natural language inference data

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–6...

    2017 doi

  24. [32]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  25. [33]

    Eliminating oversaturation and artifacts of high guidance scales in diffusion models

    Seyedmorteza Sadat, Otmar Hilliges, and Romann M Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  26. [34]

    Seed-TTS: A family of high-quality versatile speech generation models.CoRR, abs/2406.02430, 2024

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xi- aoyang Li, Xingxing ...

    2024 doi arXiv

  27. [35]

    Librispeech-pc: Benchmark for evaluation of punc- tuation and capitalization capabilities of end-to-end asr models

    Aleksandr Meister, Matvei Novikov, Nikolay Karpov, Evelina Bakhturina, Vitaly Lavrukhin, and Boris Gins- burg. Librispeech-pc: Benchmark for evaluation of punc- tuation and capitalization capabilities of end-to-end asr models. In2023 IEEE Automatic Speech Recognition and Under...

    2023

  28. [36]

    AudioCaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. InProceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume...

    2019

  29. [37]

    Denk, Zalán Borsos, Jesse H

    Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse H. Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank. MusicLM: Generating music from text.CoRR, abs/2301.11325,...

    2023 doi arXiv

  30. [38]

    The song describer dataset: A corpus of audio captions for music-and- language evaluation.CoRR, abs/2311.10057, 2023

    Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, Elio Quinton, György Fazekas, and Juhan Nam. The song describer dataset: A corpus of audio captions for music-and- language evaluation.CoR...

    2023 doi arXiv

  31. [39]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  32. [40]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 ofProceedings of Machine Learning Res...

    2023

  33. [41]

    rain” may be an event in sfx, “an urban street in a downpour

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. In Interspeech 2022, pages 2063–2067, 2022. doi: 10.21437/ Interspeech.2022-9996. 11 A Architecture and Objective...

    2022

  34. [44]

    rule-based templates

    Dialogue scenes support at most two speakers. relationshort phrase Speaker roles, turn-taking relation, or foreground interaction, such as a host and guest or two commuters. vocalshort phrase Speaker identity descriptors, vocal timbre, delivery, emotion, ac- cent, or recording...

  35. [45]

    Do not include markdown, explana- tions, or extra text

    Output valid JSON only. Do not include markdown, explana- tions, or extra text

  36. [46]

    Do not add new keys

    Use only the predefined field names in the schema. Do not add new keys

  37. [47]

    - Do not paraphrase, expand, or rewrite it

    summary: - Copy the source-provided core summary verbatim. - Do not paraphrase, expand, or rewrite it. - If the input contains separate summary and detailed-description fields, use only the designated summary field. - If no explicit core summary is provided, copy the original ...

  38. [48]

    - Otherwise leave it empty

    speech: - If spoken words are explicitly quoted, copy them exactly. - Otherwise leave it empty. - Do not create speech content for clips without human speech

  39. [49]

    Other- wise leave it empty

    lyrics: fill only when song lyrics are explicitly provided. Other- wise leave it empty

  40. [50]

    Select exactly one type: speech: dominant human speech; 13 singing: dominant singing or vocal performance; music: non-vocal instrumental music; sfx: discrete sound events such as impacts, machines, animals, or weather; mixed: two or more foreground acoustic categories among sp...

  41. [51]

    - Otherwise leave it empty

    language: - Use an ISO language code only when human speech or singing is present. - Otherwise leave it empty

  42. [52]

    sfx should describe specific discrete events rather than generic labels

  43. [53]

    ambience describes persistent environmental background

  44. [54]

    texture describes recording or acoustic characteristics only

  45. [55]

    vocal describe voice characteristics only

  46. [56]

    1", "2",

    speaker_count must be one of: "1", "2", "unknown", or ""

  47. [57]

    music should describe only explicitly supported musical in- formation, including instruments, rhythm, style, or mood when available

  48. [58]

    summary":

    Do not infer missing information from world knowledge or context. All unsupported fields must remain empty. All field val- ues should be concise noun phrases rather than complete sen- tences. JSON Schema (output exactly this structure): { "summary": "", "type": "", "lang": "",...

  49. [2023]

    doi: 10.48550/arXiv.2312.15821

    doi arXiv

  50. [2025]

    doi: 10.48550/arXiv.2503.20215

    doi arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.