Pith. sign in

REVIEW 2 major objections 5 minor 297 references

A single MoE decoder can add full audio understanding and generation without erasing its text backbone's reasoning, alignment, or agentic skill.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 07:45 UTC pith:BUVY2XMU

load-bearing objection Solid open systems paper: simple unified decoder + multi-stage SFT + text-only Cascade RL keeps a strong MoE backbone competitive on audio while mostly retaining text scores; residual IFBench/NIAH drops make the abstract’s “no regression” claim a bit too clean. the 2 major comments →

arxiv 2607.05196 v2 pith:BUVY2XMU submitted 2026-07-06 cs.CL cs.AIcs.LGcs.SDeess.AS

Unified Audio Intelligence Without Regressing on Text Intelligence

Zhifeng Kong , Sang-gil Lee , Jaehyeon Kim , Boxin Wang , Zihan Liu , Sungwon Kim , Yang Chen , Arushi Goel
show 12 more authors
Rajarshi Roy Wenliang Dai Zhuolin Yang Yangyi Chen Dongfu Jiang Sreyan Ghosh Tuomas Rintamaki Andrew Tao Jonathan Raiman Mohammad Shoeybi Bryan Catanzaro Wei Ping
This is my paper
classification cs.CL cs.AIcs.LGcs.SDeess.AS
keywords audio-text LLMunified multimodal generationspeech recognitiontext-to-speechtext-to-audiomixture-of-expertscascade RLno text regression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a strong text-only mixture-of-experts LLM can be turned into a unified audio-text model that both understands and generates speech and general sound, while keeping almost all of the original text intelligence. The authors build Audex by encoding audio into the same embedding space as text and treating quantized speech and audio tokens the same way as text tokens inside one Transformer decoder. They train on a large carefully mixed audio-text corpus with a multi-stage supervised curriculum, then restore and strengthen text performance with text-only cascade reinforcement learning. The resulting model matches or leads open models on audio understanding, recognition, translation, text-to-speech, general audio generation, and speech-to-speech interaction, and shows only marginal or no regression on reasoning, knowledge, alignment, long-context, and agentic benchmarks relative to its text backbone. The practical point is that open multimodal audio generation need not force a trade-off against text capability if architecture and post-training are kept simple and staged.

Core claim

Audex shows that a single decoder architecture—audio encoder plus MLP projection into the text embedding space, with speech and non-speech codec tokens appended to the vocabulary and trained under ordinary next-token loss—can reach state-of-the-art open audio understanding and generation while preserving the reasoning, alignment, knowledge, long-context, and agentic performance of its Nemotron-Cascade-2 text backbone after multi-stage SFT plus text-only cascade RL.

What carries the argument

The unified single-Transformer-decoder design that projects continuous audio features into the text embedding space and treats text tokens and quantized speech/audio codec tokens uniformly under one cross-entropy objective, combined with multi-stage supervised fine-tuning followed by text-only cascade RL.

Load-bearing premise

That text-only cascade reinforcement learning after multimodal supervised fine-tuning is enough to protect and restore text intelligence without meaningful forgetting of audio skills or collapse of long-context attention.

What would settle it

A controlled re-run of the same multi-stage SFT followed by the same text-only cascade RL that produces large, consistent drops on the backbone's own reasoning, long-context NIAH, or agentic suites, or that collapses audio metrics after RL, would falsify the no-regression claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Audex-30B-A3B (and a 2B variant), a unified audio-text MoE LLM built on Nemotron-Cascade-2-30B-A3B. Audio is encoded via AF-Whisper + MLP into the text embedding space; generation treats text tokens and discrete speech/audio codec tokens (X-Codec2 / X-Codec RVQ) uniformly in a single decoder. Training uses curated audio-text data (157.4B audio + 320.5B text tokens), multi-stage SFT (text SFT reuse, audio warmup with frozen text embeddings, audio generation, then joint understanding/generation), followed by text-only Cascade RL + MOPD. The central claim is SOTA or near-SOTA open performance on audio understanding, ASR/AST, TTS, general audio generation, and speech-to-speech, while preserving the backbone’s reasoning, knowledge, alignment, long-context, and agentic scores with only marginal or no regression (Table 1, §2, §5).

Significance. If the empirical results hold under independent re-evaluation of the released checkpoints, this is a strong systems contribution: a simple, infrastructure-compatible architecture that largely closes the usual text-regression gap seen in multimodal-output models (e.g., Qwen3-Omni vs Qwen3). Credit is due for the multi-stage vs single-stage ablation (Table 4), intermediate-stage tracking across RL (Appendix B Table 15), CFG and sampling studies (Fig. 4, Appendix C.3), audio-warmup embedding freezes (Appendix C.1), and public checkpoint release. The work advances open unified audio LLMs without requiring audio pretraining from scratch and supplies a practical recipe for retaining long-context and reasoning after multimodal SFT.

major comments (2)
  1. Abstract, §2 and Table 1: the repeated claim of “marginal or no regression” on text intelligence is overstated relative to the reported numbers. IFBench drops 82.9 → 77.8, NIAH@1M drops 99.0 → 83.4, ArenaHard v2 83.5 → 81.6, and several agentic scores also decline modestly. These are not catastrophic, and Audex still far outperforms Qwen3-Omni’s regressions, but the absolute language should be tempered to “small regressions on a subset of alignment/long-context metrics, with parity or gains on most reasoning and knowledge suites.” Otherwise the central “without regressing” framing oversells the evidence already present in the tables.
  2. §5.4 and Table 13: speech-to-speech is evaluated via an explicit cascade (Audex ASR → Audex text reasoning → Audex TTS) rather than single-pass native speech-to-speech generation. The paper correctly notes the cascade, yet the Abstract and §2 list “speech-to-speech generation” among the model’s native capabilities alongside true generative tasks (TTS, TTA). Clarify the distinction and either report true end-to-end S2S metrics or qualify the claim so readers do not equate the cascade score with native duplex or single-pass S2S models.
minor comments (5)
  1. Table 1 and §5.2: TTA is restricted to fixed 10 s clips; the limitation is acknowledged but should be stated more prominently in the main results summary, since duration/consistency is a known gap versus diffusion baselines.
  2. §3.3 / vocabulary construction: the padding of the embedding table to 205 312 for TP divisibility is fine, but a short note on whether the unused RVQ layers (5–8) ever receive gradient or remain dead would help reproducibility.
  3. Figure 1 caption and §3: the dual speech/audio decoder paths are clear, but the streaming ConvNeXt decoder for X-Codec2 is only briefly mentioned; a one-sentence latency/quality trade-off would strengthen the systems contribution.
  4. Appendix B Table 15: transient ASR/MMSU dips under MOPD that later recover are useful; consider adding a one-line discussion in §4.4.3 so readers do not miss the recovery pattern.
  5. Typos / polish: “Inthiswork,webuildandrelease” (p. 2), occasional missing spaces after periods, and inconsistent capitalization of “Audex SFT” vs “Audex-SFT”.

Circularity Check

0 steps flagged

No circular derivation: empirical systems paper whose audio and text claims are measured on external public benchmarks, not forced by construction or load-bearing self-citation.

full rationale

Audex is an engineering/systems paper: a unified audio-text MoE LLM trained with multi-stage SFT plus text-only Cascade RL, then evaluated on independent public suites (AIME/HMMT/LiveCodeBench, MMLU/GPQA, ArenaHard/IFBench, NIAH, OpenASR/LibriSpeech/Fleurs, MMAU/MMAR, Seed-TTS-Eval, AudioCaps/SongDescriber FD_openl3, BigBenchAudio/VoiceBench). None of the reported scores reduce by definition to fitted constants or to a self-defined quantity. Architecture (AF-Whisper encoder + MLP projection + extended vocab for X-Codec/X-Codec2 tokens), CFG (10% unconditional padding), and multi-stage vs single-stage SFT are design choices validated by ablations (Tables 3–4, Appendix C), not circular predictions. Reuse of Nemotron-Cascade-2 SFT/RL recipes and the Cascade-RL anti-forgetting citation (Wang et al. 2025a, §4.4.3) is prior method reuse; the paper still reports intermediate-stage audio stability (Appendix B Table 15) and head-to-head text comparison to the backbone (Tables 1, 5). Residual text drops (e.g., IFBench, NIAH@1M) are disclosed, not hidden by definitional identity. No self-definitional loop, fitted-input-as-prediction, uniqueness theorem, or renamed known law is present.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

Load-bearing content is empirical: public benchmarks, a curated multimodal mix, codec/token design choices, and a multi-stage post-training curriculum. Free parameters are training and inference knobs that materially affect the reported tradeoff. Axioms are standard ML evaluation and transfer assumptions, not formal theorems. No new physical entities are postulated.

free parameters (5)
  • Stage-wise text data blend weights
    Text share is set by hand (e.g., 44–75% across stages; 0.69 vs 0.56 and 0.75 vs 0.88 ablations) and is required to avoid text collapse during audio SFT (§4.4; Figs. 6–7).
  • CFG scale λ for generation
    Inference quality depends on chosen λ≈3 for TTA and λ≈1.5 for TTS, with Top-k=80 and temperatures 1.0/0.1 (Fig. 4; Appendix C.3).
  • Audio codec configuration
    Speech uses X-Codec2 FSQ 50Hz/65k; non-speech uses X-Codec first 4 RVQ layers flattened to 200 tok/s; vocabulary padding to 205312 is an engineering choice that defines the generation interface (§3.3).
  • Learning rates and trainable module schedule
    Warmup LR 2e-3 on adapters/audio embeddings, later 2e-5 full LLM, frozen text embeddings in warmup, frozen audio encoder throughout—choices that control the text/audio tradeoff (Table 3; Appendix C.1).
  • Fixed 10-second TTA duration
    Text-to-audio is trained and evaluated at fixed 10s because variable length was unstable; this bounds the generation claim (§4.1; §5.2).
axioms (4)
  • domain assumption Public text and audio benchmarks (AIME, LiveCodeBench, MMAU, OpenASR, Seed-TTS-Eval, FDopenl3, etc.) are adequate proxies for the claimed capabilities.
    All main claims are benchmark-score claims (Table 1; §5; Appendix A).
  • domain assumption Text-only Cascade RL is resistant to catastrophic forgetting of multimodal skills learned in SFT.
    Authors explicitly rely on this from Nemotron-Cascade and report stable audio metrics across RL stages (§4.4.3; Appendix B).
  • ad hoc to paper Freezing text token embeddings during audio warmup is necessary to avoid text degradation.
    Supported by their ablation (Appendix C.1) and used as the production recipe, but it is an empirical design rule of this paper.
  • ad hoc to paper Multi-stage SFT is preferable to consolidated single-stage SFT for preserving long-context attention.
    Single-stage collapses NIAH to near zero while multi-stage keeps it high (Table 4; §4.4.2).
invented entities (1)
  • Audex multi-stage audio post-training curriculum independent evidence
    purpose: Add audio understanding and generation to a text MoE backbone while limiting text regression.
    The staged sequence (text SFT → audio warmup with frozen text embeddings → audio gen SFT → joint und+gen → text Cascade RL/MOPD) is the paper’s operational invention; independent evidence is the released model’s measured scores, not an external physical prediction.

pith-pipeline@v1.1.0-grok45 · 48995 in / 3467 out tokens · 36973 ms · 2026-07-11T07:45:57.242420+00:00 · methodology

0 comments
read the original abstract

Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.

Figures

Figures reproduced from arXiv: 2607.05196 by Andrew Tao, Arushi Goel, Boxin Wang, Bryan Catanzaro, Dongfu Jiang, Jaehyeon Kim, Jonathan Raiman, Mohammad Shoeybi, Rajarshi Roy, Sang-gil Lee, Sreyan Ghosh, Sungwon Kim, Tuomas Rintamaki, Wei Ping, Wenliang Dai, Yang Chen, Yangyi Chen, Zhifeng Kong, Zhuolin Yang, Zihan Liu.

Figure 1
Figure 1. Figure 1: Audex-30B-A3B architecture. The LLM backbone is Nemotron-Cascade-2-30B-A3B. Audio inputs are [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Templates for (a) text-only chats (see tool-calling templates in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Audex training stages. In the SFT stage, we study two training curriculums: (1) the multi-stage [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The Effect of CFG Value 𝜆 on TTA and TTS. CFG is required to achieve good TTA quality, and the optimal range is 𝜆 ∈ [3, 5]. CFG is not necessary for good TTS quality, although using 𝜆 ≈ 1.5 may lead to slightly better WER than not using CFG while doubling the inference budget. is possibly because most of the training data are fixed to be 10 seconds. We aim to solve the duration and consistency issues with … view at source ↗
Figure 5
Figure 5. Figure 5: Audio Gen. SFT Results between Freezing and Unfreezing Text Embeddings in Audio Warmup. [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of Text Data Blending Ratios (0.56 vs 0.69) in the Audio Gen. + Audio Und. SFT Stage. [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of Text Data Blending Ratios (0.75 vs 0.88) in the Single-Stage Consolidated SFT Stage. [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparisons across different inference hyperparameters for text-to-audio generation. [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Comparisons across different inference hyperparameters for text-to-speech generation. [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

297 extracted references · 128 linked inside Pith

  1. [1]

    Yang, Zhuolin and Liu, Zihan and Chen, Yang and Dai, Wenliang and Wang, Boxin and Lin, Sheng-Chieh and Lee, Chankyu and Chen, Yangyi and Jiang, Dongfu and He, Jiafan and Pi, Renjie and Lam, Grace and Lee, Nayeon and Bukharin, Alexander and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei , journal=

  2. [2]

    Wang, Boxin and Lee, Chankyu and Lee, Nayeon and Lin, Sheng-Chieh and Dai, Wenliang and Chen, Yang and Chen, Yangyi and Yang, Zhuolin and Liu, Zihan and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei , journal=

  3. [3]

    arXiv preprint arXiv:2512.20848 , year=

    Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning , author=. arXiv preprint arXiv:2512.20848 , year=

    arXiv

  4. [4]

    NVIDIA , howpublished =

  5. [5]

    arXiv preprint arXiv:2504.18425 , year=

    Kimi-audio technical report , author=. arXiv preprint arXiv:2504.18425 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2402.01831 , year=

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities , author=. arXiv preprint arXiv:2402.01831 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2503.03983 , year=

    Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities , author=. arXiv preprint arXiv:2503.03983 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2507.08128 , year=

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models , author=. arXiv preprint arXiv:2507.08128 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2507.16632 , year=

    Step-audio 2 technical report , author=. arXiv preprint arXiv:2507.16632 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2511.15848 , year=

    Step-Audio-R1 Technical Report , author=. arXiv preprint arXiv:2511.15848 , year=

    arXiv

  11. [11]

    arXiv preprint arXiv:2604.15804 , year=

    Qwen3.5-omni technical report , author=. arXiv preprint arXiv:2604.15804 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2604.10905 , year=

    Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music , author=. arXiv preprint arXiv:2604.10905 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2407.10759 , year=

    Qwen2-audio technical report , author=. arXiv preprint arXiv:2407.10759 , year=

    Pith/arXiv arXiv

  14. [14]

    Tian, Jinchuan and Lee, Sang-gil and Kong, Zhifeng and Ghosh, Sreyan and Goel, Arushi and Yang, Chao-Han Huck and Dai, Wenliang and Liu, Zihan and Ye, Hanrong and Watanabe, Shinji annd Shoeybi, Mohammad and Catanzaro, Bryan and Valle, Rafael and Ping, Wei , journal=

  15. [15]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  17. [17]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=

    2026

  18. [18]

    Team, Kimi and Bai, Tongtong and Bai, Yifan and Bao, Yiping and Cai, SH and Cao, Yuan and Charles, Y and Che, HS and Chen, Cheng and Chen, Guanduo and others , journal=

  19. [19]

    Proceedings of the 26th annual international conference on machine learning , pages=

    Curriculum learning , author=. Proceedings of the 26th annual international conference on machine learning , pages=

  20. [20]

    Glot International , volume=

    Speak and unSpeak with PRAAT , author=. Glot International , volume=

  21. [21]

    arXiv preprint arXiv:2307.16372 , year=

    Lp-musiccaps: Llm-based pseudo music captioning , author=. arXiv preprint arXiv:2307.16372 , year=

    Pith/arXiv arXiv

  22. [22]

    International Conference on Machine Learning , pages=

    Neural audio synthesis of musical notes with wavenet autoencoders , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  23. [23]

    arXiv preprint arXiv:2404.07616 , year=

    Audio Dialogues: Dialogues dataset for audio and music understanding , author=. arXiv preprint arXiv:2404.07616 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2305.10790 , year=

    Listen, think, and understand , author=. arXiv preprint arXiv:2305.10790 , year=

    Pith/arXiv arXiv

  25. [25]

    2024 , howpublished=

    OpenAI , title=. 2024 , howpublished=

    2024

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    P-flow: a fast and data-efficient zero-shot TTS through speech prompting , author=. Advances in Neural Information Processing Systems , volume=

  27. [27]

    arXiv preprint arXiv:2406.15487 , year=

    Improving text-to-audio models with synthetic captions , author=. arXiv preprint arXiv:2406.15487 , year=

    Pith/arXiv arXiv

  28. [28]

    Advances in neural information processing systems , volume=

    Voicebox: Text-guided multilingual universal speech generation at scale , author=. Advances in neural information processing systems , volume=

  29. [29]

    arXiv preprint arXiv:2210.02747 , year=

    Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2310.16338 , year=

    Generative pre-training for speech with flow matching , author=. arXiv preprint arXiv:2310.16338 , year=

    Pith/arXiv arXiv

  31. [31]

    PloS one , volume=

    The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English , author=. PloS one , volume=. 2018 , publisher=

    2018

  32. [32]

    Simple-TTS: End-to-End Text-to-Speech Synthesis with Latent Diffusion , author=

  33. [33]

    Pedalboard Documentation

    Spotify. Pedalboard Documentation. Pedalboard 0.9.16 documentation

  34. [34]

    International Conference on Machine Learning , pages=

    Autovc: Zero-shot voice style transfer with only autoencoder loss , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  35. [35]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  36. [36]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  37. [37]

    2017 , publisher=

    The MUSDB18 corpus for music separation , author=. 2017 , publisher=

    2017

  38. [38]

    2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages=

    Natural tts synthesis by conditioning wavenet on mel spectrogram predictions , author=. 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages=. 2018 , organization=

    2018

  39. [39]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    Pith/arXiv arXiv

  40. [40]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Efficient diffusion training via min-snr weighting strategy , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  41. [41]

    arXiv preprint arXiv:2404.07724 , year=

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models , author=. arXiv preprint arXiv:2404.07724 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2302.00482 , year=

    Conditional flow matching: Simulation-free dynamic optimal transport , author=. arXiv preprint arXiv:2302.00482 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2005.05957 , year=

    Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis , author=. arXiv preprint arXiv:2005.05957 , year=

    Pith/arXiv arXiv 2005

  44. [44]

    Advances in Neural Information Processing Systems , year=

    Attention is all you need , author=. Advances in Neural Information Processing Systems , year=

  45. [45]

    arXiv preprint arXiv:2312.15821 , year=

    Audiobox: Unified audio generation with natural language prompts , author=. arXiv preprint arXiv:2312.15821 , year=

    Pith/arXiv arXiv

  46. [46]

    Transactions of the Association for Computational Linguistics , volume=

    Byt5: Towards a token-free future with pre-trained byte-to-byte models , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  47. [47]

    arXiv preprint arXiv:2410.19168 , year=

    Mmau: A massive multi-task audio understanding and reasoning benchmark , author=. arXiv preprint arXiv:2410.19168 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2505.13032 , year=

    MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix , author=. arXiv preprint arXiv:2505.13032 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2310.00704 , year=

    Uniaudio: An audio foundation model toward universal audio generation , author=. arXiv preprint arXiv:2310.00704 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2301.02111 , year=

    Neural codec language models are zero-shot text to speech synthesizers , author=. arXiv preprint arXiv:2301.02111 , year=

    Pith/arXiv arXiv

  51. [51]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

    Speechx: Neural codec language model as a versatile speech transformer , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

  52. [52]

    Advances in neural information processing systems , volume=

    Understanding and improving layer normalization , author=. Advances in neural information processing systems , volume=

  53. [53]

    The Eleventh International Conference on Learning Representations , year=

    BigVGAN: A Universal Neural Vocoder with Large-Scale Training , author=. The Eleventh International Conference on Learning Representations , year=

  54. [54]

    Journal of Machine Learning Research , volume=

    Scaling instruction-finetuned language models , author=. Journal of Machine Learning Research , volume=

  55. [55]

    International Conference on Machine Learning , pages=

    AudioLDM: Text-to-Audio Generation with Latent Diffusion Models , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  56. [56]

    Proceedings of the 31st ACM International Conference on Multimedia , pages=

    Text-to-audio generation using instruction guided latent diffusion model , author=. Proceedings of the 31st ACM International Conference on Multimedia , pages=

  57. [57]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

    Audioldm 2: Learning holistic audio generation with self-supervised pretraining , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

  58. [58]

    Computer Science

    Improving image generation with better captions , author=. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , volume=

  59. [59]

    Advances in Neural Information Processing Systems , volume=

    Improving multimodal datasets with image captioning , author=. Advances in Neural Information Processing Systems , volume=

  60. [60]

    2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages=

    Audio set: An ontology and human-labeled dataset for audio events , author=. 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages=. 2017 , organization=

    2017

  61. [61]

    Audiocaps: Generating captions for audios in the wild , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

    2019

  62. [63]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

  63. [64]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023

  64. [65]

    International Conference on Machine Learning , pages=

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  65. [66]

    arXiv preprint arXiv:2305.18474 , year=

    Make-an-audio 2: Temporal-enhanced text-to-audio generation , author=. arXiv preprint arXiv:2305.18474 , year=

    Pith/arXiv arXiv

  66. [67]

    arXiv preprint arXiv:2407.04416 , year=

    Improving Audio Generation with Visual Enhanced Caption , author=. arXiv preprint arXiv:2407.04416 , year=

    Pith/arXiv arXiv

  67. [68]

    arXiv preprint arXiv:2406.19388 , year=

    Taming Data and Transformers for Audio Generation , author=. arXiv preprint arXiv:2406.19388 , year=

    Pith/arXiv arXiv

  68. [69]

    arXiv preprint arXiv:2407.14358 , year=

    Stable Audio Open , author=. arXiv preprint arXiv:2407.14358 , year=

    Pith/arXiv arXiv

  69. [70]

    arXiv preprint arXiv:2202.00512 , year=

    Progressive distillation for fast sampling of diffusion models , author=. arXiv preprint arXiv:2202.00512 , year=

    Pith/arXiv arXiv

  70. [71]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Scalable diffusion models with transformers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  71. [72]

    arXiv preprint arXiv:1606.08415 , year=

    Gaussian error linear units (gelus) , author=. arXiv preprint arXiv:1606.08415 , year=

    Pith/arXiv arXiv

  72. [73]

    Forty-first International Conference on Machine Learning , year=

    Scaling rectified flow transformers for high-resolution image synthesis , author=. Forty-first International Conference on Machine Learning , year=

  73. [74]

    arXiv preprint arXiv:1710.03740 , year=

    Mixed precision training , author=. arXiv preprint arXiv:1710.03740 , year=

    Pith/arXiv arXiv

  74. [75]

    Advances in Neural Information Processing Systems , volume=

    Flashattention: Fast and memory-efficient exact attention with io-awareness , author=. Advances in Neural Information Processing Systems , volume=

  75. [76]

    arXiv preprint arXiv:2407.03648 , year=

    High Fidelity Text-Guided Music Generation and Editing via Single-Stage Flow Matching , author=. arXiv preprint arXiv:2407.03648 , year=

    Pith/arXiv arXiv

  76. [77]

    ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Vggsound: A large-scale audio-visual dataset , author=. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2020 , organization=

    2020

  77. [78]

    arXiv preprint arXiv:2406.02507 , year=

    Guiding a Diffusion Model with a Bad Version of Itself , author=. arXiv preprint arXiv:2406.02507 , year=

    Pith/arXiv arXiv

  78. [79]

    Advances in neural information processing systems , volume=

    Elucidating the design space of diffusion-based generative models , author=. Advances in neural information processing systems , volume=

  79. [80]

    arXiv preprint arXiv:2311.08355 , year=

    Mustango: Toward controllable text-to-music generation , author=. arXiv preprint arXiv:2311.08355 , year=

    Pith/arXiv arXiv

  80. [81]

    arXiv preprint arXiv:2311.03079 , year=

    Cogvlm: Visual expert for pretrained language models , author=. arXiv preprint arXiv:2311.03079 , year=

    Pith/arXiv arXiv

Showing first 80 references.