Pith. sign in

REVIEW 4 major objections 6 minor 13 cited by

TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TangoFlux shows a 515M-parameter flow-matching model can generate up to 30 seconds of 44.1kHz audio in 3.7 seconds on one A40 GPU and match or beat far larger diffusion text-to-audio models on objective benchmarks and human judgments of…

desk verdict A fast, open, small text-to-audio model worth knowing about; the iterative CLAP-ranked alignment is the interesting part, but the SOTA claim needs the missing FlashAudio comparison. read the letter →

arxiv 2412.21037 v2 pith:VEX4F6XO submitted 2024-12-30 cs.SD cs.AIcs.CLeess.AS

Chia-Yu Hung , Navonil Majumder , Zhifeng Kong , Ambuj Mehrish , Amir Ali Bagherzadeh , Chuan Li , Rafael Valle , Bryan Catanzaro
show 1 more author
Soujanya Poria
This is my paper · ORCID
classification cs.SDcs.AIcs.CLeess.AS
keywords text-to-audiogenerationflowmatchingrectifiedflowsdirectpreferenceoptimizationCLAPrewardmodelaudioalignmentdiffusiontransformersefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TangoFlux aims to show that a compact 515M-parameter model trained entirely on non-proprietary data can generate up to 30 seconds of 44.1kHz audio in 3.7 seconds on a single A40 GPU while matching or beating much larger diffusion text-to-audio models on fidelity and prompt alignment. The core idea is that text-to-audio alignment can be driven by self-generated preference data: generate several candidate audios per prompt, rank them with CLAP cosine similarity, treat the top and bottom as winner and loser, and optimize a DPO-style preference loss on rectified flows. On AudioCaps the paper reports state-of-the-art FD, CLAP, and Inception scores, near-matching KL divergence to Tango2, and the best z-scores, rankings, and Elo scores in a 50-prompt out-of-distribution human evaluation. A sympathetic reader would care because strong audio generation and alignment may be possible with a much cheaper, open-weight model rather than requiring large proprietary systems.

What carries the argument

The central machinery is CRPO, an iterative loop built on a hybrid MMDiT-DiT transformer trained as a rectified flow on Stable Audio Open's frozen VAE latent space. At iteration k, the current model samples multiple audios per prompt, CLAP ranks them by text-audio cosine similarity, and the argmax and argmin pair becomes the preference data for direct preference optimization adapted to flow matching. TangoFlux's objective adds the flow-matching loss on the winning audio (LCRPO = LDPO-FM + LFM) to prevent the winning loss from drifting, and rectified flows give straight noise-to-data paths so a 50-step Euler solver suffices, which is why inference runs in 3.7 seconds and degrades little at just 10 steps.

What would settle it

Take a set of prompts where human raters and CLAP disagree on which of two generated clips is more faithful; if the CRPO checkpoint improves CLAP-ranked choices but lowers human relevance or quality scores relative to TangoFlux-base on those prompts, the assumption that CLAP ranking carries the alignment would be refuted.

Watch

Extended reading notes

Core claim

The load-bearing discovery is that preference pairs for text-to-audio alignment need not come from human labels or static datasets: they can be generated online by the model itself and ranked by CLAP. Across five CRPO iterations, each building preference data from the current checkpoint, TangoFlux reaches 75.1 FD, 1.15 KL, 0.480 CLAP, and 12.2 IS; the same optimization on fixed preference data saturates after two iterations and degrades, while online generation keeps improving through iteration four. The paper also finds that the vanilla DPO-for-diffusion loss raises both winning and losing flow-matching losses while widening their margin, and that adding the winning sample's flow-matching loss stabilizes optimization. Human evaluation places TangoFlux first on both overall audio quality and relevance among four text-to-audio models, with its largest objective gains appearing on multi-event prompts.

Load-bearing premise

The entire alignment pipeline assumes that CLAP cosine similarity is a faithful proxy for human judgment of prompt-audio relevance, because each preference pair's winner and loser are chosen purely by CLAP ranking.

Editorial extensions

If this is right

  • A 515M-parameter open-weight model can replace larger diffusion systems on standard text-to-audio benchmarks at roughly a sixth of the inference cost.
  • Self-generated, CLAP-ranked preference data can substitute for human annotation in text-to-audio alignment, making iterative self-improvement feasible without expensive labels.
  • Online preference data generation is essential: reusing the same preference pairs saturates and degrades, so future alignment pipelines for generative audio should regenerate data each iteration.
  • Multi-event and temporally complex prompts benefit most from the alignment, suggesting faithfulness to dense captions is where the method's gains are concentrated.
  • Fast sampling at 10 steps with modest quality loss makes the model practical for interactive and near-real-time audio applications.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If CLAP's notion of similarity diverges from human fidelity on some audio dimensions, CRPO-trained checkpoints could overfit CLAP-like artifacts; interleaving small human-annotated sets into the ranking signal would test and correct for this.
  • The same online ranking-and-optimizing loop likely transfers to neighboring generation tasks such as text-to-music and sound-effect synthesis whenever a contrastive audio-text embedding model is available.
  • Because the paper reports the largest gains on multi-event prompts, a natural stress test is strict temporal ordering, where CLAP may not capture order errors; this could reveal where preference optimization still needs structural rewards.
  • Using a different, independently trained reward model instead of CLAP would isolate how much of the gain comes from the online loop and how much from CLAP's specific embedding space.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TangoFlux, a 515M-parameter text-to-audio model based on rectified flow with a hybrid MMDiT/DiT transformer backbone, trained on non-proprietary data (WavCaps and AudioCaps) and capable of generating up to 30 seconds of 44.1kHz audio in 3.7 seconds on an A40 GPU. The authors also propose CLAP-Ranked Preference Optimization (CRPO), an iterative alignment framework that generates audio samples, ranks them with a CLAP model, constructs winner-loser preference pairs, and optimizes the model with a DPO-style loss supplemented by a flow-matching loss on the winning samples. The paper reports state-of-the-art objective results on AudioCaps (FD 75.1, KL 1.15, CLAPscore 0.480, IS 12.2) and superior human-evaluation z-scores, rankings, and Elo scores on a 50-prompt out-of-distribution set, alongside ablations for online vs. offline data generation, static preference datasets, and the proposed loss regularizer.

Significance. If the claims hold, TangoFlux represents a meaningful advance in efficient, open-weight text-to-audio generation: it is substantially faster than diffusion baselines while improving objective and subjective alignment. CRPO is a practical, scalable alignment recipe that does not require human preference labels, and the paper includes useful ablations showing the importance of online data generation and the winning-loss regularizer. The use of distinct CLAP checkpoints for preference construction and evaluation partially mitigates reward-hacking concerns, and the human evaluation on complex multi-event prompts provides independent evidence for the alignment gains. The manuscript's strengths include open-sourced code and models, explicit use of non-proprietary training data, and a multi-event analysis that targets a known weakness of prior TTA models.

major comments (4)
  1. [§4.1, Table 1; §5] The central claim that TangoFlux achieves state-of-the-art performance is incomplete because FlashAudio (Liu et al., 2024c), a rectified-flow text-to-audio model cited in Related Work, is not included in Tables 1, 2, or the inference-time comparisons in §4.7. Since the headline is a relative performance claim, omitting the closest prior method in the same architecture family leaves the SOTA assertion unsupported. Please add FlashAudio to the comparison (using its published scores if the checkpoint is not available) or explicitly justify the exclusion and soften the SOTA claim accordingly.
  2. [§2.5.1, Table 4] The validation of CLAP as a reward model is partially circular. The paper shows that Best-of-N selection using a CLAP checkpoint improves CLAPscore, but the evaluation CLAPscore is computed with a different checkpoint of the same CLAP family, so part of the gain is inherited from the reward used for selection. Since CRPO's central premise is that CLAP rank ordering approximates human preference, please provide a direct validation, e.g., correlate CLAP scores with human OVL/REL ratings on the 50-prompt set or on a sample of ranked preference pairs, and report the rank correlation between the two CLAP checkpoints used for reward and evaluation.
  3. [§4.1, Tables 1, 2, and 5] All reported objective and subjective results are point estimates with no confidence intervals or significance tests. Several decisive margins are small (e.g., CLAPscore 0.480 vs. 0.447 for Tango 2; KL 1.15 vs. 1.11, where Tango 2 is better), and these could lie within run-to-run or evaluation noise. Please report standard errors or bootstrap confidence intervals for the objective metrics across evaluation resamples (or multiple checkpoints), and add inter-annotator agreement and significance tests for the human-evaluation z-scores, rankings, and Elo scores.
  4. [Appendix A.5] Appendix A.5, titled 'Multi-Staged Relation-Aware Evaluation', contains only two corpus tables (Tables 8 and 9) and no evaluation procedure, results, or analysis. As written, it does not substantiate the claimed relation-aware evaluation. Please complete the section with a description of how the corpora are used and the corresponding results, or remove the section and state explicitly that this evaluation is future work.
minor comments (6)
  1. [§1 (contribution iv); Abstract] Contribution (iv) says 'We plan to release the code and model weights' while the Abstract and the resource list state that all code and models are open-sourced; please reconcile these statements.
  2. [§3.3; Appendix A.1] The main results use CFG=4.5, while the appendix identifies CFG=3.5 as the optimal balance; please justify the choice of 4.5 for the headline table.
  3. [Appendix A.2] The text reports a 200-step FD of 73.9 and a 10-step FD of 77.2, but Table 1 reports FD=75.1 at 50 steps; please make the appendix numbers consistent with the main table.
  4. [Table 4] The N=1 row duplicates the Table 1 TangoFlux row; please clarify that N=1 corresponds to the default model without Best-of-N selection.
  5. [§2.2; §3.1; §3.3] Please specify whether the FLAN-T5 text encoder is frozen or fine-tuned, and provide the exact version of the stable-audio-metrics library and CLAP checkpoints used for reproducibility.
  6. [Figure 1 caption; Table 1; Abstract] The phrase 'approximately 2x faster' in Figure 1's caption conflicts with the 3.7s vs. 8.6s comparison (2.3x) in Table 1 and with 'about 6x faster' in the Abstract; please standardize the speed-up claims.

Circularity Check

1 steps flagged · score 3.0 of 10

Central flow-matching and human-eval claims are self-contained; only the CLAP-as-reward validation is partially circular.

  1. other [Section 2.5.1 and Section 4.3 (Table 4)]
    "CLAP reward score is calculated as the cosine similarity between textual and audio embeddings encoded by the model. Thus, we assume that CLAP can serve as a reasonable proxy reward model for evaluating audio outputs against the textual description. ... To validate CLAP as a proxy reward model for evaluating audio output, we further evaluate TANGO FLUX under a CLAP-driven Best-of-N policy ... increasing N yield better CLAPscore and KLpasst ... This indicates that the CLAP can identify well-aligned audio outputs that better represent the textual descriptions."

    The validation uses CLAP both as the selection mechanism (ranking with the 630k-audioset-best checkpoint) and as the reported outcome (CLAPscore from the sibling 630k-audioset-fusion-best checkpoint). Selecting the argmax of a score over N samples mechanically raises the expected maximum of any positively correlated score, so the CLAPscore improvement in Table 4 is an order-statistics consequence of the selection rule rather than independent evidence that CLAP rankings track human relevance. The paper mitigates this by using different CLAP checkpoints and by also reporting KL and FD, so the central SOTA claim remains independent; however, the specific claim that CLAP is validated as a reward model is supported only by evidence drawn from the same CLAP family.

full rationale

The central derivation of TANGOFLUX does not reduce to its inputs by construction. Rectified-flow training, the hybrid MMDiT/DiT backbone, and the CRPO objective are stated as concrete equations and procedures. The headline SOTA claim is supported by FD_openl3, KL_passt, IS, and human OVL/REL, all of which are independent of the CLAP reward used to construct preference pairs, and the paper explicitly uses different CLAP checkpoints for preference-pair creation (630k-audioset-best) and final CLAPscore evaluation (630k-audioset-fusion-best). FD, KL, IS, and human evaluation all improve, so the main result is not forced by the reward model. The one partially circular element is the validation of CLAP as a reward model in Section 4.3: samples are selected by CLAP and then scored with a sibling CLAP checkpoint, so the monotone CLAPscore improvement with N is largely an order-statistics artifact. This affects the supporting claim that CLAP is a validated proxy reward, but not the central SOTA claim. Separately, the Comparison omits FlashAudio, the closest flow-matching baseline; that is a completeness/correctness concern, not circularity.

Assumptions & free parameters 8 free parameters · 7 assumptions · 0 invented entities

TangoFlux introduces no new physical or representational entities. The numeric claims are conditioned on a cluster of tunable hyperparameters (only some of which are reported, notably beta is missing) and on the assumption that CLAP similarity is a faithful reward, which also enters the evaluation. These do not falsify the results but they set the boundary of what is established.

free parameters (8)
  • DPO reward scale beta = not reported
    Scales the preference loss in Eq. (4); the behavior of LCRPO depends on it, but the value is absent from the paper.
  • classifier-free guidance scale = 4.5
    Selected from {3.0, 3.5, 4.0, 4.5, 5.0} in Table 6; used for all main results.
  • inference steps = 50
    Euler steps for main results; appendix shows 10 steps still strong, so 50 is a convenience choice.
  • samples per prompt for preference data = 5
    N=5 generated audios per prompt; winner/loser are max/min CLAP; no ablation of this N is reported.
  • CRPO iterations = 5
    Stopped at 5 due to observed performance saturation; no formal stopping criterion is given.
  • prompts sampled each iteration = 20000
    Random subset of 45k AudioCaps prompts; hand-chosen budget.
  • timestep sampling logit-normal parameters = mean=0, variance=1
    Adopted from FLUX (Esser et al.), not tuned in this paper.
  • alignment learning rate = 1e-5
    Training hyperparameter for CRPO fine-tuning; not ablated.
assumptions (7)
  • standard math The rectified-flow interpolation x_t=(1-t)x_1+t*x_0 with velocity v=x_0-x_1 defines a valid transport model.
    Standard flow matching; cited to Lipman et al. (2023) and Liu et al. (2022).
  • standard math DPO diffusion loss is equivalent for noise prediction and flow velocity prediction.
    Paper relies on equivalence from Lipman et al. (2023) and Esser et al. (2024); no proof is repeated.
  • domain assumption CLAP cosine similarity is a valid proxy for text-audio alignment.
    Used to rank audios and to construct all preference pairs; also used with a different checkpoint as a headline metric.
  • domain assumption Highest- and lowest-CLAP audios in a batch form informative preference pairs.
    Assumes extremes are meaningful winner/loser labels; no validation against human labels is provided.
  • domain assumption Iterative online self-generation improves alignment without catastrophic collapse.
    Empirically shown for 5 iterations; validity beyond that is unknown.
  • domain assumption Frozen Stable Audio Open VAE provides a faithful audio latent space.
    Model inherits VAE quality; no fine-tuning or error analysis of VAE reconstruction on the benchmark set.
  • domain assumption Silence-padding and center-cropping to 30s does not distort training signal.
    All training audios are converted to pseudostereo 30s clips; short audios padded, long ones cropped.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization." pith.science (2026). https://pith.science/paper/VEX4F6XO

@misc{pith2026241221037,
  author       = {Pith},
  title        = {Pith review of: TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VEX4F6XO}},
  note         = {Machine review of arXiv:2412.21037}
}
read the original abstract

We introduce TangoFlux, an efficient Text-to-Audio (TTA) generative model with 515M parameters, capable of generating up to 30 seconds of 44.1kHz audio in just 3.7 seconds on a single A40 GPU. A key challenge in aligning TTA models lies in the difficulty of creating preference pairs, as TTA lacks structured mechanisms like verifiable rewards or gold-standard answers available for Large Language Models (LLMs). To address this, we propose CLAP-Ranked Preference Optimization (CRPO), a novel framework that iteratively generates and optimizes preference data to enhance TTA alignment. We demonstrate that the audio preference dataset generated using CRPO outperforms existing alternatives. With this framework, TangoFlux achieves state-of-the-art performance across both objective and subjective benchmarks. We open source all code and models to support further research in TTA generation.

Figures

Figures reproduced from arXiv: 2412.21037 by the authors.

Figure 1
Figure 1. A depiction of the overall training pipeline of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The trajectory of CLAP score and KL divergence across the training iterations. This plot [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Winning and Losing losses of LDPO-FM and LCRPO at each iteration. Winning and Losing losses increase each iteration, as well as their margin. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison between LDPO-FM and LCRPO w.r.t. (a) CLAPscore, (b) FDopenl3, and (c) KLpasst across iterations. As shown in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Comparison of (a) CLAP and (b) FD Scores vs Inference Time for each model. Results [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: The Gradio-based human evaluation form created for the annotators to score the model [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching

    cs.SD 2026-08 conditional novelty 6.0 of 10

    VoxAudio generates audio scenes with intelligible, temporally placed quoted speech by combining chunk-wise causal flow matching with multi-reward fine-tuning and a large transcript-annotated corpus.

  2. MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

    eess.AS 2026-08 conditional novelty 6.0 of 10

    MiDashengLM-Gen uses an LLM backbone with per-token flow matching to generate variable-length multilingual audio scenes with near-TTS speech intelligibility and competitive mixed-scene quality.

  3. RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

    cs.SD 2026-08 conditional novelty 6.0 of 10

    RAG-Audio starts frozen audio generators from a retrieved exemplar of the fMRI-decoded CLAP embedding, raising 10-way stimulus identification from 0.14-0.18 to 0.40-0.43 on Brain2Music and cutting FAD by about 10x.

  4. AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation

    cs.SD 2026-08 conditional novelty 6.0 of 10

    A structured soundscape benchmark with 25,707 binary semantic rubrics shows that rubric-based, audio-grounded evaluation tracks human semantic judgments better than CLAP-style global similarity for text-to-audio models.

  5. Unified Audio Intelligence Without Regressing on Text Intelligence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified 30B MoE audio-text LLM achieves state-of-the-art audio understanding, generation, and speech tasks while preserving text reasoning comparable to its text-only backbone.

  6. SynSFX: Multi-Model Sound Effects Synthesis Dataset for Deepfake Detection and Evaluation

    cs.SD 2026-07 conditional novelty 6.0 of 10

    SynSFX provides a multi-generator sound-effect deepfake corpus showing speech detectors fail, joint training mitigates forgetting, but generalization to unseen generators remains poor due to artifact overfitting.

  7. RFM-Editing 2: Text-Guided Audio Editing with Rectified Flow Matching and Coarse-to-Fine Diffusion Transformers

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    Hybrid two-stage diffusion transformer architecture for instruction-guided audio editing via rectified flow that performs joint attention at low resolution then alternates joint and cross-attention at high resolution ...

  8. SemanticAudio: Audio Generation and Editing in Semantic Space

    eess.AS 2026-01 conditional novelty 6.0 of 10

    SemanticAudio improves text-to-audio alignment by generating a compact semantic plan first with a Flow Matching planner and then rendering acoustic latents from that plan, and it performs training-free audio editing b...

  9. JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment

    cs.SD 2025-07 conditional novelty 6.0 of 10

    JAM is a 530M-parameter flow-matching song generator that adds word- and phoneme-level timing control and duration control, achieving strong lyric fidelity and musicality scores when ground-truth timings are provided.

  10. AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AnimeShooter provides hierarchical story and shot annotations plus reference images for 148K one-minute animation stories, and AnimeShooterGen trained on it shows improved cross-shot consistency.

  11. FdAudio: MeanFlow-Anchored Fr\'echet-Distance Post-Training for One-Step Text-to-Audio Generation

    eess.AS 2026-07 conditional novelty 5.5 of 10

    MeanFlow-anchored multi-representation FD post-training improves one-step text-to-audio quality without collapsing multi-step sampling.

  12. SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations

    cs.SD 2025-07 conditional novelty 5.0 of 10

    A three-stage diffusion pipeline maps 3D Gaussian Splatting object representations to position-dependent impact sounds, trained first on text captions and then on real recordings.

  13. MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder

    eess.AS 2025-05 conditional novelty 5.0 of 10

    MiniMax-Speech reports state-of-the-art zero-shot voice cloning quality using a learnable speaker encoder and Flow-VAE, without requiring reference transcripts.

Reference graph

Works this paper leans on

64 extracted references · 3 canonical work pages · cited by 13 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Albergo and Eric Vanden-Eijnden

    Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants, 2023. URL https://arxiv.org/abs/2209.15571

    arXiv 2023

  3. [3]

    Self-play fine-tuning converts weak language models to strong language models, 2024

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL https://arxiv.org/abs/2401.01335

    arXiv 2024

  4. [4]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023. URL https://arxiv.org/abs/2311.07919

    arXiv 2023

  5. [5]

    Qwen2-audio technical report, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024. URL https://arxiv.org/abs/2407.10759

    arXiv 2024

  6. [6]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    arXiv 2022

  7. [7]

    Simple and controllable music generation, 2024

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation, 2024. URL https://arxiv.org/abs/2306.05284

    arXiv 2024

  8. [8]

    Look, listen, and learn more: Design choices for deep audio embeddings

    Aurora Linh Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 3852--3856, 2019. doi:10.1109/ICASSP.2019.8682475

    arXiv 2019

Show all 64 references
  1. [9]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024 arXiv

  2. [10]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion, 2024 a . URL https://arxiv.org/abs/2402.04825

    2024 arXiv

  3. [11]

    Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons

    Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion, 2024 b . URL https://arxiv.org/abs/2404.10301

    2024 arXiv

  4. [12]

    Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons

    Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open, 2024 c . URL https://arxiv.org/abs/2407.14358

    2024 arXiv

  5. [13]

    Scaling laws for reward model overoptimization, 2022

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022. URL https://arxiv.org/abs/2210.10760

    2022 arXiv

  6. [14]

    Text-to-audio generation using instruction-tuned llm and latent diffusion model, 2023

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction-tuned llm and latent diffusion model, 2023. URL https://arxiv.org/abs/2304.13731

    2023 arXiv

  7. [15]

    Reinforced self-training (rest) for language modeling, 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language mode...

    2023 arXiv

  8. [16]

    Efficient diffusion training via min-snr weighting strategy, 2024

    Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy, 2024. URL https://arxiv.org/abs/2303.09556

    2024 arXiv

  9. [17]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https://arxiv.org/abs/2207.12598

    2022 arXiv

  10. [18]

    Denoising diffusion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL https://arxiv.org/abs/2006.11239

    2020 arXiv

  11. [19]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation, 2023 a

    Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation, 2023 a . URL https://arxiv.org/abs/2305.18474

    2023 arXiv

  12. [20]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models, 2023 b

    Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models, 2023 b . URL https://arxiv.org/abs/2301.12661

    2023 arXiv

  13. [21]

    Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. URL https://arxiv.org/abs/2312.06674

    2023 arXiv

  14. [22]

    Elucidating the design space of diffusion-based generative models, 2022

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models, 2022. URL https://arxiv.org/abs/2206.00364

    2022 arXiv

  15. [23]

    A udio C aps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. A udio C aps: Generating captions for audios in the wild. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Compu...

    2019 doi

  16. [24]

    sdpo: Don't use your data all at once, 2024 a

    Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all at once, 2024 a . URL https://arxiv.org/abs/2403.19270

    2024 arXiv

  17. [25]

    Adaptive non-uniform timestep sampling for diffusion model training, 2024 b

    Myunsoo Kim, Donghyeon Ki, Seong-Woong Shim, and Byung-Jun Lee. Adaptive non-uniform timestep sampling for diffusion model training, 2024 b . URL https://arxiv.org/abs/2411.09998

    2024

  18. [26]

    Auto-encoding variational bayes, 2022

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https://arxiv.org/abs/1312.6114

    2022 arXiv

  19. [27]

    Improving text-to-audio models with synthetic captions, 2024

    Zhifeng Kong, Sang gil Lee, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Rafael Valle, Soujanya Poria, and Bryan Catanzaro. Improving text-to-audio models with synthetic captions, 2024. URL https://arxiv.org/abs/2406.15487

    2024 arXiv

  20. [28]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024 arXiv

  21. [29]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024 b . URL https://arxiv.org/...

    2024 arXiv

  22. [30]

    Voicebox: Text-guided multilingual universal speech generation at scale, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale, 2023. URL https://arxiv.org/abs/2306.15687

    2023 arXiv

  23. [31]

    Baton: Aligning text-to-audio model with human preference feedback, 2024

    Huan Liao, Haonan Han, Kai Yang, Tianjiao Du, Rui Yang, Zunnan Xu, Qinmei Xu, Jingquan Liu, Jiasheng Lu, and Xiu Li. Baton: Aligning text-to-audio model with human preference feedback, 2024. URL https://arxiv.org/abs/2402.00744

    2024 arXiv

  24. [32]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL https://arxiv.org/abs/2210.02747

    2023 arXiv

  25. [33]

    Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu

    Alexander H. Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre-training for speech with flow matching, 2024 a . URL https://arxiv.org/abs/2310.16338

    2024 arXiv

  26. [34]

    Plumbley

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models, 2023. URL https://arxiv.org/abs/2301.12503

    2023 arXiv

  27. [35]

    Plumbley

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining, 2024 b . URL https://arxiv.org/abs/2308.05734

    2024 arXiv

  28. [36]

    Flashaudio: Rectified flows for fast and high-fidelity text-to-audio generation, 2024 c

    Huadai Liu, Jialei Wang, Rongjie Huang, Yang Liu, Heng Lu, Wei Xue, and Zhou Zhao. Flashaudio: Rectified flows for fast and high-fidelity text-to-audio generation, 2024 c . URL https://arxiv.org/abs/2410.12266

    2024 arXiv

  29. [37]

    Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL https://arxiv.org/abs/2209.03003

    2022 arXiv

  30. [38]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  31. [39]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization, 2024

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization, 2024. URL https://arxiv.org/abs/2404.09956

    2024 arXiv

  32. [40]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. Wav C aps: A ChatGPT -assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and ...

    2024

  33. [41]

    Ot-flow: Fast and accurate continuous normalizing flows via optimal transport, 2021

    Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport, 2021. URL https://arxiv.org/abs/2006.00104

    2021 arXiv

  34. [42]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  35. [43]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https://arxiv.org/abs/2402.13228

    2024 arXiv

  36. [44]

    Iterative reasoning preference optimization, 2024

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024. URL https://arxiv.org/abs/2404.19733

    2024 arXiv

  37. [45]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https://arxiv.org/abs/2212.09748

    2023 arXiv

  38. [46]

    Scaling laws for reward model overoptimization in direct alignment algorithms, 2024 a

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024 a . URL https://arxiv.org/abs/2406.02900

    2024 arXiv

  39. [47]

    From r to q^* : Your language model is secretly a q-function, 2024 b

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q^* : Your language model is secretly a q-function, 2024 b . URL https://arxiv.org/abs/2404.12358

    2024 arXiv

  40. [48]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024 c . URL https://arxiv.org/abs/2305.18290

    2024 arXiv

  41. [49]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    2023 arXiv

  42. [50]

    High-resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/abs/2112.10752

    2022 arXiv

  43. [51]

    Improved techniques for training gans, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016. URL https://arxiv.org/abs/1606.03498

    2016 arXiv

  44. [52]

    Denoising diffusion implicit models, 2022

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https://arxiv.org/abs/2010.02502

    2022 arXiv

  45. [53]

    Generative modeling by estimating gradients of the data distribution, 2020

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020. URL https://arxiv.org/abs/1907.05600

    2020 arXiv

  46. [54]

    Salmonn: Towards generic hearing abilities for large language models, 2024

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models, 2024. URL https://arxiv.org/abs/2310.13289

    2024 arXiv

  47. [55]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  48. [56]

    Audiobox: Unified audio generation with natural language prompts, 2023

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Summers, ...

    2023 arXiv

  49. [57]

    Diffusion model alignment using direct preference optimization, 2023

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023. URL https://arxiv.org/abs/2311.12908

    2023 arXiv

  50. [58]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing...

    2023

  51. [59]

    Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation, 2024

    Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation, 2024. URL https://arxiv.org/abs/2401.01044

    2024 arXiv

  52. [60]

    Self-rewarding language models, 2024

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024. URL https://arxiv.org/abs/2401.10020

    2024 arXiv

  53. [61]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203.14465

    2022 arXiv

  54. [62]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  55. [63]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  56. [64]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.