{"id":"c7eaa70a-8f1a-49ae-9bc9-8255f4f68024","arxiv_id":"2412.19351","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ETTA, a text-to-audio model trained on a large synthetic caption dataset, outperforms public-data baselines on AudioCaps and MusicCaps and rivals proprietary-data systems.","lead":"This paper systematically tests how data, architecture, training objective, and sampling choices affect text-to-audio generation, and releases a model (ETTA) trained on a new 1.35M-caption synthetic dataset. The study maps which design decisions matter most, giving practitioners a recipe for building competitive TTA systems from public data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-paper metric protocol mismatch, shown by the gap between reported and reproduced Stable Audio Open FDO, leaves the SOTA claim unverified.","rationale":"I read the paper in good faith: it is a comprehensive, transparent design-space study with extensive ablations, and its internal comparisons (Tables 4-6, etc.) are valuable and credible. The strongest claim, however, is explicitly comparative across papers: 'SOTA ... using only publicly available data' and 'comparable to models trained with proprietary data.' That claim is only as strong as the cross-paper metric protocol. The paper gives no evidence that baseline numbers were recomputed under a matched protocol; indeed, its own Stable Audio Open reproduction differs from the published number on AudioCaps FDO (105.88 vs 78.24), which is direct evidence of a protocol gap. The reader identified the same weakest assumption as mine, so I agree with the reader's framing. I do not see an internal inconsistency or a more fundamental flaw; the design-space conclusions are well supported by controlled ablations and by the documented negative results. The concern is not that the authors acted improperly; it is that the headline claim's evidential basis is not protocol-matched, and the magnitude of the mismatch is unknown. Therefore the verdict stays CONDITIONAL, with the condition being a matched-protocol re-evaluation or an explicit delimitation of the claim to 'best under our own evaluation pipeline.' I also note that the paper's own comparisons with the base model (Stable Audio Open) are internally consistent and strong, and those already support a weaker but still meaningful claim: the design choices improve over the base system. The cross-paper SOTA framing is the part that needs the extra evidence.","tokens_in":31926,"tokens_out":3205,"duration_ms":26561,"concrete_test":"Take the released ETTA weights and the official Stable Audio Open weights, generate audio on the AudioCaps and MusicCaps test sets using a single shared evaluation script with identical audio length (10s), resampling, loudness normalization, prompt template, and the same CLAP checkpoints (LAION 630k-best and MS-CLAP 2023). Recompute FDP, FDO, KLS, KLP, ISP, CLL, CLM for both models and for at least one additional open baseline from an official release (e.g., AudioLDM2-large). If the margins in Tables 2 and 3 persist under this matched protocol, the SOTA claim is confirmed; if the margins shrink below the reported differences (especially for FDO and CLM), the claim should be revised to 'improves over Stable Audio Open under the authors' evaluation protocol, with cross-paper comparisons unverified.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The SOTA claim (Section 4.3, after Tables 2 and 3: 'ETTA is the SOTA text-to-audio and text-to-music generation model using only publicly available data. It is also comparable to models trained with proprietary and/or licensed data.') rests on comparing ETTA's own measurements with 'best reported' baseline numbers. The paper gives no evidence that those numbers were produced under a matched protocol, and internal evidence suggests they were not. Specifically, in Table 2 the published 'best reported' FDO for Stable Audio Open is 78.24 on AudioCaps, while the paper's own reproduction of the same model reports FDO=105.88 with otherwise identical metrics. A 27.64-point gap from the same model and benchmark is far too large to ignore: it indicates differences in audio duration, prompt template, resampling, metric implementation, or checkpoint. Under that gap, ETTA's FDO=80.13 is not directly comparable to 78.24. The same issue affects Table 3, where Stable Audio Open's published FDO is 127.20 and the paper's reproduction is 127.20, but the 'best reported' values come from different source pipelines, and the paper reports no recomputation for any baseline. CLAP scores (CLL, CLM) are sensitive to the exact CLAP checkpoint and audio preprocessing; the paper uses two CLAP checkpoints, but baselines reported other checkpoints or none. For KL, IS, and FD, official implementations vary (PANNs vs PaSST versions, feature extraction details). Because the claimed margin over several public-data baselines is small (e.g., KL_S 0.84 vs 0.94 in MusicCaps; FD_P 13.12 vs 16.51 in AudioCaps), protocol noise of the observed magnitude could change or erase the ranking. The claim 'comparable to models trained with proprietary data' is also under-specified because the base model Stable Audio Open is itself trained in part on licensed data, and the fine-tuned ETTA-FT-AC-100k models approach or exceed some proprietary systems only on one or two metrics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale empirical study of design choices for text-to-audio diffusion and flow-matching models. It introduces AF-Synthetic, a 1.35M-caption synthetic dataset generated with Audio Flamingo and filtered by CLAP similarity; proposes ETTA-DiT, an improved DiT implementation; and systematically ablates training data, model architecture and size, training objective, and sampling strategy. The resulting model, ETTA, is evaluated on AudioCaps and MusicCaps and claimed to be state-of-the-art among models trained on public data, while also being competitive with models trained on proprietary data. The paper additionally documents mixed and negative results in Appendix F, releases code, and provides a demo page.","tokens_in":32225,"tokens_out":7375,"duration_ms":58378,"significance":"If the headline claim is supported, the paper would be a valuable reference for TTA design: it isolates data, architecture, objective, and sampler contributions more systematically than prior work, and the AF-Synthetic dataset and NFE/CFG Pareto analyses are useful contributions independent of the SOTA claim. The documentation of negative results (Appendix F) and the release of code are strengths. However, the SOTA claim currently rests on cross-paper metric comparisons that are not protocol-matched, and several design conclusions lack variance estimates or are weakened by internal inconsistencies. These issues are fixable but require additional experiments or careful reformulation.","major_comments":[{"comment":"The SOTA claim is not supported by the evidence as presented because 'best reported' baseline numbers are compared with the authors' own measurements without protocol matching. Table 2 itself reveals the problem: the published FDO for Stable Audio Open is 78.24, while the paper's reproduction of the same model with the same metric list reports 105.88. A 27.6-point gap on the same model and benchmark implies differences in evaluation pipeline (audio duration, resampling, metric implementation, or checkpoint). Consequently, ETTA's FDO of 80.13 cannot be directly compared with the published 78.24. Please either reproduce all competing baselines with the exact evaluation pipeline used for ETTA, or change the claim to 'competitive with published numbers' and explicitly state the protocol caveat. The same issue affects Table 3 and the SongDescriber results in Table 23.","section":"Section 4.3, Tables 2 and 3"},{"comment":"All objective metrics are reported as single point estimates without variance, multiple seeds, or significance tests. Several design conclusions are drawn from differences comparable to typical run-to-run noise: in Table 4, the switch to logit-normal time sampling changes FDP by 0.57 and KLS by 0.06 in opposite directions, and in Table 5 AF-AudioSet vs AF-Synthetic on MusicCaps differ by 0.19 in FDP and 0.04 in KLS while the text claims AF-Synthetic is 'consistently better'. Without error bars or repeated runs, these attributions are not established. At minimum, report multiple seeds for the key ablation chains (Tables 4 and 5) or state explicitly which differences are within expected noise.","section":"Section 4.2, Tables 4-6, 17-18"},{"comment":"The AF-Synthetic construction filters captions using CLAP similarity from the 'Laion-CLAP 630k-audioset-fusion-best' checkpoint, and the evaluation suite includes CLL from 'LAION's 630k-best' checkpoint plus CLM from MS-CLAP. Please state clearly whether the filtering checkpoint and the CLL evaluation checkpoint are the same; the naming suggests they may be different. Even if they differ, because both dataset selection and a headline text-adherence metric use CLAP-family embeddings, the paper should include a robustness check showing that ETTA's CLAP gains are not an artifact of training on captions pre-selected by CLAP. Concretely, evaluate with a CLAP checkpoint not used anywhere in the pipeline and show that the relative ranking over baselines is preserved; the human REL scores and KL/FD gains are helpful but do not fully address this point.","section":"Section 3.1 and Section 4.3, metrics"},{"comment":"The data-quality conclusion ('quality matters more') is compromised because the AF-AudioSet baseline appears to be evaluated at the onset of training instability. Appendix D states that AF-AudioSet 'quickly diverges around 250K steps and is unable to continue its training', and Table 5 reports results from models trained for 250K steps. The comparison between AF-AudioSet and AF-Synthetic in Table 5 may therefore conflate data quality with training stability. Please either use a checkpoint taken before divergence (e.g., 200K steps) or otherwise show that the comparison is unaffected by the divergence.","section":"Section 4.2 and Appendix D (Figure 4), Table 5"}],"minor_comments":[{"comment":"The text states 'All metrics improve as the number of function evaluations (NFE) increases', but Table 19 shows Euler FDP worsens from 11.83 at NFE=50 to 12.10 at NFE=100; the statement should be qualified or removed.","section":"Section 4.3, Figure 3 and Table 19"},{"comment":"The table header says 'million captions' and lists TangoPromptBank as 1.21, while the text says 2.33M audio-caption pairs were used; please align the table and text.","section":"Section 4.2, Table 5"},{"comment":"The text says ETTA achieves the 'highest CLAP scores' on SongDescriber, but CLL for ETTA (0.44) is tied with AudioLDM2-large (0.44); please say 'highest CLM' or 'tied for highest CLL'.","section":"Section 4.3 and Table 23"},{"comment":"There are typos such as 'sigificaly' and 'simillar'; Table 11 captions contain 'sniping' and 'footnote', which appear to be errors; and the reference for Schneider et al. (2024) contains the garbled word 'moˆusai'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the headline 'SOTA on public data' claim will be cited without the protocol caveat. I would require the authors to either reproduce all baselines under a matched evaluation pipeline or substantially soften the claim. The paper's strengths -- the AF-Synthetic dataset, the systematic ablations, and the documented negative results -- justify a major revision rather than rejection, provided the comparison issue is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about the ETTA paper. First, the actual contribution is a large, careful, and unusually honest ablation of the text-to-audio design space, plus the AF-Synthetic caption dataset. Second, the headline claim — SOTA on public data, comparable to proprietary systems — is not verified, because the comparisons use other papers' \"best reported\" numbers without a matched evaluation protocol.\n\nWhat is new and good: AF-Synthetic is the first million-scale synthetic caption dataset with CLAP similarity filtering, built by scaling the AF-AudioSet pipeline to 3.6K hours across five sources. The ablations are systematic: data, DiT implementation details (AdaLN, RoPE, zero-initialized final layers), v-diffusion vs OT-CFM with logit-normal t-sampling, model depth/width, and sampler/CFG Pareto curves. The internal results are consistent and the margins are large. Even without the cross-paper comparisons, ETTA's improvements over the authors' own reproduction of Stable Audio Open are substantial (e.g., FD_P roughly halved on both benchmarks). The paper also documents negative results — inpainting pretraining, text encoder choices, autoguidance, Min-SNR, limited-interval CFG — which raises my trust in the positive findings.\n\nThe soft spots. The \"SOTA on public data\" sentence in Section 4.3 relies on Tables 2-3 mixing their own measurements with \"best reported\" values. Internal evidence says the protocols are not matched: the published FDO for Stable Audio Open on AudioCaps is 78.24, but their own reproduction of the same model reports 105.88. That 27.6-point gap is too large to ignore, and it makes ETTA's 80.13 not directly comparable to 78.24. This is a load-bearing flaw for the headline claim, though not for the design-space analysis, which uses a consistent protocol throughout. I also note no error bars on objective metrics, CFG/NFE were selected using the test benchmarks, and there is no direct link to the AF-Synthetic dataset in the paper.\n\nWho is this for: anyone building or evaluating text-to-audio models will get good value from the ablations and the dataset. This deserves a serious referee. My recommendation: send it, and ask the authors to either re-evaluate baselines under their own protocol or substantially soften the SOTA claim.","headline":"The design-space study is the real contribution; the SOTA claim leans on unverified cross-paper numbers, so read it for the recipe, not the headline.","tokens_in":32911,"tokens_out":3979,"would_cite":true,"duration_ms":35521,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A systematic sweep of the text-to-audio design space finds synthetic captions are the dominant quality lever, and the resulting model, ETTA, rivals proprietary-data systems.","keywords":["text-to-audio generation","synthetic captions","AF-Synthetic","diffusion transformer","flow matching","latent diffusion","classifier-free guidance","creative audio generation"],"falsifier":"Run a matched evaluation in one shared harness: take ETTA and the leading baselines from Tables 2 and 3, and score all of them with identical 10-second preprocessing, identical implementations of the Fréchet distance, KL divergence, inception score, and CLAP metrics, and identical audio and text encoders. If several baselines match or beat ETTA's numbers under those conditions, the state-of-the-art-on-public-data claim would not survive.","tokens_in":31719,"feed_emoji":"🎧","tokens_out":8297,"duration_ms":68137,"temperature":0.7,"pith_summary":"This paper tries to establish which design choices actually drive text-to-audio (TTA) quality, by running a controlled sweep over training data, model architecture, training objective, and sampling. Its central empirical claim is that the single largest gain comes from replacing noisy or weakly related captions with 1.35 million synthetic captions that are strongly correlated with their audio, generated by an audio understanding model and filtered by a learned audio-text similarity score. A stabilised Diffusion Transformer implementation, an optimal-transport conditional flow matching objective, and logit-normal timestep sampling add further, smaller gains. The resulting model, ETTA, is claimed to be state of the art among models trained on publicly available data on AudioCaps and MusicCaps, and competitive with models trained on proprietary or licensed data. If correct, this identifies the bottleneck of current TTA systems in caption quality rather than in model capacity, and shows that a public-data system can approach closed-data performance.","feed_headline":"Synthetic captions let public text-to-audio rival closed models","feed_subtitle":"A controlled sweep finds caption quality is the biggest lever; ETTA matches systems trained on private data.","key_machinery":"The object that carries the argument is AF-Synthetic, a dataset of 1.35 million audio-caption pairs built by having an audio understanding model propose captions for ten-second audio segments and then keeping, for each segment, the caption with the highest similarity to the audio as measured by a learned audio-text scoring model, rejecting pairs below a similarity threshold of 0.45 and keyword-filtering low-quality audio. This dataset is what converts the paper's other components into benchmark gains. The second mechanism is ETTA-DiT, a Diffusion Transformer whose changes, including adaptive layer norm applied to every block, a zero-initialised final projection, rotary position embeddings run in FP32, and dropout, make training more stable and improve convergence. The remaining machinery is the training objective, optimal-transport conditional flow matching with logit-normal timestep sampling, and an inference recipe chosen from Pareto sweeps over solver, number of function evaluations, and classifier-free guidance scale.","core_discovery":"The paper's central claim is that ETTA, a latent diffusion model trained with an optimal-transport conditional flow matching objective on a synthetic-caption dataset, achieves state-of-the-art text-to-audio and text-to-music results among models trained only on publicly available data, and is comparable to models trained with proprietary or licensed data. The claim is built from a sequence of ablations: adding AF-Synthetic captions to the reference setup is the largest single improvement; replacing the reference DiT with ETTA-DiT gives the next step; switching from v-prediction diffusion to OT-CFM improves training stability and robustness at low sampling budgets; and logit-normal t-sampling adds a small final gain. The paper also claims that ETTA follows complex and imaginative prompts better than baselines in a human evaluation, suggesting that benchmark-style captions understate its ability. The authors present these results as an elucidation of the TTA design space, with data quality as the dominant factor and model capacity, text encoder choice, and sampling strategies as secondary levers.","pith_inferences":["If caption quality is the dominant bottleneck, a natural extension is to test whether scaling AF-Synthetic-style captions beyond 1.35 million continues to improve benchmarks or whether returns flatten, given the paper's data comparison suggests quality rather than sheer volume is what matters.","The 'comparable to proprietary models' result would be on firmer footing with a shared evaluation harness, so re-scoring all baselines under identical preprocessing and identical audio-text scoring models is the test most likely to revise the ranking.","The creative-caption protocol of twenty imaginative prompts with human OVL and REL ratings could be expanded into a public benchmark that tracks compositional understanding separately from standard AudioCaps and MusicCaps scores."],"forward_implications":["If the central claim holds, the practical gap between public-data and proprietary-data text-to-audio systems is largely a caption-quality problem, not an architecture or compute problem.","Future TTA research can prioritise building larger pools of strongly audio-correlated synthetic captions over adding model capacity, since the ablations show data quality outranks dataset size.","Flow matching with OT-CFM is a safer default than v-prediction diffusion for long TTA training runs, because it stays stable past 500k steps where v-diffusion diverges in the paper's loss curves.","Sampling choices should be tuned per objective, because the paper's Pareto curves show the Fréchet distance behaves as a convex function of the guidance scale while KL, IS, and CLAP scores keep improving, so the best guidance scale depends on whether diversity or prompt adherence is being optimised.","The creative-caption human evaluation suggests standard benchmarks underestimate model ability, implying that TTA progress should also be measured on compositional and imaginative prompts."],"supporting_citations":[{"why":"Supplies the synthetic-caption pipeline, including Audio Flamingo captions and the CLAP similarity threshold of 0.45, that AF-Synthetic scales up.","marker":"(Kong et al., 2024b)"},{"why":"Describes Audio Flamingo, the audio understanding model used to generate the ten candidate captions per audio segment.","marker":"(Kong et al., 2024a)"},{"why":"Provides Stable Audio Open, the base model, VAE, and reference DiT implementation that ETTA starts from and is compared against.","marker":"(Evans et al., 2024c)"},{"why":"Defines the Diffusion Transformer architecture that ETTA-DiT modifies with AdaLN, zero-initialised projections, RoPE, and dropout.","marker":"(Peebles & Xie, 2023)"},{"why":"Introduces the flow matching objective that ETTA uses in its optimal-transport conditional flow matching training.","marker":"(Lipman et al., 2022)"},{"why":"Supplies the conditional flow matching formulation and optimal-transport construction used for OT-CFM.","marker":"(Tong et al., 2023)"},{"why":"Provides classifier-free guidance, the inference-time conditioning method whose scale is swept in the paper's Pareto analysis.","marker":"(Ho & Salimans, 2022)"},{"why":"Supplies the CLAP audio-text similarity model used both for filtering AF-Synthetic captions and as an evaluation metric.","marker":"(Wu et al., 2023)"},{"why":"Defines the AudioCaps benchmark on which ETTA and the baselines are evaluated.","marker":"(Kim et al., 2019)"},{"why":"Defines the MusicCaps benchmark used to evaluate text-to-music generation.","marker":"(Agostinelli et al., 2023)"}],"fun_headline_variants":["Synthetic captions unlock public-data text-to-audio on par with private","Text-to-audio design space mapped: caption quality is king","ETTA: public data only, but beats private-trained audio rivals","Caption quality drives text-to-audio: ETTA matches closed models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparisons assume the published scores for other models were measured with the same evaluation protocol as ETTA's, meaning identical audio preprocessing, identical metric implementations, and identical audio-text scoring models, so that numbers taken from different papers can be read as directly comparable.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic captions unlock public-data text-to-audio on par with private","Text-to-audio design space mapped: caption quality is king","ETTA: public data only, but beats private-trained audio rivals","Caption quality drives text-to-audio: ETTA matches closed models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2432,"prompt_tokens":981,"completion_tokens":1451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1375}},"tokens_in":597,"tokens_out":1451,"duration_ms":10172,"temperature":1.0,"reasoning_tokens":1375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:40:37.407997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a matched evaluation in one shared harness: take ETTA and the leading baselines from Tables 2 and 3, and score all of them with identical 10-second preprocessing, identical implementations of the Fréchet distance, KL divergence, inception score, and CLAP metrics, and identical audio and text encoders. If several baselines match or beat ETTA's numbers under those conditions, the state-of-the-art-on-public-data claim would not survive.","supporting_citations":[],"review_version":1}