{"id":"596be5b5-e592-48f7-8b80-8b7330e4eda7","arxiv_id":"2509.02398","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TTA-Bench offers a seven-dimension, 2,999-prompt evaluation of ten text-to-audio models with 118,000 human ratings, covering quality, robustness, fairness, bias, and toxicity.","lead":"The authors built a 2,999-prompt test suite that grades ten text-to-audio models on sound quality, speed, unusual inputs, robustness to typos, fairness, and toxic output. It may become the standard report card for audio-generation systems, which currently lack a common safety and reliability benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generalization prompts' out-of-distribution claim is unverified: rare labels are drawn from the AudioSet ontology, so component-level overlap with training data is likely; the measured 'generalization gap' may reflect novel phrasing rather than true OOD.","rationale":"The reader's conditional verdict is appropriate, and I agree that the weakest assumption is the unverified out-of-distribution status of the generalization prompts. I focused on this rather than the other listed issues because it is the one that threatens the validity of a whole dimension of the benchmark, not just reporting quality. The robustness contradiction, garbled bias formula, unvalidated gender-detection API, and missing figure reference are real but secondary: they can be corrected without changing what the generalization experiment measures. The generalization dimension, by contrast, is explicitly presented as a novel contribution, and the paper's conclusion that current TTA models 'struggle to generalize beyond seen domains' rests on the 300 prompts being outside the training distribution. The construction details make this assumption especially fragile: rare labels come from the AudioSet ontology, and 'unseen' labels are generated by an LLM without verification against any training corpus. Since all ten models are trained on AudioSet-family data, the label vocabulary is likely in-distribution even if the final prompt sentences are novel. The proposed overlap analysis and stratification by overlap would settle whether the measured drop is true distribution shift or an artifact of novel phrasing. I would keep the CONDITIONAL verdict: the paper should verify and report this OOD check before the generalization results are taken as reliable.","tokens_in":25967,"tokens_out":5432,"duration_ms":64713,"concrete_test":"For each of the 300 generalization prompts, extract the component label set used in construction. Build text corpora from the declared training data of the ten models (AudioCaps captions, AudioSet class labels/descriptions, Freesound tags, FMA metadata, Audio-Alpaca, etc.). Compute (a) exact and 3-gram overlap between each prompt and each corpus, (b) CLAP text-text similarity between each prompt and its nearest training caption, and (c) whether each component label appears verbatim in the model's training-label vocabulary. Then stratify the Figure 3 accuracy/alignment drop by overlap tercile and by zero-component-overlap subsets. If the drop persists in the lowest-overlap tercile and in prompts with no component overlap, the OOD interpretation survives; if the drop concentrates in high-overlap prompts or disappears when overlap is controlled, the generalization metric should be reinterpr","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TTA-Bench provides reliable insights into generalization depends on the 300 generalization prompts being genuinely out-of-distribution for all ten evaluated models. The construction procedure in 'Generalization Prompt Collection' starts from the AudioSet ontology, asks an LLM to split labels into common and rare pools, adds 30 LLM-invented 'unseen' labels, samples label combinations, and asks an LLM to render each combination as an imaginative sentence. Because almost every evaluated model is trained on AudioSet or AudioCaps (Table 2), rare AudioSet-ontology labels are, at the label level, in-distribution even if the final sentences are novel. The paper reports no overlap check between the final prompt texts and the models' training captions, no component-label membership check, and no analysis of how the quality/alignment drop varies with lexical or embedding similarity to training data. If the drop in Figure 3 / Table 6 is driven by LLM phrasing style rather than genuinely unseen events, the headline finding that 'models struggle to generalize beyond seen domains' is not established. This is the most load-bearing weakness because generalization is one of the seven benchmark dimensions and a stated novelty of TTA-Bench; if the OOD premise fails, that dimension measures something else, and the benchmark's claim to reliable, holistic evaluation is weakened.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TTA-Bench, a text-to-audio evaluation benchmark covering seven dimensions organized under functional quality, reliability, and social responsibility: accuracy, efficiency, generalization, robustness, fairness, bias, and toxicity. The benchmark comprises 2,999 prompts constructed through dataset extraction, LLM-assisted generation, manual writing, and adaptation of image-to-prompts for audio, and it proposes a unified protocol combining objective metrics (AES, CLAP, RTF) with 118,314 human annotations from expert and general raters. Ten TTA models are benchmarked, and the paper reports dimension-wise results, concluding that current models perform well on in-distribution accuracy but struggle with generalization, robustness, bias, and toxicity. The dataset and evaluation tools are stated to be open-sourced.","tokens_in":26279,"tokens_out":5694,"duration_ms":70637,"significance":"If the methodological concerns are addressed, this is a valuable and potentially standard-setting resource. The scale of annotation (118,314 human judgments), the breadth across seven dimensions, the inclusion of expert and non-expert raters, and the adaptation of the I2P toxicity taxonomy to the audio domain are concrete strengths. The dataset and tools are open-sourced, which supports reproducibility and follow-up work. The paper is also honest about the limitations of existing TTA evaluation and makes a credible case that robustness, fairness, bias, and toxicity are understudied. However, the benchmark's reliability as a holistic instrument depends on two load-bearing issues: the unverified out-of-distribution status of the generalization prompts, and the absence of uncertainty quantification in most model comparisons. Both are fixable but currently weaken the central claims.","major_comments":[{"comment":"The generalization dimension is premised on 'rare/unseen' events. The rare label pool is derived from the AudioSet ontology, and Table 2 shows that nearly every evaluated model is trained on AudioSet or AudioCaps. The paper provides no overlap check between the 300 final prompt texts and the models' training captions, no component-label membership check, and no analysis linking the measured quality/alignment drop to lexical or embedding similarity with training data. Consequently, the headline conclusion that 'models struggle to generalize beyond seen domains' is not established; the drop in Figure 3/Table 6 could be driven by LLM phrasing style or by label combinations that are novel as sentences but in-distribution at the event-label level. Please add a quantitative OOD verification (e.g., n-gram or CLAP-embedding similarity against AudioCaps and other training corpora, or a held-out e","section":"Data Construction, Generalization Prompt Collection; Table 2; Figure 3/Table 6"},{"comment":"The paper makes fine-grained model comparisons (e.g., Tango 2 vs AudioLDM 2 in Table 5; 'Tango 2 maintains strong performance' in Generalization Results) but reports no confidence intervals, bootstrap estimates, or significance tests for any model-level difference. Subjective scores are based on 3 expert and 10 crowd raters per clip, so system means have nontrivial uncertainty. The only significance tests in the paper (Table 12) compare expert vs non-expert preferences, not model performance. The absence of uncertainty quantification is load-bearing because many claims in the text are ranking statements, and adjacent systems often differ by fractions of a point. Please add per-system CIs and pairwise significance tests or bootstrap intervals for Tables 5-7, and temper claims where differences are within noise.","section":"Experimental Results, Tables 5-7; Appendix 3"},{"comment":"The bias analysis relies on an unnamed 'commercial system API' for gender detection, with no validation of its accuracy on synthesized audio. When AudioLDM is excluded from 75.3% of outputs and Stable Audio Open from about 40%, the MAD values in Table 7 are computed on small, non-representative subsets, yet are compared across systems as if commensurate. Figure 21 reports gender proportions without any uncertainty or sample-size information. Please identify the API or use an open-source detector, report its accuracy on TTA-generated audio, give the per-system sample sizes behind each MAD, and report bias estimates with confidence intervals.","section":"Evaluation Method, Bias; Bias Results; Figure 21"},{"comment":"Toxicity labels are produced by five crowd participants using a majority-vote procedure, but no inter-annotator agreement (e.g., Krippendorff's alpha) is reported, and the treatment of 'Uncertain' labels in the denominator of the toxicity rate is not specified. Table 7's toxicity rates are used to rank systems and to claim that 'sexual content generally has lower toxicity rates' and that 'shocking content and hate speech categories tend to have higher toxicity rates.' With roughly 30 prompt instances per category per system, differences of a few percentage points are likely within annotation noise. Please report agreement statistics, CIs for the rates, and a clear handling rule for 'Uncertain' labels.","section":"Appendix 3, Toxicity Annotation Protocol; Table 7"}],"minor_comments":[{"comment":"The text refers to 'Figure X' when discussing the visualization of preference gaps; this placeholder must be replaced with the actual figure number.","section":"Appendix 4, Expert vs Non-Expert Preferences"},{"comment":"The caption says the x-axis represents 10 systems in alphabetical order, but the plot shows numeric ticks (1-10) and no model names. Readers cannot tell which bar corresponds to which model.","section":"Figure 4"},{"comment":"Model-name capitalization is inconsistent: 'TANGO 2' appears in Table 7 and the toxicity results, while 'Tango 2' is used elsewhere; 'Stable-Audio' vs 'Stable Audio Open'; and 'Aurrfusion' appears in the Figure 21 caption.","section":"Throughout"},{"comment":"The median absolute deviation formula is garbled in the text ('MAD = 1/Nb sum | bNb - 1/Nb |'), and the citation to Pearson (1894) does not seem to define this quantity. Please rewrite the formula and use a standard reference or define it explicitly.","section":"Evaluation Method, Bias"},{"comment":"The uppercase perturbation includes five fixed capitalization ratios (5%, 25%, 50%, 75%, 100%). The paper does not analyze sensitivity to this choice; a sentence on whether the robustness ranking is stable across ratios would improve the discussion.","section":"Data Construction, Robustness"}],"recommendation":"major_revision","confidential_remarks":"This is a resource paper with a useful dataset and a large annotation effort. The main issues are statistical overclaiming and an unverified generalization premise, both of which are addressable in revision. I do not see circularity concerns; self-citations to MusicEval/RAMP are background material, not the object under test. The paper fits the journal's scope if the reliability issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — this is a substantial piece of resource-building. TTA-Bench gives the text-to-audio community a 2,999-prompt benchmark covering seven dimensions (accuracy, efficiency, generalization, robustness, fairness, bias, toxicity) with 118k human annotations across ten models, and the dataset and tools are open-sourced. The dimensions beyond quality — fairness, bias, toxicity — are genuinely firsts for TTA, and the annotation effort is real. That alone makes it useful.\n\nWhat I would push back on: the generalization results are presented as evidence of out-of-distribution behavior, but the construction starts from the AudioSet ontology. Rare labels are still AudioSet labels, and all ten models train on AudioSet/AudioCaps, so the component concepts are in-distribution even if the final LLM-rendered sentences are novel. The paper never checks overlap between prompt text and training captions, nor how the performance drop tracks similarity to training data. So the headline that models struggle to generalize beyond seen domains is not actually established. This is a load-bearing weakness for one of the seven dimensions.\n\nThere are also several smaller reporting problems. Tables 5–7 have no confidence intervals or tests for model comparisons; the bias analysis leans on an unnamed commercial gender-detection API; the MAD formula in the bias section is garbled as printed; the main text and appendix disagree on whether Tango 2 is robust — main text says it degrades substantially, appendix says it is robust; and the expert-versus-nonexpert section references a \"Figure X\" that does not exist. None of these sink the benchmark, but they need fixing for the results to be taken at face value. Minor inconsistency too: the robustness formula multiplies by 100%, but reported values sit around 1.0.\n\nNet: this is a valuable resource for anyone working on TTA evaluation or responsible generative audio. It deserves a serious referee. With revisions — especially recharacterizing generalization as novel prompt composition rather than true OOD, and cleaning up the reporting — it becomes a solid contribution. I would support sending it to review.","headline":"A substantial open-sourced benchmark for text-to-audio evaluation that fills a real gap, but the OOD generalization claim is overreached and the reporting needs cleanup.","tokens_in":26781,"tokens_out":2905,"would_cite":true,"duration_ms":32206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TTA-Bench evaluates text-to-audio models on seven dimensions, from accuracy to toxicity, using 2,999 prompts and 118,314 human annotations; it finds models degrade on rare, complex, or perturbed prompts and show gender bias and toxicity.","keywords":["text-to-audio generation","evaluation benchmark","human annotation","generalization","robustness","fairness","bias","toxicity"],"falsifier":"Check the 300 generalization prompts (and their event-label combinations) for exact or near-verbatim n-gram overlap with the training caption sets of the ten models (AudioSet, AudioCaps, Freesound, Audio-Alpaca). If overlap is found and correlates with model generalization scores—or if re-running the comparison with verified out-of-distribution prompts changes which models lead the ranking—the out-of-distribution premise and the measured generalization gaps are called into question.","tokens_in":25883,"feed_emoji":"🎧","tokens_out":12776,"duration_ms":125835,"temperature":0.7,"pith_summary":"The paper sets out to replace the narrow, quality-only evaluation of text-to-audio (TTA) generation with a single benchmark that also measures reliability and social responsibility. It constructs 2,999 prompts plus an evaluation protocol combining objective metrics (CLAP, AES, real-time factor) with 118,314 human ratings from expert and general listeners, and applies them to ten open TTA models. The central finding is that current models handle familiar, simple prompts well but degrade noticeably on rare/unseen descriptions, multi-event compositional prompts, and surface-level input perturbations, while also showing measurable gender bias and a nonzero rate of toxic audio generation. If adopted as a standard, the benchmark would make robustness, fairness, and safety as trackable as fidelity, and it would push model development toward generalization and responsibility rather than in-distribution quality alone. The paper also documents that experts and general users diverge systematically on four of five subjective metrics, so how a model is rated depends on who is listening.","feed_headline":"First seven-axis benchmark grades text-to-audio models","feed_subtitle":"TTA-Bench scores accuracy, speed, fairness, and safety via 2,999 prompts and 118,314 human ratings.","key_machinery":"The load-bearing object is the benchmark's construction pipeline rather than any single formula: a 50-scene taxonomy with annotated event counts and temporal relations (none, parallel, sequential, complex) for accuracy; common/rare label pools sampled from the AudioSet ontology, plus LLM-written 'never heard in the real world' scenes, for generalization; six surface perturbation types (uppercase, synonym, misspelling, whitespace, rewrite, punctuation) for robustness; gender-neutralized prompts and demographic substitution pairs for bias and fairness; and I2P-adapted plus manually written sound-level toxic prompts for toxicity. The protocol's named measures include the robustness score RSp (m","core_discovery":"The paper claims that TTA-Bench is the first evaluation framework to treat text-to-audio models as systems that must be accurate, reliable, and socially responsible at once. It backs the claim with 2,999 prompts spanning seven dimensions—accuracy, efficiency, generalization, robustness, fairness, bias, toxicity—and a protocol pairing CLAP, AES, and latency metrics with 118,314 expert and general-user ratings. On ten models, the paper reports strong in-distribution accuracy but a drop of up to about one point of quality and alignment on rare/unseen prompts, declines as event count and relation complexity rise, model-specific gender skew in generated voices, and toxic outputs in all five categ","pith_inferences":["The generalization numbers rest on an unverified premise: that LLM-written 'rare/unseen' prompts (e.g., 'crystalline ice flute resonance') are out-of-distribution for all ten models. Searching each model's training corpora (AudioSet, AudioCaps, Freesound, Audio-Alpaca) for these phrases or their event-label components would test this; partial overlap would distort the reported generalization gap.","The bias dimension depends on a commercial gender-recognition API applied to generated speech, which adds an unquantified measurement error. An extension would cross-validate API labels against human perception of voice gender, and would probe other attributes such as accent or vocal age.","The robustness test perturbs only surface text (typos, case, synonyms). An audio-side extension—noise, compression, resampling, or adversarial waveforms—would show whether text-robust models remain robust once the perturbation moves into the acoustic domain.","The expert/lay divergence pattern hints at two separate subjective factors—a shared perception of raw quality and a divergent appreciation for expressiveness—testable by factor analysis or inter-rater reliability modeling on the 118,314 annotation matrix."],"forward_implications":["TTA development targets can shift from in-distribution fidelity to verifiable generalization: the benchmark quantifies, per model, how much quality and alignment are lost on rare, multi-event, or perturbed prompts.","Robustness, fairness, and toxicity become normal, comparable metrics rather than one-off study topics, so a model release can be accompanied by a consistent reliability and safety scorecard.","Because expert and general raters disagree significantly on alignment, usefulness, enjoyment, and complexity (but not on raw quality), future evaluations should report both perspectives separately; a single average hides a systematic listener bias.","The toxicity protocol supplies an operational, sound-level definition of audio toxicity (screams, violence, sexual, shocking, illegal), usable for red-teaming and content-safety audits of generative audio systems.","The finding that I2P-adapted and manually written toxic prompts trigger different per-category toxicity rates implies safety claims should be built on multiple prompt sources, not one."],"supporting_citations":[{"why":"Supplies the pairwise fairness-score formula and the holistic text-to-image benchmark design that TTA-Bench adapts to the audio domain.","marker":"Bakr et al. 2023"},{"why":"Supplies the AudioCaps captions used as accuracy base prompts and as the source of ungendered human-activity sentences for the bias test set.","marker":"Kim et al. 2019a"},{"why":"Supplies the I2P toxicity taxonomy and the 150 image prompts that are transcribed into audio-specific toxic test prompts.","marker":"Schramowski et al. 2023"},{"why":"Provides the CLAP model used as the objective text-audio alignment metric in accuracy, generalization, and fairness scoring.","marker":"Elizalde, Deshmukh, and Wang 2023"},{"why":"Provides the Audiobox-Aesthetics (AES) model used for objective quality and enjoyment scoring.","marker":"Tjandra et al. 2025"},{"why":"NL-Augmenter supplies the misspelling transformation used as one of the six robustness perturbation types.","marker":"Dhole et al. 2023"},{"why":"The prior temporally-aligned TTA benchmark whose limited scope motivates TTA-Bench's broader evaluation design.","marker":"Xie et al. 2025"},{"why":"Cited for the generalization definition and the bias-detection approach used in the holistic evaluation literature.","marker":"Lee et al. 2023"}],"fun_headline_variants":["Seven-axis benchmark probes TTA models beyond sound quality","TTA-Bench: 2,999 prompts, 118,314 ratings grade TTA models","New TTA benchmark checks fairness and toxicity, not just audio quality","TTA-Bench reveals robustness drops and gender bias in audio models","Benchmark tests TTA models on accuracy, fairness, and safety"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The generalization measurement assumes the LLM-written 'rare or unseen' prompts and their component audio labels are truly absent from the training data of all ten benchmarked models, but the paper never checks the prompts against the corpora those models were trained on.","fun_headline_variants_meta":{"raw":{"variants":["Seven-axis benchmark probes TTA models beyond sound quality","TTA-Bench: 2,999 prompts, 118,314 ratings grade TTA models","New TTA benchmark checks fairness and toxicity, not just audio quality","TTA-Bench reveals robustness drops and gender bias in audio models","Benchmark tests TTA models on accuracy, fairness, and safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2666,"prompt_tokens":682,"completion_tokens":1984,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":1889}},"tokens_in":426,"tokens_out":1984,"duration_ms":16880,"temperature":1.0,"reasoning_tokens":1889,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:35:36.956942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the 300 generalization prompts (and their event-label combinations) for exact or near-verbatim n-gram overlap with the training caption sets of the ten models (AudioSet, AudioCaps, Freesound, Audio-Alpaca). If overlap is found and correlates with model generalization scores—or if re-running the comparison with verified out-of-distribution prompts changes which models lead the ranking—the out-of-distribution premise and the measured generalization gaps are called into question.","supporting_citations":[{"cited_title":"M.; Sun, P.; Shen, X.; Khan, F","cited_arxiv_id":null,"evidence_quote":"Supplies the pairwise fairness-score formula and the holistic text-to-image benchmark design that TTA-Bench adapts to the audio domain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the I2P toxicity taxonomy and the 150 image prompts that are transcribed into audio-specific toxic test prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Audiobox-Aesthetics (AES) model used for objective quality and enjoyment scoring."},{"cited_title":"D.; Gangal, V.; Gehrmann, S.; Gupta, A.; Li, Z.; Mahamood, S.; Mahendiran, A.; Mille, S.; Srivastava, A.; Tan, S.; Wu, T.; Sohl-Dickstein, J.; Choi, J","cited_arxiv_id":null,"evidence_quote":"NL-Augmenter supplies the misspelling transformation used as one of the six robustness perturbation types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior temporally-aligned TTA benchmark whose limited scope motivates TTA-Bench's broader evaluation design."}],"review_version":1}