{"id":"f79fc2d0-1b51-4b1f-a55e-2050823c0bd8","arxiv_id":"2501.08587","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Four text-to-audio systems were evaluated against a human reference in the DCASE 2024 Task 7 challenge, with a 36% quality gap and strong but small-sample FAD-to-human correlation.","lead":"This paper reports the DCASE 2024 sound scene synthesis challenge, where four systems generated 4-second audio from text prompts such as 'dog barking with traffic in the background.' It found that the best system still scored well below a human-created reference, while automatic and human ratings agreed closely for the five systems tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FAD metric uses one sound engineer's recordings as the unique reference per prompt (§3.1, §4.1); for open-ended text-to-audio this penalizes acoustically valid but different scenes, so the reported FAD-human correlations may measure reference-distance rather than quality.","rationale":"The paper is a transparent challenge report rather than a novel scientific derivation; the human evaluation is carefully conducted (14 blind raters, self-ratings removed, Cronbach's alpha 0.959), and the authors honestly report the small number of systems and the reasons for discontinuing the task. We therefore do not think the paper should be rejected. However, the strongest claim leans on FAD-human correlations to validate the framework. The reader's weakest_assumption—that a single sound-engineer reference is treated as the unique target for each open-ended prompt—is the most load-bearing concern. It is more fundamental than the small-sample issue because it would undermine FAD's validity even with many systems. The paper flags the small sample in §5.2 but does not flag this reference-dependence. A multi-engineer reference test can empirically settle whether FAD rankings and correlations are stable or depend on the particular engineer's choices. Our recommendation is to keep the CONDITIONAL verdict, with the explicit condition that the FAD results be shown to be robust to reference-set variation.","tokens_in":6207,"tokens_out":11115,"duration_ms":103288,"concrete_test":"Recruit 3 additional sound engineers to independently create reference recordings for the 24 captions used in the subjective evaluation, following the same quality guidelines as §3.1. Using the provided fadtk code, compute FAD-P scores for the 4 submitted systems plus the baseline against each engineer's reference set. (a) Compute Spearman rank correlations of the 5 systems' FAD scores across the 4 reference sets. (b) Recompute the Pearson correlations between each reference set's FAD scores and the existing human foreground-fit, background-fit, and audio-quality ratings. If FAD rankings vary across reference sets (Spearman ρ < 0.8) or any FAD-human correlation drops below 0.5 or changes sign, the single-reference assumption is load-bearing and the reported conclusions are reference-dependent. If rankings and correlations are stable across engineers, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that its FAD-based objective evaluation, validated by high correlations with human ratings (0.94, 0.94, 0.77 in §5.2), provides a standardized framework for sound scene synthesis. This rests on the assumption in §4.1 that FAD(r,g) with r the reference set measures generation quality. The reference set was created by a single sound engineer (§3.1), with exactly one recording per prompt in the 250-caption evaluation set. For open-ended text-to-audio, many acoustically different recordings legitimately match a caption like 'Foreground with Background in the background'. FAD therefore penalizes any system whose outputs are valid but acoustically distant from that engineer's specific choices. The paper acknowledges the small sample (5 systems) in §5.2, but it does not acknowledge this reference-set assumption. If a different engineer had designed the reference, FAD scores could re-rank systems and the correlations could change. Thus the objective component of the framework is not established as a measure of perceptual quality; it may simply measure distance to one reference rendering.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on DCASE 2024 Challenge Task 7, a text-to-sound generation task for sound scenes. The authors describe a standardized evaluation framework combining an objective metric, Fréchet Audio Distance (FAD) with PANN-Wavegram-Logmel embeddings, and human perceptual ratings on three scales (Foreground Fit, Background Fit, Audio Quality), aggregated into a weighted Perceptual Score. The dataset consists of 310 audio-caption pairs (60 development, 250 evaluation) created by a sound engineer, and the task constrains outputs to 4-second mono clips without music or intelligible speech. Four submitted systems and an AudioLDM baseline were evaluated. The main reported findings are a substantial gap between the sound-engineer reference and the best submitted system, and strong correlations between FAD and the subjective metrics (0.94, 0.94, 0.77). The paper also discusses the decision to discontinue the task in 2025 due to cost and shifting research scope.","tokens_in":6415,"tokens_out":5661,"duration_ms":53052,"significance":"If the evaluation framework is valid, it provides a reusable protocol for comparing text-to-audio models in a constrained sound-scene setting. The manuscript has concrete strengths: it specifies the prompt structure, dataset sizes, rater blinding, self-rating removal, inter-rater agreement (Cronbach's alpha = 0.959), and it releases official evaluation software. The authors also explicitly acknowledge that the small number of systems limits the strength of the FAD-human correlation evidence. However, the paper's central claim that FAD is validated as a perceptual quality measure is threatened by the use of a single-engineer reference set and by the embedding-selection history in the authors' prior work. The paper also contains a numerical inconsistency in the headline performance-gap figure. These issues are load-bearing and require revision before the framework can be considered established.","major_comments":[{"comment":"The FAD reference set in Eq. (1) is built from audio created by a single sound engineer, apparently one recording per prompt (§3.1). For open-ended text-to-audio generation, many acoustically different recordings can legitimately satisfy the same caption, so FAD(r,g) penalizes any valid output that is far in embedding space from that engineer's specific rendition. This threatens the paper's interpretation of FAD as a perceptual quality measure and could change system rankings if a different engineer's recordings were used as the reference. The authors should test the stability of the reported correlations and rankings by constructing reference sets from multiple independent engineers (or by using a larger set of references per prompt) and should state this limitation explicitly in the paper.","section":"§3.1, §4.1, Eq. (1)"},{"comment":"The headline claim of a \"substantial 36% performance gap\" between the reference (8.793) and the best submitted system (5.832) is not supported by the numbers in Table 1: the relative gap is (8.793 − 5.832)/8.793 ≈ 33.7%, not 36%. Please either correct the percentage or explain the calculation; this number appears in the Abstract and Section 5.2 and is reported as a key result.","section":"§5.2, Table 1"},{"comment":"The reported FAD-human correlations (0.94, 0.94, 0.77) are computed on only five systems (four submissions plus the baseline). Moreover, the PANN-Wavegram-Logmel embedding used for FAD was chosen in the authors' prior work [8] specifically to maximize correlation with human perception. Because that prior selection is not independent of the present validation, the correlations should be interpreted with caution. The paper acknowledges the small sample but does not discuss this selection issue; please add a discussion of this limitation and, ideally, provide confidence intervals or a leave-one-system-out analysis to assess robustness.","section":"§5.2, §4.1"}],"minor_comments":[{"comment":"In the Introduction, \"motivated by the recent advances generative models\" is missing \"in\" after \"advances\"; it should read \"recent advances in generative models.\"","section":"§1"},{"comment":"In the sentence about Room Tone 1, there is a stray space before the period and \"sounds\" may be intended to be part of the category name; please rephrase for clarity (e.g., \"Room Tone 1 (labeled as 'Nothing') sounds\").","section":"§3.2"},{"comment":"The paper states that 24 evaluation captions were used for subjective rating but does not specify whether the FAD scores in Table 1 and Figure 1 were computed on those 24 captions or on the full 250-caption evaluation set; please clarify this to ensure the correlation analysis is interpretable.","section":"§5.1"},{"comment":"The phrase \"weak evidence\" for the FAD-human correlation is slightly misleading: with n=5, a correlation of 0.94 is large in magnitude but statistically fragile. Consider reporting exact p-values, confidence intervals, or a permutation test to make the strength of the evidence precise.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a challenge report whose main contribution is the evaluation framework and dataset. The reference-set assumption and the numerical error in the headline gap are issues that must be fixed before acceptance. The editor may also wish to consider whether the journal's scope is best served by a paper that is primarily a challenge summary, though the release of evaluation software and the detailed protocol are useful community resources."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a straightforward, honest challenge report. The new content is the 2024 Task 7 results—four submitted systems, the baseline, FAD scores, perceptual ratings, and the gap to a sound-engineer reference. The evaluation protocol is described concretely: caption structure, dataset sizes, rater blinding, self-rating removal, inter-rater agreement (alpha = 0.959). That kind of transparency is genuinely useful for anyone comparing text-to-audio systems.\n\nThe paper does not overclaim. It explicitly says the FAD-human correlation is weak evidence, given the small number of data points. It openly discusses why the task was discontinued, including time and cost. The references look appropriate, and the self-citation to the prior embedding-selection paper is relevant, not padding.\n\nWhere are the soft spots? Two, both real but not fatal.\n\nFirst, the headline FAD-human correlations (0.94, 0.94, 0.77) are computed on five systems. No confidence intervals. The paper acknowledges this in one sentence, but the figures without intervals can still mislead a casual reader. It is a minor reporting issue, not a hidden flaw.\n\nSecond, and to my mind the more significant assumption: the reference set is one sound engineer's recordings, one per prompt. For open-ended text-to-audio, a caption like \"dog barking with traffic in the background\" admits many acoustically valid renderings. FAD measures distance to that engineer's specific choices, so the objective score conflates quality with reference proximity. A different engineer could plausibly produce a different reference distribution and re-rank systems. The reader's stress-test note raises exactly this point, and I think it lands. The paper does not discuss this limitation in Section 4.1 or Section 5.2; it should have.\n\nStill, the central results—the 36% gap, the ranking of submissions, the participation drop—stand on their own. The framework is not invalid; it is just a reference-based benchmark, not a universal quality metric.\n\nWho is this for? Practitioners organizing or entering text-to-audio challenges, and researchers who want a compact snapshot of the 2024 state of the art. It is not a methods paper.\n\nMy recommendation: send it to peer review. It deserves a serious referee. The referee should push the authors to add confidence intervals for the correlations and to add a paragraph acknowledging the single-reference assumption and its effect on FAD interpretation. With that, it is a solid archival challenge report.","headline":"Honest, useful challenge report with clean evaluation protocol; the FAD-human correlation is real but rests on 5 points and a single engineered reference, and the paper itself flags the small sample.","tokens_in":6987,"tokens_out":648,"would_cite":true,"duration_ms":8500,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-to-sound-scene systems still trail expert recordings by 36 percent, and FAD tracks human judgment closely enough to serve as a proxy.","keywords":["sound scene synthesis","text-to-audio generation","DCASE 2024 challenge","Fréchet Audio Distance","perceptual evaluation","generative audio evaluation","sound scene benchmark","audio quality assessment"],"falsifier":"Have a second sound engineer record a fresh reference set for the same 250 captions, recompute FAD for the four submitted systems against both reference sets, and compare with the existing human ratings; if the FAD-based ranking or the 36 percent gap shifts while human ratings stay stable, the single-reference assumption is the source of the instability.","tokens_in":6010,"feed_emoji":"🔊","tokens_out":8065,"duration_ms":72136,"temperature":0.7,"pith_summary":"Text-to-audio systems can now synthesize environmental sound scenes, but there has been no agreed way to compare them. This paper reports the DCASE 2024 challenge Task 7, which evaluated four submitted systems against a fixed set of sound-engineer reference recordings using the Fréchet Audio Distance (FAD) and human perceptual ratings. The organizers found that the best system scores 36 percent below the reference on a weighted perceptual score, that FAD correlates with human foreground and background fit at $r=0.94$ and with audio quality at $r=0.77$, and that the manual evaluation cost made the challenge unsustainable to repeat. The paper's contribution is a reusable evaluation recipe and a quantified measure of how far synthetic sound scenes are from professional quality.","feed_headline":"Best text-to-sound system stays 36% below expert audio","feed_subtitle":"A new benchmark pairs Fréchet Audio Distance with human ratings to show how far generative sound still has to go.","key_machinery":"The central object is the Fréchet Audio Distance computed over PANN-Wavegram-Logmel embeddings, paired with a perceptual score built from Foreground Fit, Background Fit, and Audio Quality ratings: $\\text{Perceptual Score} = (2FF + BF + AQ)/4$. FAD compares the mean ($\\mu$) and covariance ($\\Sigma$) of the embedding distributions of the reference and generated audio sets, and the specific embedding was chosen because prior work showed that FAD's agreement with human perception depends on the embedding. The reference side of both measures is a 250-caption evaluation set recorded by a single sound engineer, and the weighted perceptual formula gives foreground accuracy the largest say in the final ranking.","core_discovery":"The paper's central claim is that a standardized framework combining FAD computed on PANN-Wavegram-Logmel embeddings with ratings from 14 expert listeners gives a stable, interpretable comparison of sound scene synthesis systems. Using that framework, the best submitted system reaches an average perceptual score of 5.832 against the sound engineer reference's 8.793, a gap of about 36 percent, while FAD values range from 35.985 for the best system to 53.728 for the lowest-ranked one. The objective and subjective measures agree strongly on foreground fit ($r=0.94$) and background fit ($r=0.94$) and less strongly on overall audio quality ($r=0.77$), which the paper treats as useful but weak evidence because only five systems were compared.","pith_inferences":["The strong FAD-human correlation suggests a testable two-stage benchmark design: use FAD to pre-screen many systems, then spend limited human rating effort only on the top FAD candidates.","The single-reference design is the fragile part of the framework, because for open-ended captions many acoustic realizations can legitimately fit the same text and would be penalized for departing from one engineer's choices.","The 36 percent gap might shrink or grow if the reference set were expanded to multiple engineers' recordings, and checking that sensitivity would clarify whether the gap reflects model weakness or reference idiosyncrasy.","The difficulty of sustaining annual human evaluation points toward automated or semi-automated proxies, but the paper's own $r=0.77$ correlation on audio quality warns that FAD alone would misrank systems when overall quality is the criterion."],"forward_implications":["FAD can serve as an inexpensive screen for future sound scene synthesis comparisons, since it tracks human foreground and background fit at $r=0.94$ across the systems tested.","The 36 percent gap between the best system and the reference quantifies the headroom remaining for generative sound models, giving later work a concrete improvement target.","The 'Foreground with Background in the background' caption structure and the separate FF, BF, and AQ ratings allow future evaluations to diagnose failure by scene layer rather than by overall quality alone.","The organizers' accounting of roughly 120 hours of expert effort plus platform and compute costs explains why the task was discontinued, implying that sustainable generative-audio benchmarks will need cheaper reference and rating protocols.","The drop from 32 submissions in the 2023 edition to 4 in 2024, paired with the removal of training-data constraints, suggests that evaluator overhead and reliance on large pre-existing models shape participation as much as synthesis skill."],"supporting_citations":[{"why":"Defines the Fréchet Audio Distance, the objective metric whose agreement with human ratings the paper computes.","marker":"[6]"},{"why":"Provides the PANN-Wavegram-Logmel embedding over which FAD is computed.","marker":"[7]"},{"why":"Shows FAD's perceptual agreement is embedding-dependent, motivating the choice of the PANN embedding.","marker":"[8]"},{"why":"Supplies AudioLDM, the baseline synthesis engine whose 57.061 FAD anchors the bottom of the comparison.","marker":"[3]"},{"why":"Reports the 2023 foley challenge, the previous edition whose format and 32-submission participation this paper uses as a comparison point.","marker":"[2]"},{"why":"Technical report of the top-ranked submitted system, whose 5.832 perceptual score and 35.985 FAD define the best-system gap to the reference.","marker":"[9]"},{"why":"Technical report of the second-ranked submitted system, contributing one of the four data points in the FAD-human correlation.","marker":"[10]"},{"why":"Technical report of the third-ranked submitted system, contributing one of the four data points in the FAD-human correlation.","marker":"[11]"},{"why":"Technical report of the fourth-ranked submitted system, contributing one of the four data points in the FAD-human correlation.","marker":"[12]"}],"fun_headline_variants":["New audio benchmark shows best text-to-sound still 36% off","Sound synthesis benchmark: FAD and human ears align, gap remains","Text-to-sound systems lag experts by 36% in new DCASE test","DCASE 2024 sound benchmark exposes 36% gap to expert audio","Best sound synth still 36% below expert, says new FAD-human benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a single set of reference recordings made by one sound engineer is the correct target for each caption, so a system is judged by how close it comes to that one distribution.","fun_headline_variants_meta":{"raw":{"variants":["New audio benchmark shows best text-to-sound still 36% off","Sound synthesis benchmark: FAD and human ears align, gap remains","Text-to-sound systems lag experts by 36% in new DCASE test","DCASE 2024 sound benchmark exposes 36% gap to expert audio","Best sound synth still 36% below expert, says new FAD-human benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1470,"prompt_tokens":790,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":580}},"tokens_in":406,"tokens_out":680,"duration_ms":6144,"temperature":1.0,"reasoning_tokens":580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:22:27.718587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second sound engineer record a fresh reference set for the same 250 captions, recompute FAD for the four submitted systems against both reference sets, and compare with the existing human ratings; if the FAD-based ranking or the 36 percent gap shifts while human ratings stay stable, the single-reference assumption is the source of the instability.","supporting_citations":[{"cited_title":"First, the generative aspect of organizing this challenge has been costly and labor intensive","cited_arxiv_id":null,"evidence_quote":"Defines the Fréchet Audio Distance, the objective metric whose agreement with human ratings the paper computes."},{"cited_title":"While the submit- ted systems demonstrated promising capabilities, the significant gap between synthetic and reference audio quality indicates substantial room for improvement","cited_arxiv_id":null,"evidence_quote":"Provides the PANN-Wavegram-Logmel embedding over which FAD is computed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows FAD's perceptual agreement is embedding-dependent, motivating the choice of the PANN embedding."},{"cited_title":"Sound Scene Synthesis at the DCASE 2024 Challenge","cited_arxiv_id":"2501.08587","evidence_quote":"Supplies AudioLDM, the baseline synthesis engine whose 57.061 FAD anchors the bottom of the comparison."},{"cited_title":"This is a more flexible setup than the category- based generation used in the last year [2]","cited_arxiv_id":null,"evidence_quote":"Reports the 2023 foley challenge, the previous edition whose format and 32-submission participation this paper uses as a comparison point."},{"cited_title":"A Proposal for Foley Sound Synthesis Challenge","cited_arxiv_id":"2207.10760","evidence_quote":"Technical report of the top-ranked submitted system, whose 5.832 perceptual score and 35.985 FAD define the best-system gap to the reference."},{"cited_title":"Foley sound synthesis at the dcase 2023 challenge,","cited_arxiv_id":null,"evidence_quote":"Technical report of the second-ranked submitted system, contributing one of the four data points in the FAD-human correlation."},{"cited_title":"Audiocaps: Gen- erating captions for audios in the wild,","cited_arxiv_id":null,"evidence_quote":"Technical report of the fourth-ranked submitted system, contributing one of the four data points in the FAD-human correlation."}],"review_version":1}