{"id":"81d1c675-3aee-4656-8734-aea587acef98","arxiv_id":"2507.00475","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AudioBERTScore, a training-free metric combining max-norm and p-norm similarity of audio embeddings, correlates more strongly with human subjective scores for text-to-audio synthesis than conventional metrics.","lead":"This paper introduces AudioBERTScore, a metric that scores synthesized environmental audio by comparing frame-level embeddings of generated and reference sounds. It reports higher correlation with human ratings of quality and text relevance than conventional objective metrics such as MCD and WARP-Q.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-sample hyperparameter selection and post-hoc data exclusion make the reported correlations unverified; a held-out evaluation is needed.","rationale":"The reader's weakest assumption was that the hyperparameters p=10^6 and λ=−3.5 selected on the PAM test set generalize. My stress test identifies the same root cause: the test set is used for both hyperparameter selection and final evaluation, which makes the Table 3 correlations optimistically biased. I additionally flag the post-hoc exclusion of low-REL samples and the absence of significance testing, both of which compound the concern. However, these issues do not overturn the reader's CONDITIONAL verdict—they strengthen the condition that a held-out or cross-validated evaluation is required before accepting the headline numbers. The proposed method's simpler variant (F1max with AST) shows a correlation advantage over MCD and WARP-Q that may survive honest evaluation, so the paper is not reject-worthy, but the specific contribution of the p-norm and negative λ is unverified. I therefore recommend no change to the reader's verdict: CONDITIONAL, pending a proper held-out evaluation. My agreement is 'agree' because the main load-bearing concern is the test-set tuning, as the reader identified; the additional issues are consequences of the same evaluation-protocol weakness.","tokens_in":8109,"tokens_out":5460,"duration_ms":61067,"concrete_test":"Split the PAM test set (after the exclusion) into a tuning fold and a held-out evaluation fold (e.g., 50/50). Select p and λ using only the tuning fold, then compute LCC/SRCC on the held-out fold, repeating over several random splits to obtain means and 95% confidence intervals. Also repeat the entire protocol without the REL<3.5 exclusion. If the held-out correlations for the tuned AudioBERTScore no longer exceed F1max and WARP-Q by a margin larger than the confidence interval (e.g., >0.05), the central claim is overfit; if they remain clearly higher, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that AudioBERTScore significantly outperforms MCD and WARP-Q—rests on Table 3, but the numbers in that table are not unbiased estimates. In Section 4.2.2, the hyperparameters p=10^6 and λ=−3.5 are explicitly selected by maximizing correlation on the PAM test set (Figures 5–6), and Table 3 then reports correlations on the same test set. The test set therefore serves as both the tuning and evaluation set, inflating the reported correlations to an unknown degree. The post-hoc exclusion in Section 4.1 of 17 reference samples with REL<3.5 and the corresponding 68 synthesized samples further reshapes the test distribution; if the metric performs poorly on low-REL samples, this exclusion directly inflates the REL correlations. Additionally, no significance tests or confidence intervals are reported, so the word 'significantly' is unsupported: the gain over F1max is only 0.056 LCC for OVL and 0.056 for REL, and the claimed advantage over PAM on REL is 0.017 LCC—likely within noise. The underlying idea of a reference-based, training-free embedding similarity metric is plausible, and the simpler F1max variant may still beat the baselines, but the specific numbers in Table 3 do not provide a reliable estimate of performance on new TTA outputs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AudioBERTScore, a reference-based, training-free objective metric for evaluating text-to-audio (TTA) synthesis. It extracts frame-level embeddings from synthesized and reference audio using pretrained models (AST, ATST-Frame, BYOL-A), computes a cosine similarity matrix, and aggregates it via BERTScore-style max-norm precision/recall/F1, extended with a p-norm and an interpolation weight lambda. Experiments on the PAM test set report LCC and SRCC against human OVL and REL scores, claiming that the best configuration (AST layer 13, p=10^6, lambda=-3.5) outperforms MCD, WARP-Q, and CLAPScore, and is competitive with PAM. The paper also releases code on GitHub.","tokens_in":8395,"tokens_out":7199,"duration_ms":73755,"significance":"If the reported correlations were unbiased, this would be a useful contribution: it offers a training-free, language-independent, reference-based metric that can be applied to arbitrary TTA outputs, and the open-source implementation is a definite strength. The p-norm aggregation is a reasonable extension of BERTScore-style metrics to non-local environmental sounds. However, the empirical evaluation as reported is not yet convincing because the hyperparameters and feature-extractor layer are selected on the same test set used for the final comparison, and the post-hoc exclusion of low-REL samples further compromises the estimates. The underlying method is plausible, but the specific numerical claims in Table 3 should be treated as development-set results until a held-out evaluation is provided.","major_comments":[{"comment":"The final configuration (AST 13th layer, p=10^6, lambda=-3.5) is selected by maximizing correlation on the PAM test set in Figures 3, 5, and 6, and Table 3 then reports correlations computed on exactly that same test set. The reported LCC/SRCC values are therefore in-sample estimates, and the magnitude of selection-induced inflation is unknown. Because the central claim of Section 4.2.3 rests on these numbers, the authors should provide a held-out evaluation (e.g., a separate tuning split, cross-validation, or a second dataset) and report the performance of the chosen configuration on data not used for any selection.","section":"4.2.2 / Table 3"},{"comment":"The exclusion of 17 reference samples with REL < 3.5 and their 68 synthesized samples is post-hoc: it is motivated by the proposed method's aim rather than by a pre-specified criterion, and it changes the test distribution in a way that can inflate REL correlations if low-REL items are hard cases. Please report results on the full 100-reference/400-synthesized test set as well, or justify the exclusion with external evidence and show that the conclusions are robust to the exclusion.","section":"4.1"},{"comment":"Equation (7) defines lambda in [0,1], yet Section 4.2.2 explores negative lambda and the final configuration uses lambda=-3.5. This is an extrapolation outside the defined interpolation range, and the paper gives no formal interpretation or justification for this extrapolation. Since this value is central to the reported gains, the authors should either extend the model definition to lambda in R with a clear interpretation, or restrict the final configuration to the originally defined range.","section":"Eq. (7) / Section 4.2.2"},{"comment":"The claim that the proposed method 'significantly outperformed' MCD and WARP-Q is not supported by significance tests or confidence intervals. The differences between the proposed variants and PAM are small (e.g., REL LCC 0.546 vs. 0.529, REL SRCC 0.567 vs. 0.556), and Figures 5 and 6 have no error bars. Add bootstrap or permutation-based significance tests and confidence intervals for the LCC/SRCC comparisons, or soften the claim accordingly.","section":"4.2.3"}],"minor_comments":[{"comment":"The x-axis labels in Figure 6 appear as '5 4 3 2 1 0' with no minus signs; since the text describes negative lambda values, the labels should be '-5 -4 -3 -2 -1 0'.","section":"Figure 6"},{"comment":"The statement 'Performance peaked at p = 100' is immediately followed by 'p = 10^6 slightly outperformed p = 100', which is contradictory; please clarify that p=10^6 was evaluated outside the range shown in Figure 5, and ensure the superscript notation is rendered correctly.","section":"4.2.2"},{"comment":"Citation numbers for MCD and WARP-Q are inconsistent: MCD appears as [4] and [8], and WARP-Q appears as [9] and [24]; please use one consistent set of references.","section":"Table 1 / Section 2.1"},{"comment":"The heading 'Correlation between correlation scores of OVL-REL' should be rephrased for readability, e.g., 'Relationship between OVL and REL correlations'.","section":"4.2.1"},{"comment":"The caption 'The score calculation based on the maximum value and the features extracted use the best settings' is ungrammatical; rephrase to something like 'Each row uses the best-performing layer and feature setting for that extractor.'","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid and lands directly on the central claim: Table 3 should be regarded as a development-set result unless a properly held-out evaluation is added. The post-hoc exclusion and the lack of significance testing compound the problem. I would be willing to review a revised version that addresses these issues. The paper is within scope for the DCASE workshop and the underlying idea is worth publishing if the empirical claims can be made unbiased."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is a plausible extension of BERTScore to environmental sound, but the headline numbers in Table 3 are not trustworthy as estimates of real performance, because the authors tuned p and lambda on the PAM test set and then report correlations on that same set. The stress-test note is right.\n\nWhat's genuinely new: replacing the max-norm with a p-norm interpolation (and allowing negative lambda) is a small but real variant; the non-locality motivation for environmental sounds makes sense. The paper also open-sources the code and compares fairly against MCD and WARP-Q. The simpler F1max variant with AST layer 13 already beats both baselines (OVL LCC 0.368 vs 0.241; REL LCC 0.490 vs 0.202), so the core idea has independent support even before the p-norm tweaks.\n\nThe soft spots are load-bearing for the central claim, though not for the whole method. First, Section 4.2.2 explicitly selects p=10^6 and lambda=-3.5 by maximizing correlation on the PAM test set, and Table 3 then reports correlations on that same set. That makes the reported LCCs optimistic to an unknown degree. Second, Section 4.1 excludes 17 reference samples with REL < 3.5 and their 68 synthesized counterparts; that post-hoc filtering based on subjective scores reshapes the distribution and likely inflates REL correlations. Third, the phrase \"significantly outperformed\" is unsupported—there are no significance tests or confidence intervals, and the gain over F1max is only 0.056 LCC on both OVL and REL, while the claimed edge over PAM on REL is 0.017. Fourth, and oddly, Table 3's exact configuration p=10^6, lambda=-3.5 is not actually shown in Figures 5 or 6; Figure 6 only sweeps lambda for p=100, so the reader cannot see the reported result.\n\nNone of this kills the idea. A held-out or cross-validated evaluation, plus a significance test, could easily salvage it. The max-only variant stands on its own.\n\nWho this is for: people working on TTA evaluation or on reference-based metrics using audio foundation models. It is worth a serious referee, but only with revisions: add validation, fix the figure/table mismatch, and drop the word \"significant\" until it is earned.\n\nRecommendation: send to peer review with a request for major revision.","headline":"A plausible training-free TTA evaluation metric, but the headline correlations are unverified because hyperparameters were tuned on the same test set and the exact reported configuration is not shown.","tokens_in":8905,"tokens_out":3450,"would_cite":false,"duration_ms":36207,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-free audio metric matches human ratings better than MCD and WARP-Q","keywords":["AudioBERTScore","text-to-audio","objective evaluation","semantic similarity","p-norm","environmental sound","audio embeddings","subjective correlation"],"falsifier":"Take any held-out set of text-to-audio synthesized sounds with human OVL and REL ratings, compute AudioBERTScore with the published settings, and compare its LCC/SRCC against MCD and WARP-Q. If the gap shrinks to statistical non-significance, the PAM-tuned hyperparameters are overfit; a positive control would be showing the same ranking of extractors and norms on a second dataset.","tokens_in":7949,"feed_emoji":"🎧","tokens_out":4809,"duration_ms":45919,"temperature":0.7,"pith_summary":"The paper aims to fix the weak correlation between existing objective metrics and human judgments when evaluating synthesized environmental audio. It proposes AudioBERTScore, a training-free metric that compares sequences of audio embeddings from a pretrained audio model and combines a conventional max-norm BERTScore with a p-norm similarity to capture both localized and spread-out sound structure. On the PAM test set, the proposed score correlates more strongly with human overall quality (OVL) and text relevance (REL) ratings than MCD and WARP-Q. The practical payoff is a cheap automatic proxy for subjective listening tests in text-to-audio development.","feed_headline":"AudioBERTScore tops classic metrics in matching human audio ratings","feed_subtitle":"Training-free metric uses audio embeddings and a p-norm similarity to reach higher correlation with human OVL and REL scores.","key_machinery":"The central object is the cross-embedding similarity matrix $M_{ij}=\\mathrm{sim}(\\tilde{s}_i,\\tilde{r}_j)$ between frame-level embeddings of synthesized and reference audio. Precision, recall, and F1 are defined by averaging row-wise and column-wise p-norms, which interpolate between a pure average ($p=1$) and the max-norm ($p\\to\\infty$); the final score is an affine blend $\\lambda\\cdot\\text{max}+(1-\\lambda)\\cdot\\text{p-norm}$ that the paper allows to go negative. The feature extractor is a pretrained audio Transformer, with AST layer 13 chosen empirically, and the p-norm hyperparameter pair $p=10^6,\\ \\lambda=-3.5$ is selected by maximizing correlation on the PAM test set. The max-norm component inherits BERTScore's locality assumption, while the p-norm component is designed to reflect non-local environmental sound structure.","core_discovery":"The central claim is that replacing the max-norm in BERTScore with an interpolation between max-norm and p-norm, allowing a negative interpolation weight, produces an objective audio evaluation metric whose scores track human OVL and REL judgments better than existing unsupervised reference-based metrics. Using the 13th layer of an AudioSet-fine-tuned AST model as the feature extractor and $p=10^6$, $\\lambda=-3.5$, AudioBERTScore attains LCC/SRCC of 0.424/0.433 for OVL and 0.546/0.567 for REL, while MCD correlates near zero and WARP-Q around 0.2. In the REL comparison it also exceeds CLAPScore, and it trails PAM only on OVL. The authors interpret the improvement as evidence that environmental sounds have non-local structure that max-only similarity misses.","pith_inferences":["A natural next test is whether the same $p$ and $\\lambda$ values transfer to TTA models beyond the four generators in the PAM test set; the paper does not report cross-set validation.","The negative $\\lambda$ implies the max-norm component is being subtracted in the blend, which is an unusual extrapolation; a per-category breakdown (impulsive vs. continuous sounds) could clarify why this helps.","Combining AudioBERTScore with a reference-free method like PAM may yield a metric that retains high OVL correlation while improving REL correlation, a direction the authors flag as future work."],"forward_implications":["AudioBERTScore can serve as a drop-in, training-free replacement for MCD and WARP-Q when ranking or monitoring text-to-audio systems.","The p-norm extension makes the BERTScore family applicable to sounds whose relevant structure is distributed over time, not just localized events.","Pretrained Transformer embeddings fine-tuned on environmental audio (AST) give markedly better correlation than CNN-based embeddings (BYOL-A).","Because the score uses only audio references, it inherits the language independence of MCD and WARP-Q while improving correlation with subjective judgments."],"supporting_citations":[{"why":"BERTScore: the text metric whose max-norm similarity AudioBERTScore extends.","marker":"[13]"},{"why":"SpeechBERTScore: the prior adaptation of BERTScore to speech that motivates using audio embeddings.","marker":"[12]"},{"why":"ATST-Frame: one of the pretrained audio feature extractors compared in the paper.","marker":"[6]"},{"why":"AST: the audio spectrogram Transformer whose layer-13 features give the best correlation.","marker":"[23]"},{"why":"PAM: provides the test set of synthesized/reference pairs with OVL and REL subjective scores and is a comparison method.","marker":"[11]"},{"why":"AudioCaps: source of the natural reference audio used in the PAM test set.","marker":"[16]"},{"why":"MCD: the conventional mel-cepstral distortion baseline used for comparison.","marker":"[4]"},{"why":"WARP-Q: the conventional reference-based quality baseline used for comparison.","marker":"[24]"}],"fun_headline_variants":["AudioBERTScore beats classic metrics on human audio ratings","p-norm twist helps AudioBERTScore track human sound judgments","New metric outperforms MCD and WARP-Q in audio evaluation","AudioBERTScore: better correlation with human scores using p-norm","Training-free audio metric improves on BERTScore via p-norm"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage rests on the hyperparameters $p=10^6$, $\\lambda=-3.5$, and AST layer 13 being tuned on the PAM test set itself; if these choices do not transfer to other text-to-audio outputs, the measured correlations are not an unbiased estimate of metric quality.","fun_headline_variants_meta":{"raw":{"variants":["AudioBERTScore beats classic metrics on human audio ratings","p-norm twist helps AudioBERTScore track human sound judgments","New metric outperforms MCD and WARP-Q in audio evaluation","AudioBERTScore: better correlation with human scores using p-norm","Training-free audio metric improves on BERTScore via p-norm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000116,"raw_usage":{"total_tokens":1031,"prompt_tokens":856,"completion_tokens":175,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":88}},"tokens_in":472,"tokens_out":175,"duration_ms":2519,"temperature":1.0,"reasoning_tokens":88,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:14:17.423869+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any held-out set of text-to-audio synthesized sounds with human OVL and REL ratings, compute AudioBERTScore with the published settings, and compare its LCC/SRCC against MCD and WARP-Q. If the gap shrinks to statistical non-significance, the PAM-tuned hyperparameters are overfit; a positive control would be showing the same ranking of extractors and norms on a second dataset.","supporting_citations":[{"cited_title":"BERTScore: Evaluating text generation with bert,","cited_arxiv_id":null,"evidence_quote":"BERTScore: the text metric whose max-norm similarity AudioBERTScore extends."},{"cited_title":"Speech- BERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,","cited_arxiv_id":null,"evidence_quote":"SpeechBERTScore: the prior adaptation of BERTScore to speech that motivates using audio embeddings."},{"cited_title":"Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks","cited_arxiv_id":"2306.04186","evidence_quote":"ATST-Frame: one of the pretrained audio feature extractors compared in the paper."},{"cited_title":"AST: Audio spectrogram trans- former,","cited_arxiv_id":null,"evidence_quote":"AST: the audio spectrogram Transformer whose layer-13 features give the best correlation."},{"cited_title":"AudioCaps: Generating captions for audios in the wild,","cited_arxiv_id":null,"evidence_quote":"AudioCaps: source of the natural reference audio used in the PAM test set."},{"cited_title":"Mel-cepstral distance measure for objective speech quality assessment,","cited_arxiv_id":null,"evidence_quote":"MCD: the conventional mel-cepstral distortion baseline used for comparison."},{"cited_title":"W ARP-Q: Quality prediction for generative neural speech codecs,","cited_arxiv_id":null,"evidence_quote":"WARP-Q: the conventional reference-based quality baseline used for comparison."}],"review_version":1}