{"id":"804a6e39-26e8-451d-b5e9-470e747a92c5","arxiv_id":"2412.01356","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Soft audio-caption relevance labels estimated from caption text similarity improve text-based audio retrieval over binary-labels contrastive training.","lead":"This paper trains audio retrieval systems with soft relevance scores instead of binary positive and negative pairs, deriving the scores from how similar captions are to each other. The approach improves retrieval accuracy on two standard audio benchmarks, most clearly on the Clotho dataset and when combined with large-scale pretraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The relevance scores depend entirely on an untested transitivity assumption; a same-audio caption similarity check on AudioCaps/Clotho would settle whether the premise holds.","rationale":"The reader identifies the same load-bearing concern: the paper's entire relevance scoring mechanism rests on the unvalidated claim that caption textual similarity implies audio-content relevance. My stress-test agrees and sharpens the concern by pointing to the specific, testable consequence: if the transitivity premise holds, captions of the same audio should be textually more similar to each other than to captions of different audios, and the logistic mapping in Eq. (5) should be calibrated consistently across datasets. The proposed check is concrete and inexpensive, using existing validation/test sets with multiple captions per audio, and it directly tests the foundational assumption rather than the downstream retrieval numbers. I do not see an additional weakness that would change the verdict: the empirical results are honestly reported (including the non-significant AudioCaps result without pretraining), the comparison against InfoNCE appears to isolate the soft-label effect under the authors' stated equivalence, and the paper gives credit to prior crowdsourcing work for the logistic mapping. Therefore the appropriate verdict remains CONDITIONAL on addressing the proxy assumption and providing reproducibility artifacts, which is exactly the reader's verdict. Hence UNCHANGED.","tokens_in":8945,"tokens_out":5430,"duration_ms":48682,"concrete_test":"Compute, on the AudioCaps/Clotho validation or test sets where each audio has five captions, the Sentence-BERT cosine similarity h within captions of the same audio vs. across different audios. If the premise holds, the within-audio similarity must be substantially higher than the cross-audio similarity, and the logistic mapping Eq. (5) applied to the within-audio mean must yield high relevance (e.g., >0.5) while cross-audio yields low (<0.2). If the distributions overlap or the within-audio similarity maps below the relevance threshold, the transitivity assumption fails to support the method. A second check is to compare g(xi,yj) with human relevance judgments on the held-out Clotho subset from [14] to see whether Eq. (5) is calibrated beyond the fitting subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states: 'audio captions exhibiting high mutual textual similarity can be appropriate descriptions of each other's audio contents.' All computed relevances g(xi,yj) in Eq. (1) derive from this premise via the caption similarity h(xi,xj), and the logistic mapping in Eq. (5) is calibrated on a Clotho crowdsourcing subset from [14]. The premise is never validated against audio content, and the calibrated mapping is transferred to AudioCaps and WavCaps without checking whether caption-similarity distributions match. If the premise fails, the training targets are not audio-caption relevances but pure text-text similarities, and the observed retrieval gains could stem from soft-label regularization rather than from recovering true relevance. The paper's central explanation—that textual similarity recovers cross-modal relevance—would then be unsupported even if the empirical improvement is real. This is load-bearing because the method's novelty is precisely the use of caption similarities as a proxy for audio relevance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for text-based audio retrieval in which training relevance labels are computed as non-binary scores from the textual similarity between captions. Sentence-BERT embeddings are used to measure caption similarity, a logistic function (Eq. 5) converts these similarities into audio-caption relevance scores, and a dual-encoder model is trained with a ListNet listwise ranking objective (Eqs. 2-4). Experiments on AudioCaps and Clotho, with and without WavCaps pretraining, compare the proposed 'ListNet audio' approach against an InfoNCE binary-relevance baseline. The paper reports significant gains on Clotho (mAP@10 30.4 vs 28.2) and significant gains on both datasets after pretraining, while the AudioCaps gain without pretraining is not significant. The method is also adapted to audio-based text retrieval.","tokens_in":9094,"tokens_out":4663,"duration_ms":41600,"significance":"If the central proxy assumption is valid, the method is significant because it provides a cheap, scalable way to generate soft relevance labels for audio-text retrieval without expensive human annotation. The experimental protocol is a strength: five independent runs, paired t-tests, and a pretraining/fine-tuning scenario are all included, and the supplementary details of the dual-encoder architecture and training configuration are clearly stated. The main weakness is that the load-bearing premise—that high textual similarity between captions implies cross-relevance of their paired audio clips—is asserted in Section III-A but never validated against audio content. The paper also over-generalizes its conclusions given the non-significant AudioCaps result without pretraining. With additional validation and more careful claims, the method would be a useful contribution to audio-language learning.","major_comments":[{"comment":"The entire method rests on the premise stated in Section III-A: 'audio captions exhibiting high mutual textual similarity can be appropriate descriptions of each other's audio contents.' This premise is never tested against audio content. The computed relevance g(xi, yj) is purely a function of the text-text similarity h(xi, xj), so if two captions are textually similar but describe different audio events, the training labels are wrong. I ask the authors to validate the premise directly: for example, on AudioCaps and Clotho, which have multiple captions per audio clip, one can test whether captions of the same audio have significantly higher h(xi, xj) than captions of different audio clips, or have human annotators judge a sample of the computed relevance scores. Without such evidence, the observed retrieval gains could be due to soft-label regularization rather than to the recovery of true audio-caption relevance, which is the paper's claimed mechanism.","section":"Section III-A, Eq. (1), Eq. (5)"},{"comment":"The paper's conclusions overstate the empirical support. The paired t-test reported in Section V-B shows that ListNetaudio improves significantly on Clotho (t(5224)=11.658, p<0.001), but on AudioCaps the improvement is not significant (t(4699)=0.532, p>0.05). Nevertheless, Section VI states that 'experimental results validated the effectiveness of the proposed method in both text-based audio retrieval and audio-based text retrieval.' The audio-based text retrieval results in Table II are not significance-tested at all, and the ListNetaudio+text comparison against InfoNCE on AudioCaps (54.7 vs 54.5) is not significance-tested. The supported claims are limited to Clotho without pretraining and to both datasets with pretraining; the conclusions should be revised accordingly, and significance tests should be provided for all central comparisons.","section":"Section V-B, Table II, Section VI"},{"comment":"The logistic function in Eq. (5) was fitted in the authors' prior crowdsourcing study [14] on a limited Clotho subset, and the paper transfers this mapping to AudioCaps and WavCaps without checking whether the distribution of caption similarities h(xi, xj) is similar across datasets. If the similarity distribution differs, the relevance scores will be miscalibrated. The paper should report the distribution of h(xi, xj) for each dataset or otherwise justify the transferability of the fitted parameters. This is load-bearing for the claim that the method generalizes beyond the dataset on which the mapping was calibrated.","section":"Section V-A, Eq. (5)"}],"minor_comments":[{"comment":"The notation uses Y as if it were the full training set, but the experimental setup in Section IV-B uses mini-batches of 32 audio-caption pairs. Please clarify that Y in Eqs. (2)-(4) denotes the mini-batch during training.","section":"Section III-B, Eqs. (2)-(4)"},{"comment":"The sentence 'ListNetaudio+text utilizing computed non-binary relevances outperforms InfoNCE' is not supported by significance testing; on AudioCaps the difference is 54.7 vs 54.5 with overlapping standard deviations. Please either add significance tests or soften the claim.","section":"Section V-B"},{"comment":"The logistic mapping gives g(xi, yi) = 0.864 for the annotated pair (since h(xi, xi)=1), so the diagonal is not assigned relevance 1.0. This is presumably intentional because human relevance ratings may be below 1, but it should be explicitly stated and justified.","section":"Section V-A, Eq. (5)"},{"comment":"The evaluation metrics are described in terms of 'relevant items' but it is not explicitly stated that, at test time, the ground truth is binary relevance defined by the annotated caption(s). Please make this explicit.","section":"Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the unvalidated transitivity assumption is valid and is the primary reason for major revision. The paper would be substantially strengthened by a direct validation of the caption-similarity-to-audio-relevance premise, and the empirical claims should be aligned with the significance tests. The manuscript is within the scope of the journal and the experimental methodology is otherwise solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Xie et al. propose replacing binary audio-caption relevance with soft labels computed from caption-caption textual similarity: g(xi,yj) is a logistic transform of Sentence-BERT cosine similarity between captions, and training uses a ListNet ranking objective. That specific estimation route is new for audio retrieval, and the paper is clear about what it does. The experiments are properly repeated (five runs, paired t-tests), and the main comparisons are meaningful: ListNet audio beats InfoNCE on Clotho text-based retrieval (mAP@10 30.4 vs 28.2, p<0.001), and after WavCaps pretraining it beats InfoNCE on both AudioCaps and Clotho. The gains are modest, one to two mAP@10 points, but they are credible and consistent with the motivation about false negatives in binary contrastive training.\n\nCredit where due: the paper states its proxy assumption explicitly, uses a principled listwise objective, and reports the nonsignificant AudioCaps result rather than hiding it in the main table. The ablation against min-max scaling for f is also decent.\n\nThe soft spots are real, and the main one is load-bearing. Section III-A says captions with high textual similarity can describe each other's audio content. All computed relevances rest on that transitivity, and the paper never validates it by checking, say, whether high-similarity captions from different audio samples actually share acoustic content. The logistic mapping in Eq. (5) comes from the authors' earlier crowdsourced Clotho ratings, and its transfer to AudioCaps and WavCaps is assumed; caption-similarity distributions may differ. If the premise is wrong, the training targets are text-text similarities, not audio-caption relevances, and the gain could be soft-label regularization rather than recovered relevance. That would not make the empirical result useless, but it would change what the paper claims.\n\nTwo smaller issues. The AudioCaps gain without pretraining is not significant (t=0.532, p>0.05), and the abstract does not qualify this; read strictly, the abstract overstates. And the paper does not compare against [16], the estimated-correspondence baseline from the same DCASE line, which is the natural competing way to get non-binary relevance; omitting it weakens the claim that caption similarity is the useful signal. No code or data are provided, which matters more because the relevance labels are cheap to recompute.\n\nWho is this for: people working on audio-text retrieval and cross-modal soft-label training. It is a solid workshop-to-journal-level contribution, not a breakthrough. I would send it to peer review, conditional on a same-audio caption-similarity validation and a comparison with [16]. The central idea deserves referee time even though the effect size is modest.","headline":"A solid, honest paper showing a modest but real gain from soft relevance labels in audio retrieval, with one load-bearing proxy assumption left unvalidated.","tokens_in":9639,"tokens_out":2519,"would_cite":true,"duration_ms":21154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that graded audio-caption relevance scores computed from the textual similarity of captions, trained with a listwise ranking objective, improve text-based audio retrieval over binary-relevance InfoNCE training.","keywords":["text-based audio retrieval","audio-caption relevance","non-binary relevance","textual similarity","listwise ranking","sentence embeddings","contrastive learning","audio-language retrieval"],"falsifier":"A reader could test this by taking a held-out set of audio-caption pairs with human graded relevance, computing the paper's text-similarity-based scores, and checking whether they predict the human ratings; if they do not, or if replacing the computed scores with random soft labels leaves retrieval performance unchanged, the central claim would be contradicted.","tokens_in":8730,"feed_emoji":"🎧","tokens_out":8430,"duration_ms":66325,"temperature":0.7,"pith_summary":"The paper claims that text-based audio retrieval is best trained with graded, non-binary relevance scores rather than the binary matched/not-matched labels used by contrastive methods such as CLAP. It derives these scores from caption text: captions with high Sentence-BERT cosine similarity are assumed to describe similar audio, so each audio sample is treated as partially relevant to captions other than its own. A dual-encoder model is trained with a listwise ranking objective, ListNet, that compares the full predicted relevance list to these computed scores. On Clotho this raises mAP@10 from 28.2 to 30.4, and after pretraining on WavCaps the gains on both Clotho and AudioCaps are statistically significant; the AudioCaps gain without pretraining is small and not significant.","feed_headline":"Score audio relevance from caption text, beating binary labels","feed_subtitle":"Listwise training on soft relevance scores lifts retrieval on Clotho and AudioCaps.","key_machinery":"The central mechanism is caption-to-caption textual similarity used as a proxy for audio-to-caption relevance. Sentence-BERT maps each caption to an embedding, cosine similarity gives $h(x_i,x_j)$, and the logistic function in Eq. (5) converts that similarity into a graded relevance score $g(x_i,y_j)$ in $[0,1]$, so an audio sample inherits partial relevance to every caption whose text resembles its own caption. The ListNet objective then turns the target scores and the model's predicted cosine-similarity scores into two softmax distributions over audio samples and minimizes their cross-entropy, replacing InfoNCE's binary positive/negative sampling with a full-list ranking comparison.","core_discovery":"The central claim is that relevance between audio and arbitrary captions can be inferred from caption text alone, without additional human annotation. For a caption pair $(x_i,x_j)$, the paper computes the cosine similarity $h(x_i,x_j)$ of their Sentence-BERT embeddings and maps it through the logistic function $g(x_i,y_j)=1/(1+e^{2.73-4.58\\,h(x_i,x_j)})$, giving each audio sample a graded relevance to every caption in the training set. The dual encoder is then trained by minimizing the cross-entropy between the softmax distribution of these target scores and the softmax distribution of the model's predicted scores, a listwise ranking objective. In experiments this yields mAP@10 gains on Clotho (30.4 vs. 28.2) and AudioCaps (55.0 vs. 54.5) without pretraining, and significant gains on both datasets when the model is first pretrained on WavCaps; the AudioCaps-only gain without pretraining is not statistically significant.","pith_inferences":["If the transitivity-through-text assumption holds at scale, soft relevance labels could be mined cheaply from weakly labeled or synthetic caption sets, reducing the annotation cost of audio-language pretraining.","The lack of a significant AudioCaps gain without pretraining hints that the benefit depends on dataset properties such as caption redundancy and audio diversity; a controlled study varying those properties could test that.","A symmetric test using audio-side similarity from a pretrained audio encoder to generate graded relevance labels would isolate where the textual transitivity assumption starts to fail.","One practical extension is to calibrate the logistic mapping separately for each dataset or caption distribution rather than transferring a curve fitted on a Clotho subset."],"forward_implications":["Existing audio-caption datasets can be used to train retrieval with graded relevance without collecting new human relevance judgments, because the relevance scores are derived from caption text.","The same training recipe improves audio-based text retrieval, ranking captions for a given audio query, on both AudioCaps and Clotho.","The method is model-agnostic and can be applied to any captioning dataset, including image-caption data, for cross-modal retrieval.","After large-scale pretraining on WavCaps, graded-relevance training with ListNet yields statistically significant gains over the InfoNCE baseline on both AudioCaps and Clotho.","InfoNCE can be seen as a special case of the listwise objective in which the target distribution puts all probability mass on one sample, which helps explain why the graded target distribution is the source of the improvement."],"supporting_citations":[{"why":"Provides Sentence-BERT embeddings used to compute caption textual similarity h(x_i,x_j).","marker":"[11]"},{"why":"Defines the ListNet listwise ranking objective used to train with graded relevance scores.","marker":"[17]"},{"why":"Supplies crowdsourced graded audio-caption relevance ratings on a Clotho subset, used to fit the logistic mapping in Eq. (5).","marker":"[14]"},{"why":"InfoNCE is the binary-relevance contrastive baseline that the proposed method is compared against.","marker":"[27]"},{"why":"Provides the dual-encoder architecture (audio and text encoders) used in all experiments.","marker":"[16]"},{"why":"AudioCaps dataset supplies audio-caption pairs for training and evaluation.","marker":"[9]"},{"why":"Clotho dataset supplies audio-caption pairs and the crowdsourced relevance subset for calibrating the logistic function.","marker":"[10]"},{"why":"WavCaps large-scale weakly labeled dataset used for pretraining in the large-scale experiment.","marker":"[5]"}],"fun_headline_variants":["Soft caption relevance beats binary labels in audio search","Learn audio-caption relevance from caption text alone","Listwise training on caption similarity lifts audio retrieval","Non-binary relevance from caption text improves audio retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that captions that are textually similar describe audio content similar enough to make each caption relevant to the other's audio; this transitivity through text is assumed, not validated against audio content.","fun_headline_variants_meta":{"raw":{"variants":["Soft caption relevance beats binary labels in audio search","Learn audio-caption relevance from caption text alone","Listwise training on caption similarity lifts audio retrieval","Non-binary relevance from caption text improves audio retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2504,"prompt_tokens":980,"completion_tokens":1524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":1464}},"tokens_in":596,"tokens_out":1524,"duration_ms":10159,"temperature":1.0,"reasoning_tokens":1464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:25:52.363778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test this by taking a held-out set of audio-caption pairs with human graded relevance, computing the paper's text-similarity-based scores, and checking whether they predict the human ratings; if they do not, or if replacing the computed scores with random soft labels leaves retrieval performance unchanged, the central claim would be contradicted.","supporting_citations":[{"cited_title":"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,","cited_arxiv_id":null,"evidence_quote":"Provides Sentence-BERT embeddings used to compute caption textual similarity h(x_i,x_j)."},{"cited_title":"Learning to Rank: From Pairwise Approach to Listwise Approach,","cited_arxiv_id":null,"evidence_quote":"Defines the ListNet listwise ranking objective used to train with graded relevance scores."},{"cited_title":"Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances,","cited_arxiv_id":null,"evidence_quote":"Supplies crowdsourced graded audio-caption relevance ratings on a Clotho subset, used to fit the logistic mapping in Eq. (5)."},{"cited_title":"Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval","cited_arxiv_id":"2408.11641","evidence_quote":"Provides the dual-encoder architecture (audio and text encoders) used in all experiments."},{"cited_title":"AudioCaps: Generating Cap- tions for Audios in The Wild,","cited_arxiv_id":null,"evidence_quote":"AudioCaps dataset supplies audio-caption pairs for training and evaluation."},{"cited_title":"Clotho: an Audio Captioning Dataset,","cited_arxiv_id":null,"evidence_quote":"Clotho dataset supplies audio-caption pairs and the crowdsourced relevance subset for calibrating the logistic function."},{"cited_title":"WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research,","cited_arxiv_id":null,"evidence_quote":"WavCaps large-scale weakly labeled dataset used for pretraining in the large-scale experiment."}],"review_version":1}