{"id":"4211f236-002a-4ae1-aac5-384af8359bd5","arxiv_id":"2412.20914","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Cascaded co-attention between CLAP audio features and RoBERTa text features improves text-to-audio retrieval mAP by about 16 percent on Clotho and 15 percent on AudioCaps relative to the authors' earlier GPTtar method.","lead":"This paper adds cross-modal co-attention layers to a language-based audio retrieval system so that text and audio representations can refine each other before matching, and it tests shallow, stacked, and iterated versions of the design. On the Clotho and AudioCaps benchmarks, the best iterated version reports about 16 percent and 15 percent relative improvements in mean average precision over the authors' earlier GPTtar system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 16.6%/15.1% SOTA improvements rest on the unstated assumption that GPTtar [2] was re-run under this paper's exact protocol, including the unusual 2-second AudioCaps clipping; without that, the gains may be preprocessing artifacts.","rationale":"The reader's weakest assumption precisely identifies the load-bearing premise: the GPTtar baseline must have been run under identical conditions. Section IV-A introduces preprocessing choices (especially 2-second AudioCaps clipping) that are not standard and are not shown to match the original GPTtar setup. If the comparison is not controlled, the entire headline gain could vanish. The paper gives no explicit statement that GPTtar was re-run, and no code or seeds are released, so this is an unresolved empirical risk rather than a purely stylistic issue. My independent review did not find a different, more fundamental flaw: the architecture is coherent, the reported internal trend (Baseline < GPTtar < Single < Stacking < Iterating) is monotone and plausible, and the relative improvement arithmetic is correct. Therefore the correct verdict remains CONDITIONAL, exactly as the reader concluded: the result is not rejected, but it cannot be accepted as a SOTA claim without reproducing the baseline under the specified protocol and ideally adding external baselines from the literature.","tokens_in":9001,"tokens_out":6902,"duration_ms":68292,"concrete_test":"Obtain the GPTtar code from [2] (or re-implement it) and run it under the exact protocol of this paper: same Clotho 10s fragments, same AudioCaps 2s clips, same GPT-augmented captions after cosine filtering, same RoBERTa/CLAP encoders, same batch sizes (32/64), depth 5, and same evaluation splits. Verify that the reproduced mAP@10 equals 26.5 (Clotho) and 32.5 (AudioCaps). If it does not, the reported 16.6%/15.1% improvements are not attributable to the co-attention modules.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table III is against GPTtar [2], the authors' own prior work. The paper states that 'we followed the experiment settings in [2]' but never states that the GPTtar numbers were reproduced under the present protocol. Section IV-A describes a nonstandard preprocessing choice: Clotho is truncated to 10s fragments and AudioCaps is clipped to 2s clips. If the GPTtar row was imported from [2], where audio lengths or GPT caption filtering may have differed, the reported relative improvements (30.9 vs 26.5 mAP@10 on Clotho; 37.4 vs 32.5 on AudioCaps) would reflect protocol drift, not the co-attention contribution. The concern is concretely load-bearing because the headline claim is a relative gain over GPTtar; if that baseline shifts, the margin shrinks or disappears. The paper also does not report variance, seeds, or code, so there is no way to check whether the comparison is controlled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a co-attention network for language-based audio retrieval, built on CLAP audio encoders and RoBERTa text encoders, with three variants: a single co-attention module, a stacking module, and an iterating module. The authors use GPT-augmented captions and train with an NT-Xent contrastive loss. They report experiments on Clotho and AudioCaps and claim a 16.6% relative mAP improvement on Clotho and a 15.1% improvement on AudioCaps over the GPTtar method [2], which they label state-of-the-art. The main contribution is the cascaded co-attention architecture for cross-modal interaction in audio retrieval.","tokens_in":9150,"tokens_out":3837,"duration_ms":39674,"significance":"If the reported gains are real and the comparison is properly controlled, this is a useful incremental contribution: it applies a well-established co-attention idea to the under-explored text-to-audio retrieval setting and shows that cascaded cross-modal attention can improve over self-attention-only baselines. The use of public datasets, standard contrastive training, and two ablative variants (stacking vs. iterating) are strengths. However, the novelty is limited given that co-attention has been extensively used in VQA and multimodal retrieval, no code or trained models are released, and the empirical evidence rests on a single comparison table with only two comparators. The central claim of state-of-the-art performance therefore depends heavily on the comparability of the GPTtar baseline, which is not established in the manuscript.","major_comments":[{"comment":"The headline 16.6%/15.1% gains are measured relative to GPTtar [2], but the paper does not state whether the GPTtar numbers were produced by re-running GPTtar under the exact protocol of this paper (10-second Clotho fragments, 2-second AudioCaps clips, batch sizes 64 and 32, the same GPT caption augmentation and cosine filtering, and the same splits). Section IV-B only says 'we followed the experiment settings in [2]', which is ambiguous. If the GPTtar row in Table III was imported from [2] with different preprocessing (for example, a different AudioCaps segment length or a different GPT caption selection), the reported margins could be artifacts of protocol drift rather than effects of the co-attention module. The authors should re-run GPTtar under the exact protocol used for their own models and report both rows from the same controlled setup.","section":"§IV-A, §IV-C, Table III"},{"comment":"The contrastive loss definition is internally inconsistent in its index use. Eq. (10) defines similarity between a generic pair (Ai, Tk), and Eq. (11) uses Tk as the positive text for the audio Ai in the numerator, while the denominator sums over Tj for j≠i. If the batch pairs are aligned, the positive text for Ai should be Ti, not Tk; if Tk is meant to be any positive pair, then the index i in the denominator is undefined. As written, the loss is not a well-defined NT-Xent objective. This is load-bearing because the model is trained with this loss; the authors must clarify the pairing convention and correct the indices.","section":"§III-E, Eqs. (10)–(14)"},{"comment":"The iterating module, which gives the best results in Table III, is not fully specified. Eq. (8) defines Ak = GA([SA(Ak−1), Tn]), but Tn is not defined inside the iterative loop: it is unclear whether Tn is computed once from the text self-attention branch before audio refinement, or whether it is updated jointly at each layer, and whether the text and audio branches share parameters across layers. Because the iterating variant is the central claimed contribution, this ambiguity prevents reproduction of the best-performing model and needs to be resolved with a precise layer-wise computation graph.","section":"§III-D, Eq. (8)"},{"comment":"The comparison reports single-run numbers with no variance, seeds, or significance tests. Some differences are small (30.9 vs. 30.6 mAP@10 on Clotho between iterating and stacking), and without run-to-run variability it is impossible to know whether the ranking is stable. The authors should report means and standard deviations over multiple runs with different seeds, at least for the main comparison rows.","section":"§IV-C, Table III"}],"minor_comments":[{"comment":"The cross-references 'Table ??' and 'Figure ??(b)' appear unresolved in the text; these placeholders must be replaced with the actual table and figure numbers.","section":"§III-B, Table I"},{"comment":"There is a typo 'Fugure 4(a)' in the paragraph introducing the stacking module.","section":"§III-D"},{"comment":"There is a typo 'dimensio' in the sentence introducing the joint multimodal space dimension.","section":"§III-E"},{"comment":"The notation in the multi-head guided-attention equations is inconsistent: GFt|a is said to be in R^{h×d}, but the concatenation of h heads followed by a projection Wo should produce d dimensions, and the stated shape Wo ∈ R^{m×d×dm} is unclear. Please use consistent dimension variables (for example, h for heads and dh for per-head dimension).","section":"§III-C, Eqs. (5)–(6)"},{"comment":"The metric label 'mAP@10' appears in the table header, but the text refers only to 'mean average precision'; please clarify whether mAP is computed over the top-10 ranked items and define the metric precisely.","section":"§IV-C, Table III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's state-of-the-art row is the authors' own prior work (GPTtar [2]), which is not by itself a defect but does raise the bar for showing that the comparison is controlled. The paper does not explicitly state that GPTtar was re-run under the same protocol, and the unusual 2-second AudioCaps clipping makes this a concrete correctness risk. The loss-function index issue in Eqs. (10)–(14) is also a genuine correctness gap that must be fixed. If the authors can provide a clean controlled comparison and clarify the loss and the iterating module, the contribution could be publishable as an incremental methods paper; as it stands, the central claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a modest, coherent extension of the authors' own GPTtar pipeline. The new bit is a bidirectional guided co-attention module, with stacking and iterating depth variants, inserted between CLAP and RoBERTa embeddings and trained with NT-Xent. The headline numbers (16.6% and 15.1% relative mAP gains over GPTtar) are plausible but rest on an unverified assumption about how the GPTtar baseline was produced.\n\nWhat is genuinely new: no prior audio-retrieval work I know combines guided text- and audio-attention units with stacked or iterated cascades, and the architecture in Sec. III is described consistently. The internal trend supports the direction: every added component improves monotonically (baseline → GPTtar → +single → +stacking → +iterating) on both datasets. The equations for projection, guided attention, and NT-Xent are standard and mostly consistent, apart from index typos, N doing double duty as frames and batch size, and a stray dkt in Eq. (4).\n\nThe soft spots, in proportion:\n\n(1) The SOTA claim hangs entirely on the GPTtar row in Table III, and the paper never says GPTtar was re-run under the present protocol. \"We followed the experiment settings in [2]\" is not the same as reproducing the baseline, especially when the protocol includes a nonstandard 2-second AudioCaps clipping (which part? which offset?). If the GPTtar numbers were imported from [2], the 16.6%/15.1% margins could be preprocessing drift, not co-attention effect. This is not a demonstrated error, but it is load-bearing, and the stress-test concern survives a careful reading.\n\n(2) Only two comparators. Several stronger methods from the related work are never evaluated, which makes \"state-of-the-art\" an overclaim. Self-citation is not the problem; the thin comparison set is.\n\n(3) Reproducibility: no code, no seeds or variance, a broken reference to Table I, and the \"durange\" header typo in Table II are minor but add friction.\n\nWho it is for: people building text-audio retrieval systems who want a testable co-attention module. It deserves a serious referee: an editor can ask for code and a precise statement of how the GPTtar numbers were obtained, and with a reproduced baseline the paper is a publishable incremental result. Without that, the claims need to be rescaled to \"improvement within our protocol.\"","headline":"Plausible incremental co-attention result whose SOTA claim depends on an unverified reproduction of the authors' own baseline; referee-worthy to pin that down.","tokens_in":9789,"tokens_out":4963,"would_cite":false,"duration_ms":46669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A cascaded co-attention network with iterated guided attention reports 16.6% and 15.1% mAP improvements over GPTtar on Clotho and AudioCaps.","keywords":["language-based audio retrieval","co-attention mechanism","text-audio retrieval","contrastive learning","CLAP","RoBERTa","GPT caption augmentation","mean average precision"],"falsifier":"Re-run GPTtar under the preprocessing described in Section IV-A (10-second Clotho clips, 2-second AudioCaps clips, batch sizes 32 and 64, depth 5) and compare mAP. If GPTtar reproduces its original 26.5 and 32.5 mAP values under these conditions, the central claim stands; if GPTtar's numbers were taken from a different setting and change materially, the measured gains shrink or disappear.","tokens_in":8711,"feed_emoji":"🎧","tokens_out":4305,"duration_ms":39505,"temperature":0.7,"pith_summary":"This paper tries to establish that a co-attention mechanism, in which text and audio embeddings attend to each other rather than only within each modality, improves language-based audio retrieval. It proposes a single co-attention module made of self-attention and guided-attention units, then cascades that module in two ways: stacking and iterating. On Clotho and AudioCaps, the best iterating model reports mAP of 30.9 and 37.4, exceeding the GPTtar baseline's 26.5 and 32.5 by 16.6% and 15.1% relative. If true, this means crossing attention between words and audio frames is a workable route to better semantic alignment in retrieval.","feed_headline":"Co-attention lifts audio-retrieval accuracy 16.6%","feed_subtitle":"Cascaded guided attention refines text–audio alignment on Clotho and AudioCaps, beating the prior GPTtar pipeline.","key_machinery":"The load-bearing mechanism is the cascaded co-attention module. Each single module contains two self-attention blocks (one per modality) plus a guided-attention block in which queries from one modality attend to keys and values of the other, with multi-head attention. Stacking chains these modules layer after layer; iterating computes the text representation once and uses it as the query source when refining audio features at each depth. Equipped with NT-Xent loss, the module learns to project both modalities into a shared space where contrastive similarity ranks matching audio-text pairs above negatives.","core_discovery":"The central discovery is that co-attention, not just self-attention, drives retrieval gains: using audio queries to attend to text key-value pairs and text queries to attend to audio frames produces embeddings better aligned for ranking. Cascading the module in depth helps further; the iterating variant, which first computes attended text features and then uses them to guide audio attention across layers, performs best. The framework trains with NT-Xent contrastive loss over CLAP audio embeddings and RoBERTa text embeddings, with GPT-augmented captions filtered by cosine similarity.","pith_inferences":["The paper does not isolate the co-attention module from GPT caption augmentation, so an ablation with fixed captions would tell whether the gains come from cross-modal attention or from richer caption text.","Because the iterating module uses text-guided attention on audio but not the reverse in the same iterating fashion, a symmetric iterating design might further refine text embeddings.","The 2-second clipping of AudioCaps is an unusual preprocessing choice; testing whether the co-attention advantage persists at full clip lengths would clarify whether the gains are specific to short segments.","A natural extension is applying the same cascaded co-attention to text-to-video retrieval, where fine-grained cross-modal alignment between words and frames is also central."],"forward_implications":["Co-attention between words and audio frames can be added to existing CLAP/RoBERTa retrieval pipelines and yield mAP gains over self-attention-only pooling.","Deep cascading helps: iterating beats stacking, so representation refinement across layers is a productive axis for retrieval, not just for classification.","GPT caption augmentation combined with co-attention is compatible; the reported gains hold on both short clips (AudioCaps, 2 seconds) and longer clips (Clotho, 10 seconds).","If the reported margins are reproducible under identical preprocessing, the co-attention module itself, rather than the encoders or loss, is the source of the improvement."],"supporting_citations":[{"why":"Provides the GPTtar baseline that the proposed method compares against, including GPT caption augmentation and self-attention pooling, and the experimental settings the paper follows.","marker":"[2]"},{"why":"Supplies the CLAP audio encoder used to produce audio embeddings from natural-language supervision.","marker":"[19]"},{"why":"Supplies the RoBERTa text encoder used to encode captions.","marker":"[26]"},{"why":"Provides the Clotho dataset, one of the two evaluation benchmarks, with long audio clips and multiple captions per clip.","marker":"[27]"},{"why":"Provides the AudioCaps dataset, the second evaluation benchmark, with short audio clips and human-written captions.","marker":"[28]"},{"why":"Defines the DCASE language-based audio retrieval task and supplies the baseline method used for comparison.","marker":"[1]"},{"why":"Foundational multi-head attention mechanism that underlies the self-attention and guided-attention units in the co-attention module.","marker":"[20]"}],"fun_headline_variants":["Co-attention nets give audio retrieval 16.6% lift","Cascaded co-attention leads audio retrieval gains of 16.6%","Audio-text co-attention improves retrieval mAP by 16.6%","Deep co-attention refines text–audio alignment for search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the GPTtar baseline was re-run under exactly the same conditions, including the GPT caption augmentation, the CLAP and RoBERTa encoders, the 10-second Clotho and 2-second AudioCaps fragment clipping, batch sizes of 32 and 64, depth of 5, and the same evaluation protocol, so the reported 16.6% and 15.1% gains come from the co-attention module rather than from different experimental settings.","fun_headline_variants_meta":{"raw":{"variants":["Co-attention nets give audio retrieval 16.6% lift","Cascaded co-attention leads audio retrieval gains of 16.6%","Audio-text co-attention improves retrieval mAP by 16.6%","Deep co-attention refines text–audio alignment for search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1702,"prompt_tokens":832,"completion_tokens":870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":791}},"tokens_in":448,"tokens_out":870,"duration_ms":8488,"temperature":1.0,"reasoning_tokens":791,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:07:55.188842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run GPTtar under the preprocessing described in Section IV-A (10-second Clotho clips, 2-second AudioCaps clips, batch sizes 32 and 64, depth 5) and compare mAP. If GPTtar reproduces its original 26.5 and 32.5 mAP values under these conditions, the central claim stands; if GPTtar's numbers were taken from a different setting and change materially, the measured gains shrink or disappear.","supporting_citations":[{"cited_title":"Language- based audio retrieval with gpt-augmented captions and self-attended audio clips,","cited_arxiv_id":null,"evidence_quote":"Provides the GPTtar baseline that the proposed method compares against, including GPT caption augmentation and self-attention pooling, and the experimental settings the paper follows."},{"cited_title":"Clap learning audio concepts from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the CLAP audio encoder used to produce audio embeddings from natural-language supervision."},{"cited_title":"Clotho: an audio captioning dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the Clotho dataset, one of the two evaluation benchmarks, with long audio clips and multiple captions per clip."},{"cited_title":"AudioCaps: Generating captions for audios in the wild,","cited_arxiv_id":null,"evidence_quote":"Provides the AudioCaps dataset, the second evaluation benchmark, with short audio clips and human-written captions."},{"cited_title":"Language-based Audio Retrieval Task in DCASE 2022 Challenge","cited_arxiv_id":"2209.09967","evidence_quote":"Defines the DCASE language-based audio retrieval task and supplies the baseline method used for comparison."}],"review_version":1}