{"id":"2aaec6f9-b362-4b43-9d56-995a179494e3","arxiv_id":"2505.14562","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A single-stage contrastive alignment of audio, visual, and text using both audio and visual captions improves audio-to-visual retrieval recall@10 from 0.27 to 0.52 on AVCaps.","lead":"This paper proposes SLAVA, a single-stage contrastive training method that aligns audio, visual, and text representations using both audio and visual captions from the AVCaps dataset. The authors report a two-fold improvement in audio-based visual retrieval over a two-stage baseline, suggesting that unified training may reduce modality distribution mismatches.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.52 vs 0.27 gap is confounded: the two-stage baseline lacks the explicit audio-visual loss (Lav), so the comparison attributes to 'single-stage' what may just be direct audio-visual contrastive training.","rationale":"I read the paper in good faith: the empirical result is plausible, and adding an explicit audio-visual contrastive loss should improve audio-based visual retrieval. My concern is not that the numbers are fabricated or internally inconsistent; it is that the central conceptual claim—'single-stage outperforms two-stage'—is not actually tested. The comparison in Table II changes the loss objective at the same time as the training structure: the Wav2CLIP-style two-stage baseline has only Lvt and Lat, while SLAVAA&V has Lav plus one text loss. The paper's own reference runs show that a model with Lav but no visual captions (AudioCLIP-style) reaches 0.45, and a pure audio-visual model reaches 0.50, nearly matching the proposed 0.52. So the reported gain over the two-stage baseline is almost entirely attributable to the presence of Lav. A two-stage baseline with an explicit Lav would settle this. If it also reaches about 0.5, the paper should be reframed as 'adding audio-visual contrast helps' rather than 'single-stage helps.' The reader's concern about test-set validity and generalization is real but secondary; even a perfect external benchmark would not resolve the attribution problem. I therefore keep the conditional verdict, with the additional condition that the authors run the proposed ablation isolating Lav from the single-stage factor.","tokens_in":6652,"tokens_out":10281,"duration_ms":100972,"concrete_test":"Retrain the Wav2CLIP-style two-stage baseline with an explicit audio-visual loss in the second stage (e.g., optimize Lat + Lav with trainable text projection, or add a third stage optimizing Lav), using the same AVCaps training/validation splits and the same 20-epoch budget. If audio-based visual retrieval on the AVCaps test split reaches approximately 0.50 or higher, the 0.27-to-0.52 gap is explained by the added Lav and the single-stage claim should be reframed. Conversely, train SLAVAA&V without Lav (alternating only Lat/Lvt) to quantify the contribution of Lav within the proposed model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that single-stage training with audio and visual captions produces the audio-to-visual retrieval improvement. The Table II comparison does not isolate this factor. The Wav2CLIP-style two-stage baseline optimizes Lvt then Lat, with no Lav; the proposed SLAVAA&V optimizes Lav + (Lat or Lvt). The improvement from 0.27 to 0.52 therefore bundles two changes: (i) single-stage joint training and (ii) an explicit audio-visual contrastive loss. The paper's own AudioCLIP-style single-stage reference, which also includes Lav (with Lvt and Lat using audio captions only), already reaches 0.45, and the authors' audio-visual-only model (CLIP image + CLAP audio, no text) reaches 0.50, nearly matching the proposed 0.52. Thus most or all of the gain relative to the two-stage baseline can be explained by the presence of Lav, not by the single-stage or caption design. Without an ablation that runs the two-stage pipeline with Lav added (or removes Lav from SLAVAA&V), the paper cannot support the headline attribution 'single-stage outperforms two-stage'; it only supports 'adding explicit audio-visual contrast improves audio-visual retrieval.' This concern is independent of the test-set validity issue and would persist even on an external benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SLAVA, a single-stage contrastive learning framework that aligns audio, visual, and text modalities by fine-tuning the projection layers of pretrained CLIP and CLAP encoders on the AVCaps dataset. The paper compares a Wav2CLIP-style two-stage baseline, an AudioCLIP-style single-stage baseline, and two proposed variants: one trained with audio and visual captions (SLAVAA&V) and one with LLM-generated audio-visual captions (SLAVAAV). The main empirical claim is that the proposed SLAVAA&V model improves audio-based visual retrieval recall@10 from 0.27 (two-stage) to 0.52 on the AVCaps test split.","tokens_in":6883,"tokens_out":5302,"duration_ms":44442,"significance":"The AVCaps dataset with modality-specific captions is a useful public resource, and the paper explicitly states its loss functions in Eqs. (1)-(3), which makes the training setups reproducible. If the reported gain were attributable to the single-stage design rather than to the added audio-visual contrastive loss, the result would be a meaningful demonstration that unified trimodal training improves audio-visual retrieval. The paper also provides multiple reference systems, including an audio-visual-only model reaching 0.50, which is a useful internal diagnostic. However, because the main comparison is confounded and the evaluation is not statistically quantified, the paper cannot yet support its headline claim at its current strength.","major_comments":[{"comment":"The headline claim that 'single-stage outperforms two-stage' is confounded. The Wav2CLIP-style two-stage baseline optimizes Lvt then Lat and never includes Lav, whereas SLAVAA&V optimizes Lav + (Lat or Lvt). Hence the 0.52 vs 0.27 difference bundles two changes: (i) single-stage joint training and (ii) explicit audio-visual contrastive loss. The paper's own ablations support the importance of Lav: the audio-visual-only model (no text) reaches 0.50, and the AudioCLIP-style single-stage model with Lav reaches 0.45, while the SLAVAAV variant without Lav (Lat+Lvt) reaches 0.40. To support the attribution to single-stage training, the authors should add Lav to the two-stage baseline, or remove Lav from SLAVAA&V, and report that comparison.","section":"Section V, Table II; Section III-C/III-D"},{"comment":"The evaluation lacks statistical grounding. No standard deviations, confidence intervals, or number of seeds are reported, and the retrieval protocol is incompletely specified: the 'database' size for the retrieval tasks is not given, and the definition of recall@10 as 'proportion of relevant items among the top-10 retrieved results by the total of items in the dataset' is ambiguous or incorrect. Differences among several rows (e.g., 0.29 vs 0.30, 0.32 vs 0.33) are within typical run-to-run noise for contrastive models. Multiple seeds or error bars are needed before a 'two-fold improvement' can be assessed quantitatively.","section":"Section IV and Table II"},{"comment":"The evaluation is conducted only on the AVCaps test split, a self-created dataset of 2,061 videos cited as [17]. No external validation or cross-dataset transfer result is reported, and the test split size is not stated. The abstract and conclusion generalize to 'unified multimodal representation learning,' but the evidence is single-dataset and single-split. To make the central claim robust, the authors should at least provide a breakdown of train/validation/test sizes and discuss representativeness, or add an out-of-distribution evaluation.","section":"Section IV/V"}],"minor_comments":[{"comment":"The notation 'SLA V AA&V' and 'SLA V A A V' is visually confusing; the distinction between audio-and-visual captions (A&V) and audio-visual captions (AV) should be made explicit with consistent typography.","section":"Section III-B and Table I"},{"comment":"The sentence defining recall@10 should be rewritten; recall@10 is the fraction of queries for which at least one ground-truth item appears in the top 10 results, not 'by the total of items in the dataset.'","section":"Section IV"},{"comment":"The number of video frames M and the number of audio chunks N are never specified; these are implementation details needed to reproduce the experiments.","section":"Section III-B"},{"comment":"'Two-fold improvement' is used loosely; 0.27 to 0.52 is approximately 1.9x, so 'nearly two-fold' would be more precise.","section":"Abstract and Conclusion"},{"comment":"The two-stage schematic labels 'Lvt 1st-stage' and 'Lat 2nd-stage' are small; consider larger fonts or a table to make the difference from the single-stage diagram clearer.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is incremental relative to AudioCLIP, and the strongest empirical result depends heavily on a self-created dataset. For the journal, I would want the ablations and statistical rigor described in the major comments before acceptance. The citation pattern is not problematic, but the authors should avoid using their own dataset as the sole benchmark. The paper is within scope for an audio/speech/multimodal venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's headline claim overreaches. The 0.52 vs 0.27 gain in audio-based visual retrieval is real, but it is mostly caused by adding the explicit audio-visual contrastive loss (Lav), not by single-stage training or by using both caption types. Their own numbers show this: the AudioCLIP-style single-stage reference with audio captions and Lav reaches 0.45, a pure audio-visual model without any text reaches 0.50, and SLAVA_AV without Lav gets only 0.40. So the comparison to the Wav2CLIP-style two-stage baseline bundles two changes, and the paper never runs the ablation that would separate them (two-stage with Lav added, or the proposed method without Lav). That is load-bearing for the claim that single-stage training is the cause of the improvement.\n\nWhat is genuinely useful: AVCaps is a public dataset with modality-specific captions, and the paper lays out a clean comparison framework. The single-stage recipe that alternates audio and visual captions is simple and reproducible, and the authors include reference implementations of Wav2CLIP- and AudioCLIP-style baselines plus an audio-visual-only model. Reporting that AV-only model at 0.50 is honest, but they don't connect it to the attribution problem.\n\nOther soft spots are more minor: no error bars or multiple seeds, thin detail on the retrieval protocol (test split size, database size, leakage checks), no code, and no external benchmark. These are fixable but mean the exact magnitude is uncertain.\n\nWho is this for? People working on audio-visual-text alignment and multimodal representation learning, plus anyone building on AVCaps. The paper is a good case study in confounded baselines.\n\nBottom line: the central claim needs re-framing from \"single-stage outperforms two-stage\" to \"explicit audio-visual contrastive loss improves audio-visual retrieval.\" That is still a positive, modest result. I would send this to peer review with a required ablation and external validation, but I would not cite the headline result without seeing that ablation. For a reading group, it's worth discussing as an example of why baseline design matters.","headline":"Useful dataset, overclaimed headline: the reported gain is mostly from the explicit audio-visual loss, not from single-stage training, as the paper's own ablations show.","tokens_in":7451,"tokens_out":1853,"would_cite":false,"duration_ms":19168,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single training stage which jointly aligns audio, visual, and text representations doubles audio-based visual retrieval recall@10 from 0.27 to 0.52.","keywords":["multimodal representation learning","contrastive learning","audio-visual alignment","trimodal alignment","audio-based visual retrieval","AVCaps dataset","CLIP","CLAP"],"falsifier":"Run the same SLAVAA&V model on a held-out evaluation set with new video clips and fresh captions that were not used in training, and compare audio-based visual retrieval recall@10 to the two-stage baseline; if the gain shrinks to near 0.27 or the absolute score drops well below 0.52 on out-of-distribution data, the reported improvement is specific to the AVCaps test split rather than a general property of single-stage alignment.","tokens_in":6419,"feed_emoji":"🎧","tokens_out":6232,"duration_ms":57439,"temperature":0.7,"pith_summary":"Aligning three modalities—language, audio, and visual—has usually been done in two stages: first align visual with text, then align audio with text. This paper claims that doing all alignments in one single stage, with a dataset that supplies separate captions for the audio track and the visual content, gives better audio-visual alignment than the two-stage recipe. On the AVCaps test split, their single-stage model SLAVA with audio and visual captions raises audio-based visual retrieval recall@10 from 0.27 (two-stage reference) to 0.52, roughly a two-fold improvement, while keeping other retrieval scores competitive. The implication is that joint training on modality-specific captions is a practical route to unified audio-visual-language representations.","feed_headline":"Single-stage training doubles audio-based visual retrieval","feed_subtitle":"Joint contrastive alignment of audio, visual, and text lifts recall@10 from 0.27 to 0.52 on AVCaps.","key_machinery":"The machinery is joint contrastive minimization of three pairwise losses over a shared embedding space. For a batch, audio, visual, and text inputs are encoded by frozen pretrained encoders (CLIP for visual/text, CLAP for audio), temporally averaged, and passed through learnable linear projections to 512 dimensions. The losses are $L_{av}$ (audio-visual), $L_{at}$ (audio-text), and $L_{vt}$ (visual-text), each an InfoNCE contrastive term; the SLAVAA&V variant uses $L_{total}=L_{av}+(L_{at}\text{ or }L_{vt})$ depending on which caption type is sampled in the batch, while the audio-visual-caption variant uses $L_{total}=L_{av}+L_{at}+L_{vt}$. The key is that $L_{av}$ directly ties audio and visual representations, and the availability of both audio and visual captions in AVCaps prevents the text encoder from drifting toward only one modality, which is what limits the single-stage AudioCLIP-style reference.","core_discovery":"The central claim is that a single-stage contrastive model can align audio, visual, and textual representations better than the standard two-stage pipeline, provided the training data has captions for each modality separately. The paper introduces SLAVA, which starts from pretrained CLIP image and text encoders and a CLAP audio encoder, adds linear projection layers for each, and fine-tunes those projections jointly with the InfoNCE loss over pairwise terms: audio-visual, audio-text, and visual-text. Using the AVCaps dataset, which contains audio, visual, and audio-visual captions for each clip, the model is trained with both audio captions and visual captions in a single stage. The reported result is recall@10 of 0.52 for retrieving visuals from an audio query, versus 0.27 for a Wav2CLIP-style two-stage baseline on the same test split.","pith_inferences":["Because AVCaps has only 2,061 videos, the reported gain should be re-tested on larger or out-of-distribution audio-visual benchmarks; the paper does not report such an evaluation.","The same single-stage recipe may transfer to other modality triples, such as video-language-motion or image-language-depth, as long as the dataset has per-modality captions.","A testable extension is weighting the three pairwise losses or sampling captions by modality to push the recall further; the paper keeps all losses at equal weight.","The two-fold improvement may come partly from aligning text to audio, indirectly regularizing the audio encoder; an ablation that removes text entirely would separate that effect."],"forward_implications":["A single training stage can replace the two-stage visual-text then audio-text pipeline for trimodal alignment.","Adding an explicit audio-visual contrastive loss is what lifts audio-based visual retrieval; without $L_{av}$, the best audio-visual-caption variant reaches only 0.40.","Training with only audio captions (AudioCLIP-style) drops visual retrieval to 0.69, so keeping visual captions in the same stage protects visual alignment.","A model trained with the three pairwise losses simultaneously can match a dedicated audio-visual model (0.50 vs 0.50) while also aligning text to both modalities."],"supporting_citations":[{"why":"supplies the AVCaps dataset with separate audio, visual, and audio-visual captions that make single-stage training possible","marker":"[17]"},{"why":"provides the pretrained CLIP image and text encoders used as the visual and textual backbones","marker":"[1]"},{"why":"provides the pretrained CLAP audio encoder used as the audio backbone","marker":"[2]"},{"why":"defines the Wav2CLIP-style two-stage approach that serves as the main baseline","marker":"[7]"},{"why":"defines the AudioCLIP-style single-stage reference that uses only audio captions","marker":"[15]"},{"why":"supplies the InfoNCE contrastive loss used for all pairwise alignments","marker":"[18]"}],"fun_headline_variants":["Single-stage contrastive alignment doubles audio-visual recall","Joint audio-visual-text training beats two-stage pipeline","One-stage alignment lifts audio-based visual retrieval twofold","SLAVA: single-stage trimodal alignment with 2x retrieval gain","Unified contrastive learning improves audio-visual retrieval 2x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the AVCaps test split and retrieval protocol measure audio-visual alignment fairly; if the test split is small, overlaps with training captions, or is unrepresentative, the reported recall@10 gain may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Single-stage contrastive alignment doubles audio-visual recall","Joint audio-visual-text training beats two-stage pipeline","One-stage alignment lifts audio-based visual retrieval twofold","SLAVA: single-stage trimodal alignment with 2x retrieval gain","Unified contrastive learning improves audio-visual retrieval 2x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000434,"raw_usage":{"total_tokens":2168,"prompt_tokens":860,"completion_tokens":1308,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":1225}},"tokens_in":476,"tokens_out":1308,"duration_ms":9611,"temperature":1.0,"reasoning_tokens":1225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:31:57.194446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SLAVAA&V model on a held-out evaluation set with new video clips and fresh captions that were not used in training, and compare audio-based visual retrieval recall@10 to the two-stage baseline; if the gain shrinks to near 0.27 or the absolute score drops well below 0.52 on out-of-distribution data, the reported improvement is specific to the AVCaps test split rather than a general property of single-stage alignment.","supporting_citations":[{"cited_title":"CLAP learning audio concepts from natural lan- guage supervision,","cited_arxiv_id":null,"evidence_quote":"provides the pretrained CLAP audio encoder used as the audio backbone"},{"cited_title":"Wav2CLIP: Learning robust audio representa- tions from CLIP,","cited_arxiv_id":null,"evidence_quote":"defines the Wav2CLIP-style two-stage approach that serves as the main baseline"},{"cited_title":"Sudarsanam, I","cited_arxiv_id":null,"evidence_quote":"supplies the AVCaps dataset with separate audio, visual, and audio-visual captions that make single-stage training possible"},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"provides the pretrained CLIP image and text encoders used as the visual and textual backbones"},{"cited_title":"Audio- CLIP: Extending CLIP to image, text and audio,","cited_arxiv_id":null,"evidence_quote":"defines the AudioCLIP-style single-stage reference that uses only audio captions"}],"review_version":1}