{"id":"cda5ea59-48d8-40fe-9541-72e427ded906","arxiv_id":"2501.09291","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LAVCap improves automated audio captioning by using optimal transport to align audio and visual features and to fuse them before an LLM decoder generates text, reporting SOTA SPIDEr on AudioCaps.","lead":"LAVCap is an audio-visual captioning system that uses optimal transport to align and fuse audio and video features before an LLM writes the caption. On the AudioCaps benchmark it reports the highest combined captioning score among audio-visual methods, without extra pre-training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4)'s OT loss is invariant under audio/visual token permutation, so it cannot enforce semantic alignment; the claimed mechanism behind the SOTA gain is unverified.","rationale":"The reader's weakest_assumption correctly identified the OT alignment as the key risk, but the concern is stronger than 'the alignment could be noise.' Equation (4) is permutation-invariant in both token dimensions, so the loss provides no gradient signal that prefers a semantically correct correspondence over an arbitrary bijection. This is a structural property of the loss, not merely a missing experiment. The empirical SOTA result may still be reproducible, but the paper's explanation of why it works is unsupported. Since the central contribution is the OT alignment mechanism, the verdict remains conditional: the authors must either demonstrate that Q* is semantically aligned (using the proposed test or equivalent) or reframe the contribution as a regularizer/discriminative auxiliary loss. This does not change the reader's CONDITIONAL verdict, but it sharpens the condition that must be met.","tokens_in":7827,"tokens_out":14674,"duration_ms":209202,"concrete_test":"On the released model, compute Q* for AudioCaps validation clips. For each audio token, extract its top-assigned visual patch and score whether that patch's CLIP embedding matches the ground-truth caption's event nouns (e.g., via CLIP text similarity of the noun phrases). Compare the mean top-1 agreement against a random doubly stochastic assignment baseline. If LAVCap's Q* is not significantly above chance, the OT alignment is not semantically meaningful and the Table II gain must be attributed to other components.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LAVCap's headline result rests on the claim that the OT loss in Eq. (4) 'bridges the modality gap' and that the OT assignment map Q* in Eq. (5) provides semantically grounded cross-modal fusion. In Eq. (4), however, the loss is a row-wise and column-wise InfoNCE-style contrast between Q* and S. Because Q* is obtained from S by Sinkhorn iterations, a column permutation of the visual tokens (S' = S P, Q' = Q P) leaves every inner product in Eq. (4) unchanged; the same holds for permuting audio-token rows. Thus, for any fixed S, the loss assigns exactly the same value to any bijective token correspondence. It rewards making rows of S mutually distinguishable, not matching audio token i to the visual token that depicts the same event. Consequently, LOT cannot by itself produce semantic alignment; any gain from adding it (Table II: SPIDEr 48.5 to 51.0) could be an auxiliary discriminative or regularization effect rather than a semantic bridge. The paper provides no analysis of the learned Q*, no correspondence visualization, and no comparison to a standard audio-visual InfoNCE alignment. Without such evidence, the central contribution is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"LAVCap is an audio-visual captioning system that combines a frozen CLIP ViT-L/14 visual encoder, a CED audio encoder adapted with LoRA, and a Llama-2 7B decoder adapted with LoRA. Its two novel components are an optimal-transport (OT) alignment loss, Eq. (4), and an OT-attention fusion module, Eq. (5), both built from the Sinkhorn assignment Q* of the cross-modal similarity matrix S. Training uses a weighted combination of captioning cross-entropy and OT loss, Eq. (9). On AudioCaps the method reports SPIDEr 51.7, CIDEr 84.9, and METEOR 26.2 without additional caption datasets or post-processing, and ablations show each component contributes.","tokens_in":8096,"tokens_out":10269,"duration_ms":106564,"significance":"If the reported numbers are reproducible, LAVCap is a data-efficient state-of-the-art result for audio captioning with visual side information, and the parameter-free OT-attention fusion is a simple and interesting design choice. The paper has notable strengths: all components are ablated under the same backbone (Tables II-IV), the code is publicly released, and the MOS user study includes confidence intervals. The central mechanistic claim, however, is not yet established: the OT loss in Eq. (4) is permutation-invariant and therefore cannot by itself select a semantically correct audio-visual token correspondence. The contribution would be convincing if the authors provided a shuffled-token control, an analysis of Q*, or a comparison with a standard audio-visual contrastive alignment loss. The work is suitable for a major revision rather than rejection because the empirical result and the fusion module remain plausible and the missing analyses are feasible.","major_comments":[{"comment":"The claim that L_OT 'bridges the modality gap' is not supported, because the loss is invariant under arbitrary permutations of the audio or visual tokens. For a permutation matrix P, replacing S by S P changes the Sinkhorn solution to Q* P (the marginals are uniform), so each inner product in Eq. (4) is either unchanged or relabeled, and the loss value is identical; the same holds for row permutations of S. Consequently L_OT cannot favor one token correspondence over another; it only encourages rows and columns of Q* to be self-consistent with rows and columns of S. The 2.5 SPIDEr gain from adding L_OT in Table II may therefore be a discriminative or regularization effect rather than semantic cross-modal alignment. Please add a shuffled-token or permutation-invariance control, an analysis of the learned Q* (visualization or correspondence accuracy), or a comparison against a standard audio-visual InfoNCE loss trained on the same features.","section":"Section II-B, Eq. (4)"},{"comment":"The headline result rests on a single evaluation run. Several advantages over the strongest prior systems are small (SPIDEr 51.7 vs. 50.7 for AutoCap; METEOR 26.2 vs. 26.7 for LOAE), and no confidence intervals, multiple seeds, or significance tests are reported for the automatic metrics. Please report mean +/- standard deviation over at least three seeds for the final model and the closest baselines, or a paired test over test captions, before claiming state-of-the-art performance.","section":"Section III-B, Table I"},{"comment":"The paper omits several quantities needed to reproduce the objective and the experiments: the weights lambda_CE and lambda_OT, the temperature tau, the entropy regularization epsilon, the number of Sinkhorn iterations, the LoRA rank and alpha, and the batch size. It also does not state how the 'acquired' 48,595 training and 944 test clips were selected or whether prior methods in Table I were evaluated on the same subset. Please report these hyperparameters and the dataset subsetting procedure.","section":"Section III-A and Eq. (9)"}],"minor_comments":[{"comment":"The method name appears as 'LAVCap' in the title and abstract but as 'LA VCap' in the body; please standardize the spelling.","section":"Throughout"},{"comment":"The same symbol Q is used for both the optimization variable and the feasible set; please use a distinct symbol, for example \\mathcal{Q}, for the set of admissible transport plans.","section":"Eq. (2)"},{"comment":"The sentence 'OT loss encourages maximizing the similarity between audio-visual token sequences within a sample' does not match Eq. (4), which contrasts rows and columns of Q* with rows and columns of S; please rephrase to describe the actual objective.","section":"Section II-B"},{"comment":"The spectrogram size 1024 x 64 is inconsistent with a 10-second signal at 16 kHz and a 10-ms hop, which yields roughly 1,000 STFT frames; please clarify whether a mel filterbank or a resize is applied and report the exact input format.","section":"Section III-A"},{"comment":"Many entries for baseline methods are dashes; please state whether those metrics were not reported in the original papers or are omitted here.","section":"Table I"},{"comment":"The decoding hyperparameters for Llama 2 at inference (e.g., temperature, beam size, sampling strategy) are not reported; since the paper disclaims post-processing, these settings matter for reproducibility.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the empirical work is plausible. My main concern is that the central OT-alignment mechanism is unverified; I would like to see the permutation/shuffled-token control and multi-seed results before acceptance. I do not view this as grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: LAVCap reports strong results on AudioCaps with a data-efficient recipe, but the central mechanism—optimal transport alignment of audio and visual tokens—does not survive close reading of Eq. (4). The loss is invariant under any bijective permutation of tokens within either modality, so it cannot enforce semantic correspondence. It is better described as an auxiliary discriminative loss that may regularize features, not a semantic bridge.\n\nWhat is genuinely new: the paper is the first to apply OT to audio-visual captioning, and it proposes a parameter-free fusion module (OT-Att) that uses the Sinkhorn assignment map as attention weights. The ablations are coherent: adding visuals alone gives a small gain (SPIDEr 47.6 to 48.5), adding the OT loss gives a larger one (48.5 to 51.0), and OT-Att beats cross-attention and Q-Former under the same backbone. The code is public. That is a solid empirical contribution and deserves referee time.\n\nSoft spots, in proportion: (1) The mechanistic claim is overstated. The paper says OT loss \"bridges the modality gap\" and OT-Att provides \"semantically aligned\" fusion, but the math shows the loss cannot distinguish any token-to-token correspondence. The authors provide no analysis of the learned Q*, no visualization of correspondences, and no comparison against a standard audio-visual InfoNCE loss. Without that, the alignment story lacks support. This is a real flaw, not a nitpick, because it is the paper's stated contribution. (2) The main results lack error bars or multiple seeds, and key hyperparameters (lambda_OT, tau, epsilon, Sinkhorn iterations) are omitted. These are addressable but block independent replication. (3) The SOTA comparison is confounded by architecture and data differences; the concurrent works that used extra datasets are not matched on the LLM backbone.\n\nThe permutation-invariance argument holds; I checked it against the equations. The empirical gains could still be real, but they might come from a discriminative regularizer or from the model exploiting temporal order implicitly. The authors should either provide evidence that Q* tracks actual audio-visual correspondences or reframe the contribution as an efficient, parameter-free fusion layer without the alignment narrative.\n\nWho this is for: researchers in audio captioning, multimodal LLM fusion, and OT-based alignment. It deserves a serious peer review, but with a request for the missing analysis and hyperparameters. I would not cite it as evidence for OT-based cross-modal alignment until that gap is closed.\n\nRecommendation: send to review, flag the mechanism concern as the primary issue, and ask for the additional experiments.","headline":"Strong AudioCaps results and a clean ablation, but the OT alignment loss is permutation-invariant and cannot be doing what the paper claims.","tokens_in":8630,"tokens_out":4259,"would_cite":false,"duration_ms":47555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LAVCap claims that optimal-transport alignment and fusion of audio and visual tokens outperforms prior audio captioning systems on AudioCaps without extra training data.","keywords":["audio captioning","audio-visual fusion","optimal transport","Sinkhorn algorithm","large language models","Low-Rank Adaptation","AudioCaps","multimodal alignment"],"falsifier":"Replace the learned similarity matrix $S$ in Eqs. (2)-(5) with a random matrix (or an identity-like constant) while keeping the Sinkhorn iterations, losses, and all other training choices unchanged; if the resulting SPIDEr stays close to the reported 51.7 rather than collapsing toward the no-OT baseline of 48.5, the transport plan itself is not carrying the claimed alignment and the gain must come from elsewhere in the pipeline.","tokens_in":7640,"feed_emoji":"🎧","tokens_out":6577,"duration_ms":54336,"temperature":0.7,"pith_summary":"The paper introduces LAVCap, a captioning system that describes a 10-second audio clip by also looking at the video frames that accompany it. The authors' central claim is that the usual failure of audio-visual captioning models is a modality gap: audio and visual features live in different spaces, so naively concatenating them or using cross-attention wastes the visual information. They propose treating the alignment of audio and visual tokens as an optimal transport problem, computing a soft assignment matrix between the two token sets, and using that matrix both as a training loss and as the attention weights for fusion. With a frozen CLIP visual encoder and LoRA-adapted audio encoder and Llama 2 decoder, LAVCap reports SPIDEr 51.7 and CIDEr 84.9 on the AudioCaps test set, the highest among the models compared, despite not pre-training on extra captioning datasets. The authors also report a user study in which their generated captions receive higher mean opinion scores than the ground-truth audio-only captions.","feed_headline":"With optimal transport, LAVCap beats prior AudioCaps captioners","feed_subtitle":"On AudioCaps, the audio-visual model reaches SPIDEr 51.7 without extra pretraining data or post-processing.","key_machinery":"The central object is the optimal transport assignment map $Q^*\\in\\mathbb{R}^{N_a\\times N_v}$, obtained by solving a regularized transport problem over the dot-product similarity matrix $S$ between audio and visual tokens with Sinkhorn-Knopp iterations. $Q^*$ is doubly stochastic, so each audio token is distributed over visual tokens and vice versa. The paper uses $Q^*$ twice: in $L_{OT}$ (a cross-modal contrastive loss over the row/column pairs of $Q^*$ and $S$) to pull corresponding tokens together, and in OT-Att to re-weight the other modality's features by $\\hat{h}_a = h_a + Q^* h_v$ and symmetrically for $\\hat{h}_v$, with the two attended sequences concatenated and projected into the LLM's latent space. Because $Q^*$ is computed by a fixed algorithm from $S$, the fusion module adds no learnable parameters, which the authors argue is a data-efficient way to fuse modalities when training data is scarce.","core_discovery":"On the paper's own terms, the discovery is that the optimal transport assignment map $Q^*$ between audio tokens and visual tokens is a dual-purpose object: when its row/column similarities are pushed to be large within a sample under $L_{OT}$, it coaxes a frozen CLIP visual encoder and a trainable audio encoder into a shared semantic geometry, and when $Q^*$ is used directly as attention weights in the OT-Att fusion, it lets each modality attend to the other without any learnable fusion parameters. The authors show by ablation that adding visual features without $L_{OT}$ barely helps (SPIDEr 47.6 to 48.5), while adding $L_{OT}$ brings the full model to 51.0 on the validation-visible ablation, and the final architecture reaches 51.7 on the test split. The method therefore claims to be a data-efficient recipe: the OT map replaces a learned fusion layer, and LoRA on the audio encoder and LLM decoder substitute for full fine-tuning.","pith_inferences":["A testable extension would apply the same dual-use OT map to video-text or image-text captioning, where token-level correspondences are also loose and paired data is limited.","If the OT map is truly capturing semantic correspondences, its entries should be interpretable as attention weights that highlight the visual tokens (e.g., objects or scene regions) responsible for a given audio event; visualizing these maps on the AudioCaps test clips would make the claimed alignment concrete.","The paper's success with a frozen visual encoder and LoRA-tuned audio encoder suggests the OT loss may be doing the heavy lifting of cross-modal alignment, which would imply that even weaker audio encoders could be aligned to CLIP visual features by this loss alone."],"forward_implications":["LAVCap's reported SPIDEr 51.7 and CIDEr 84.9 on the AudioCaps test split would be the best numbers among the models compared in Table I, including systems pre-trained on additional captioning datasets.","The OT loss plus OT-Att combination would make audio-visual captioning practical in data-scarce settings, since the only trainable components are LoRA adapters and a linear projector.","The OT assignment map doubles as an attention matrix, eliminating the need for a learned fusion module like a Q-Former or cross-attention in this pipeline.","Because the visual encoder is frozen, the framework suggests that any frozen visual backbone can be plugged in and aligned to audio through the same OT procedure.","The user study result, if reliable, implies that audio-visual captions can be judged as more accurate than audio-only ground truth, which would argue for collecting video-grounded captions in future datasets."],"supporting_citations":[{"why":"Provides the Sinkhorn-Knopp iterations used to compute the optimal transport assignment map Q*.","marker":"[18]"},{"why":"Introduces the idea of optimizing an assignment map between modality tokens, which the paper adapts from video-text to audio-visual alignment.","marker":"[17]"},{"why":"The AudioCaps dataset supplies the training and test clips and ground-truth captions for all reported results.","marker":"[19]"},{"why":"The pre-trained Consistent Ensemble Distillation model is used as the audio encoder.","marker":"[20]"},{"why":"Frozen CLIP ViT-L/14 provides the visual tokens that are aligned and fused with audio.","marker":"[21]"},{"why":"Llama 2 7B is the text decoder that generates the captions.","marker":"[22]"},{"why":"Low-rank adaptation is the training strategy for the audio encoder and LLM decoder in the final model.","marker":"[23]"},{"why":"The visually-aware audio captioning method provides the cross-attention baseline and the comparison point for OT-Att.","marker":"[15]"},{"why":"The audio-visual captioning method that treats fused features as text tokens, serving as another baseline in the comparison table.","marker":"[16]"}],"fun_headline_variants":["Optimal transport fuses audio and vision for top AudioCaps captions","LAVCap's OT trick: one map for alignment and fusion, beats AudioCaps SOTA","Dual-use optimal transport makes LAVCap the new AudioCaps leader","OT alignment and attention: LAVCap tops AudioCaps without extra data","One OT map aligns and fuses, pushing LAVCap past AudioCaps SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the similarity matrix $S$ between frozen CLIP visual tokens and trainable CED audio tokens meaningfully reflects semantic correspondence, because the same $S$ that produces the transport plan is also the target of the alignment loss; if the two token spaces are not comparable at the token level, the transport plan is arbitrary and the OT loss would push the audio encoder toward meaningless structure.","fun_headline_variants_meta":{"raw":{"variants":["Optimal transport fuses audio and vision for top AudioCaps captions","LAVCap's OT trick: one map for alignment and fusion, beats AudioCaps SOTA","Dual-use optimal transport makes LAVCap the new AudioCaps leader","OT alignment and attention: LAVCap tops AudioCaps without extra data","One OT map aligns and fuses, pushing LAVCap past AudioCaps SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001686,"raw_usage":{"total_tokens":6681,"prompt_tokens":943,"completion_tokens":5738,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":5629}},"tokens_in":559,"tokens_out":5738,"duration_ms":36425,"temperature":1.0,"reasoning_tokens":5629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:06:12.564169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the learned similarity matrix $S$ in Eqs. (2)-(5) with a random matrix (or an identity-like constant) while keeping the Sinkhorn iterations, losses, and all other training choices unchanged; if the resulting SPIDEr stays close to the reported 51.7 rather than collapsing toward the no-OT baseline of 48.5, the transport plan itself is not carrying the claimed alignment and the gain must come from elsewhere in the pipeline.","supporting_citations":[{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport,","cited_arxiv_id":null,"evidence_quote":"Provides the Sinkhorn-Knopp iterations used to compute the optimal transport assignment map Q*."},{"cited_title":"Multi- granularity correspondence learning from long-term noisy videos,","cited_arxiv_id":null,"evidence_quote":"Introduces the idea of optimizing an assignment map between modality tokens, which the paper adapts from video-text to audio-visual alignment."},{"cited_title":"Audiocaps: Generating captions for audios in the wild,","cited_arxiv_id":null,"evidence_quote":"The AudioCaps dataset supplies the training and test clips and ground-truth captions for all reported results."},{"cited_title":"Ced: Consistent ensemble distillation for audio tagging,","cited_arxiv_id":null,"evidence_quote":"The pre-trained Consistent Ensemble Distillation model is used as the audio encoder."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Frozen CLIP ViT-L/14 provides the visual tokens that are aligned and fused with audio."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Low-rank adaptation is the training strategy for the audio encoder and LLM decoder in the final model."},{"cited_title":"Visually-aware audio captioning with adaptive audio-visual attention,","cited_arxiv_id":null,"evidence_quote":"The visually-aware audio captioning method provides the cross-attention baseline and the comparison point for OT-Att."},{"cited_title":"A VCap: Leveraging audio-visual features as text tokens for captioning,","cited_arxiv_id":null,"evidence_quote":"The audio-visual captioning method that treats fused features as text tokens, serving as another baseline in the comparison table."}],"review_version":1}