{"id":"51d21561-d581-4e1b-a8b2-4b9df5e9d91c","arxiv_id":"2412.20872","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A weakly supervised audio-visual video parser that reweights cross-modal interaction by feature similarity and injects pseudo-label text semantics achieves higher F-scores than prior methods on LLP.","lead":"LINK is a training method for audio-visual video parsing that adaptively reweights how much audio and visual information trust each other, and uses text embeddings of pseudo-labels as semantic priors. It reports improved event-detection F-scores over previous methods on the LLP benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (11)'s trainable μ collapses to 1, making the S-LOSS reweighting vacuous; the reported gains require an unstated implementation detail.","rationale":"The reader's weakest assumption targeted whether cosine similarity is a reliable indicator of cross-modal alignment. My concern is more fundamental and is located in the same equation: Eq. (11) declares μ trainable, but the mathematical form of λ forces the optima of the total loss to μ = 1, where λ ≡ 1 and the proposed reweighting disappears. This internal inconsistency is independent of whether cosine similarity is informative; even under ideal alignment, the mechanism cannot operate as described. Because the paper releases no code and reports single runs, the ablation results cannot be used to disambiguate whether the authors actually trained μ, fixed it, or used a different formula. This does not overturn the empirical SOTA claim outright, but it makes the stated explanation of the S-LOSS contribution unsupported. The appropriate response is to require the authors to clarify Eq. (11), disclose the optimization of μ, and release code or converged weights; hence the verdict remains conditional rather than rejection. The reader's concern about cosine reliability is partially related because both concern the credibility of the S-LOSS mechanism, but the trainable-μ collapse is the sharper, falsifiable flaw. Secondary issues noted by the reader, such as the apparent typo in Eq. (3) and missing training details, reinforce the need for an implementation release, but the Eq. (11) issue is the single most load-bearing point.","tokens_in":7613,"tokens_out":8587,"duration_ms":89896,"concrete_test":"Retrain the 'w/ S-LOSS' ablation with μ fixed to 1 (equivalently, optimize μ under Eq. (12) and report its converged value). If audio segment F1 stays at 68.4, the adaptive weighting is not the cause of the gain; if it drops to the VALOR++ 68.1, Eq. (11) is inconsistent with the reported result. Additionally, report the final trained value of μ and any constraints or gradient-stopping used on it.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Equation (11) defines λ as a function of segment cosine similarity s and a trainable scalar μ. For any μ, e^{|1−μ|} ≥ 1; hence λ = e^{|1−μ|} ≥ 1 on (−0.2, 0), and on [0, 1] λ = e^{|1−μ|} + (1 − e^{|1−μ|})s = e^{|1−μ|}(1−s) + s ≥ 1, while λ = 1 for s ≤ −0.2. Thus λ ≥ 1 for all s, with the pointwise minimum λ = 1 attained at μ = 1. Since μ appears only inside λ and λ multiplies the nonnegative loss L_avss in Eq. (12), minimizing the total loss with respect to μ drives μ → 1, reducing S-LOSS to the unweighted L_avss already present in the VALOR++ baseline. The ablation rows labeled 'w/ S-LOSS' show nontrivial changes (e.g., audio segment F1 68.1 → 68.4, audio event F1 61.2 → 62.7), so either the implementation constrains, freezes, or initializes μ differently than the text states, or the published formula does not describe the actual training loss. This is a more direct threat to the central 'adaptive balancing' claim than the cosine-similarity reliability concern: even if cosine similarity perfectly tracks event alignment, a trained μ still collapses the mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LINK, a weakly-supervised audio-visual video parsing method evaluated on the LLP benchmark. It combines three components: TSAM, which applies CBAM-style temporal-spatial attention and an AV-Adapter-inspired cross-modal interaction with four trainable weights; S-LOSS, which reweights the audio-visual segment-similarity loss by the cosine similarity of audio and visual features; and PLSIM, which fuses CLIP/CLAP text-encoded pseudo-label semantics into uni-modal features. Experiments report F1 improvements over VALOR++ and other recent baselines on all reported metrics, with the largest gains in audio uni-modal event prediction. The main claim is that adaptively balancing modality contributions and injecting pseudo-label semantics suppress noise from non-aligned audio-visual events.","tokens_in":7971,"tokens_out":12336,"duration_ms":114923,"significance":"If the results are reproducible, the paper provides a concise, component-level recipe for improving audio-visual video parsing over the current state of the art. The strengths are the direct comparison with VALOR++, the component-wise ablation study, and the consistent audio-side gains. However, the significance is limited by the small margins on the joint audio-visual metrics (0.2 F1) and by a technical problem in the S-LOSS formulation, which as written degenerates under training. The pseudo-label semantics also come from a teacher trained on the same weak labels, so the 'a priori' claim is weaker than stated. These issues are fixable, but they must be addressed before the empirical claims can be taken at face value.","major_comments":[{"comment":"The weighting λ in Eq. (11) collapses to the unweighted loss during training, so the S-LOSS component cannot explain the reported gains. Since e^{|1−μ|} ≥ 1 for every real μ, the definition gives λ ≥ 1 for all s, with the global minimum λ = 1 reached at μ = 1; for s ∈ (−0.2,0) one has λ = e^{|1−μ|}, and for s ∈ [0,1] one has λ = e^{|1−μ|}(1−s)+s ≥ 1. Because μ appears only in λ and λ multiplies the nonnegative L_avss in Eq. (12), unconstrained gradient-based minimization of the total loss drives μ toward 1, reducing S-LOSS to the plain L_avss already present in the VALOR++ baseline. The ablation row 'w/ S-LOSS' in Table II (e.g., audio event F1 61.2 → 62.7) cannot be produced by Eq. (11) as written; please correct the formula or disclose the actual implementation (e.g., a frozen or constrained μ, a particular initialization, or a stop-gradient), and report the learned value of μ if it is trainable.","section":"§II-C, Eq. (11)"},{"comment":"No variance or significance information is reported. The improvements over VALOR++ on the joint audio-visual metrics are only 0.2 F1 at both segment level (62.1 vs. 61.9) and event level (55.7 vs. 55.5). With a single run, these margins are statistically uninterpretable. Please report mean and standard deviation over at least three seeds and, ideally, a paired test or exact p-values for the main comparisons and the ablation rows.","section":"§III-B, Table I"},{"comment":"The pseudo-labels used by PLSIM come from VALOR++ [8], whose teacher was trained on the same LLP weak labels, so the semantic priors are not independent of the benchmark. This does not by itself invalidate the comparison to VALOR++, which also consumes pseudo-labels, but it weakens the paper's claim that PLSIM injects 'a priori knowledge' from language. Please discuss this dependence and, if feasible, include a sensitivity experiment with pseudo-labels from a teacher trained on a disjoint source or with corrupted pseudo-labels.","section":"§II-D, §III-A"},{"comment":"The ablation labeled 'w/ TSAM(T)' bundles temporal-spatial attention with the four adaptive cross-modal interaction weights (α1, α2, β1, β2) of CMIM, so it does not isolate the mechanism that supposedly balances modality contributions. Please ablate the CMIM weights separately (for example, by fixing them to 1 or 0) and report the learned values of α1, α2, β1, β2; without this, the reader cannot tell whether the model actually learns an adaptive balance or simply benefits from the attention module.","section":"§III-C, Table II"}],"minor_comments":[{"comment":"Equation (3) uses f^a_t in the second factor of the definition of \\hat f^m_t for both m=a and m=v; this appears to be a typo for f^m_t, and the two attention branches should be written symmetrically.","section":"§II-B, Eq. (3)"},{"comment":"In Eqs. (4) and (5), the softmax arguments are written as \\hat f^a_t \\hat f^v_t without a transpose or a specified dimension; please clarify whether the softmax is over the temporal dimension or the feature dimension.","section":"§II-B, Eqs. (4)–(5)"},{"comment":"The definition of r as the intersection over union of audio and visual events should state whether it is computed per segment or per event category; the alignment between s and r in the MSE needs to be explicit.","section":"§II-C, Eq. (10)"},{"comment":"The pseudo-label notation \\hat y^{mgt}_t is unclear; it is later replaced by \\hat y^m_t in §II-D, so please standardize the superscripts.","section":"§II-C, Eq. (8)"},{"comment":"The experimental setup does not report the optimizer, learning rate, batch size, number of epochs, or initialization of μ and the four CMIM weights; these details are necessary for reproducibility, and a code-release statement would help.","section":"§III-A"},{"comment":"Reference [13] is cited with the year 2019, but the CLAP paper (Wu et al.) was published in 2023; please verify and update the bibliographic entry.","section":"References"},{"comment":"The module is called 'temporal attention' in Eq. (1) but 'channel attention' in the paragraph after Eq. (2); please use consistent terminology.","section":"§II-B, text after Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is the S-LOSS collapse; I would not recommend rejecting the paper outright because the issue is potentially a correctable typo or an unstated implementation detail. However, the authors must provide the corrected formulation, the learned parameter values, and multi-seed statistics. The paper is on the incremental side for a top venue, but it is within scope for a specialist journal. If the authors cannot produce a corrected S-LOSS or cannot show that the learned μ differs from 1, the central 'adaptive balancing' claim would not be supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid incremental result for audio-visual parsing on LLP, but the main 'adaptive' weighting loss is mathematically vacuous as written. The trainable μ in Eq. (11) will collapse to 1 during training, at which point λ=1 for every segment and S-LOSS becomes the same unweighted L_avss already present in the VALOR++ baseline. Yet the ablation shows S-LOSS alone moving audio event F1 from 61.2 to 62.7. So either the published equation is not what the code does, or an implementation detail (freezing, constraint, initialization) is missing. That is load-bearing, because the paper sells 'adaptive balancing' as its core idea.\n\nWhat's genuinely new: the combination of a CBAM-style temporal-spatial attention, an AV-Adapter variant with four learned fusion weights, a cosine-similarity loss reweighting, and pseudo-label text semantics fused through MLPs. The PLSIM idea is a nice touch and seems to be what drives most of the audio uni-modal gain. The Table I comparison shows consistent improvements over VALOR++ on every reported metric, which, if the implementation is correct, makes this a credible SOTA. The ablation is thorough and covers the components in pairs.\n\nSoft spots beyond the μ collapse: no code or config, single-run numbers with no error bars, and joint audio-visual metrics move only 0.2 F1, so the headline gain is entirely uni-modal. Eq. (3) has a typo: the second term uses f_a where f_m should appear. Training details for μ, α, β, and the four MLPs are absent.\n\nThe paper deserves a serious referee because the empirical direction is reasonable and the flaw is fixable. But it is not publishable as-is; the authors need to either correct the S-LOSS equation, constrain μ in a principled way, or report real hyperparameters and release code. I would not desk-reject it, but I'd condition acceptance on resolving the contradiction and adding error bars.\n\nWho this is for: researchers working on weakly supervised audio-visual parsing, especially anyone building on VALOR++. It's an incremental paper, not a conceptual breakthrough.\n\nMy recommendation: send to peer review, with a strong request to resolve the S-LOSS collapse and release code. If the authors can show the weighting actually behaves as intended, it becomes a usable contribution.","headline":"A plausible empirical extension of VALOR++ on LLP, but the main 'adaptive' loss as written is mathematically vacuous—the trainable μ collapses to 1—so the ablation results imply the published formula is not what was implemented.","tokens_in":8472,"tokens_out":4302,"would_cite":false,"duration_ms":39967,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LINK claims that adaptively balancing audio and visual contributions and injecting pseudo-label semantics suppresses cross-modal noise, raising audio event F1 on the LLP dataset to 63.4 from 61.2.","keywords":["audio-visual video parsing","weakly-supervised learning","modality misalignment","adaptive modality interaction","pseudo-labels","CLIP","CLAP","temporal-spatial attention"],"falsifier":"Measure the correlation between the cosine similarity used in Eq. 11 and human-annotated audio-visual alignment on the LLP validation set (where segment-level labels exist), then retrain LINK with the reweighting function replaced by a constant weight; if the correlation is not positive and removing the weighting does not hurt audio F1, the central mechanism is unsupported.","tokens_in":7426,"feed_emoji":"🎬","tokens_out":8175,"duration_ms":70443,"temperature":0.7,"pith_summary":"The paper proposes LINK, a weakly-supervised audio-visual video parsing method built on the observation that audio and visual events in real videos are often misaligned, so naively fusing the two modalities injects noise. LINK claims to fix this by learning to balance modality contributions: a temporal-spatial attention module and adaptively weighted cross-modal interaction decide how much each modality should trust the other per segment, and a cosine-similarity-based loss reweights training samples so that boundary-like, mildly misaligned segments dominate optimization. A third module converts CLIP/CLAP pseudo-labels into text-encoded semantic features and fuses them into each modality's features as prior knowledge, suppressing the other modality's noise. On the LLP benchmark, LINK reports the highest audio-segment F1 (69.7) and audio-event F1 (63.4) among compared methods, and improves audio-visual event parsing over the VALOR++ baseline. If these results hold, adaptive interaction weighting is a more direct response to modality misalignment than stronger fusion alone.","feed_headline":"LINK lifts audio event F1 to 69.7 with adaptive fusion","feed_subtitle":"Weighting fusion by cross-modal similarity and adding pseudo-label semantics lifts audio event F1 to 63.4.","key_machinery":"The method has three load-bearing components. The temporal-spatial attention module (TSAM) applies channel and spatial attention over segment features and then passes them through a cross-modal interaction module with four learnable weights ($\\alpha_1, \\alpha_2, \\beta_1, \\beta_2$) that scale how much one modality's attended features are added to the other. The segmented audio-visual semantic similarity loss (S-LOSS) reweights the audio-visual similarity term $\\lambda L_{\\text{avss}}$ with a piecewise weight $\\lambda$ based on the cosine similarity of segment features, boosting the $(-0.2, 0)$ range and down-weighting clearly similar or clearly dissimilar pairs. The pseudo-label semantic interaction module (PLSIM) reads event categories from uni-modal pseudo-labels, converts them to text prompts such as 'A photo of …' and 'this is a sound of …', encodes these with frozen CLIP/CLAP text encoders, and fuses the resulting semantic features into the audio and visual branches via multiplicative scaling and additive bias. Each module targets the same failure mode—misaligned modalities contaminating prediction—from a different angle.","core_discovery":"The central claim is that, for weakly-supervised audio-visual video parsing, the harmful effect of non-aligned audio and visual events can be reduced by making every stage of interaction aware of how well the two modalities agree. LINK does this in three places: it reweights the audio-visual feature similarity loss with a piecewise function of the cosine similarity between CLIP and CLAP segment features, giving extra weight to the $(-0.2, 0)$ range where interaction is ambiguous rather than clearly helpful or harmful; it lets four trainable parameters control the strength of each cross-modal attention path; and it injects semantic features derived from uni-modal pseudo-labels into each modality's final representation. The paper reports that on the LLP dataset this combination raises audio segment-level F1 from 68.1 to 69.7 and audio event-level F1 from 61.2 to 63.4 compared with VALOR++, with smaller gains on visual and joint audio-visual metrics.","pith_inferences":["Inference: the cosine-similarity gating in Eq. 11 is a form of curriculum over alignment; one could test it against an oracle gating that uses the true segment-level labels to see how much of the gain comes from the particular choice of the $(-0.2, 0)$ range.","Inference: the approach suggests that CLIP/CLAP text encoders can serve as a shared semantic space for de-noising both modalities; a natural test is using the same module with different backbone encoders to see if the benefit transfers.","Inference: the conclusion notes that multi-modal event prediction is left for future work, which implies the adaptive-interaction gains are concentrated in uni-modal branches, so the method's next test is whether joint-event accuracy can be improved by the same mechanisms.","Inference: because the improvements over VALOR++ are concentrated in audio metrics, the method implicitly challenges the common practice of symmetric fusion and predicts that asymmetric fusion will generalize better on other misaligned multimodal benchmarks."],"forward_implications":["If the reweighting truly isolates useful boundary samples, similar piecewise loss schedules could be applied to other weakly-supervised multimodal tasks, such as audio-visual event localization or sound source separation.","The pseudo-label semantic interaction module implies that frozen text encoders can act as a denoising prior for uni-modal branches; this could be extended to other label spaces beyond the 25 LLP event classes.","Since the gains are largest on audio uni-modal events and smaller on audio-visual joint events, the method suggests that misalignment chiefly hurts modality-specific predictions, and that joint-event prediction may need a different remedy.","The four learnable interaction weights provide a per-model estimate of how much each modality should trust the other, which could be analyzed as a diagnostic of dataset alignment."],"supporting_citations":[{"why":"Defines the LLP dataset and the weakly-supervised video parsing task, supplying the benchmark all comparisons use.","marker":"[5]"},{"why":"Provides the CLIP/CLAP feature extraction and pseudo-label generation that LINK builds on, and is the strongest baseline in Table I.","marker":"[8]"},{"why":"Supplies the convolutional block attention design that TSAM adapts for segment-level temporal-spatial attention.","marker":"[10]"},{"why":"Contributes the AV-Adapter structure that CMIM uses for cross-modal interaction with learnable gating.","marker":"[11]"},{"why":"Provides the CLIP model whose text encoder and visual encoder generate visual features and semantic embeddings.","marker":"[12]"},{"why":"Provides the CLAP model whose audio encoder and text encoder generate audio features and semantic embeddings.","marker":"[13]"},{"why":"Supports the premise that multimodal predictions are not always better than uni-modal ones, motivating the reweighting.","marker":"[14]"},{"why":"Documents difficulty in training multimodal classification networks, used to justify the need for balancing modality contributions.","marker":"[15]"},{"why":"Supplies evidence that cross-modality gradient harmonization issues and misaligned data become noise, the failure mode LINK targets.","marker":"[16]"}],"fun_headline_variants":["LINK adapts fusion weights to cut misalignment noise in audio-visual parsing","LINK reweights audio-video similarity to boost audio event F1 to 63.4","LINK adapts cross-modal attention using pseudo-label semantics to cut noise","Non-aligned audio-visual? LINK equilibrates interaction to improve parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the cosine similarity between the CLIP audio feature and the CLAP visual feature of a segment reliably indicates whether cross-modal interaction helps that segment's prediction; if the similarity does not track true event alignment, the reweighting in Eq. 11 will amplify the wrong samples.","fun_headline_variants_meta":{"raw":{"variants":["LINK adapts fusion weights to cut misalignment noise in audio-visual parsing","LINK reweights audio-video similarity to boost audio event F1 to 63.4","LINK adapts cross-modal attention using pseudo-label semantics to cut noise","Non-aligned audio-visual? LINK equilibrates interaction to improve parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3320,"prompt_tokens":842,"completion_tokens":2478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2391}},"tokens_in":458,"tokens_out":2478,"duration_ms":15471,"temperature":1.0,"reasoning_tokens":2391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:07:47.241638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the correlation between the cosine similarity used in Eq. 11 and human-annotated audio-visual alignment on the LLP validation set (where segment-level labels exist), then retrain LINK with the reweighting function replaced by a constant weight; if the correlation is not positive and removing the weighting does not hurt audio F1, the central mechanism is unsupported.","supporting_citations":[{"cited_title":"Unified multisensory per- ception: Weakly-supervised audio-visual video parsing,","cited_arxiv_id":null,"evidence_quote":"Defines the LLP dataset and the weakly-supervised video parsing task, supplying the benchmark all comparisons use."},{"cited_title":"Modality-independent teachers meet weakly-supervised audio-visual event parser","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP/CLAP feature extraction and pseudo-label generation that LINK builds on, and is the strongest baseline in Table I."},{"cited_title":"Cbam: Convo- lutional block attention module,","cited_arxiv_id":null,"evidence_quote":"Supplies the convolutional block attention design that TSAM adapts for segment-level temporal-spatial attention."},{"cited_title":"Towards Efficient Audio-Visual Learners via Empowering Pre-trained Vi- sion Transformers with Cross-Modal Adaptation,","cited_arxiv_id":null,"evidence_quote":"Contributes the AV-Adapter structure that CMIM uses for cross-modal interaction with learnable gating."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP model whose text encoder and visual encoder generate visual features and semantic embeddings."},{"cited_title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the CLAP model whose audio encoder and text encoder generate audio features and semantic embeddings."},{"cited_title":"Balanced multimodal learning via on-the-fly gradient modulation,","cited_arxiv_id":null,"evidence_quote":"Supports the premise that multimodal predictions are not always better than uni-modal ones, motivating the reweighting."},{"cited_title":"What makes training multi-modal classification networks hard?","cited_arxiv_id":null,"evidence_quote":"Documents difficulty in training multimodal classification networks, used to justify the need for balancing modality contributions."},{"cited_title":"Scal- ing multimodal pre-training via cross-modality gradient harmonization,","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that cross-modality gradient harmonization issues and misaligned data become noise, the failure mode LINK targets."}],"review_version":1}