{"id":"6fe3d4ff-9f51-409b-8ef9-0deed0c899ee","arxiv_id":"2411.11454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A relevance-guided audio-visual fusion network (AVRSP) improves video saliency prediction on six datasets by adaptively suppressing irrelevant audio and combining multi-scale visual features.","lead":"This paper introduces AVRSP, a neural network that predicts where people look in videos by combining audio and visual features, with a learned module that reduces the influence of audio when it is irrelevant, such as background music. The model reports small but consistent improvements over prior audio-visual saliency methods on six eye-tracking datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training protocol ambiguity in §IV-B: joint training on all six audio-visual datasets with unspecified per-dataset splits could inflate the reported ~1.58% gain over CASP-Net; the SOTA claim is unverified until baselines are run under an identical protocol.","rationale":"The reader's weakest assumption is that AVRSP and the prior methods were trained and evaluated under the same protocol. The manuscript text supports this concern: §IV-B describes training 'on six audio-visual datasets' without specifying per-dataset splits, while §IV-E shows that per-dataset training is possible and used in ablations. This is the most load-bearing issue because the central claim is a comparative performance advantage; if the protocol is unmatched, the numbers in Tables I and II cannot be interpreted as architectural gains. The dimensional mismatch in Eq. 6 is also a real reproducibility gap, but it is secondary because a corrected equation could preserve the architecture's functionality. The lack of statistical significance tests further weakens the strength of the claim, but protocol ambiguity is the primary threat to validity. I agree with the reader's conditional verdict: the paper should be accepted only after the protocol is clarified and the comparison is rerun under matched conditions.","tokens_in":15602,"tokens_out":4639,"duration_ms":47229,"concrete_test":"Ask the authors to (1) state whether the 100-epoch audio-visual training uses each dataset's official training split only, and (2) reproduce Tables I-II with AVRSP and every baseline trained per-dataset under identical epochs, optimizer, and input sampling, reporting seed variance. If the average Ours(AV)-CASP-Net(AV) margin falls below roughly one standard deviation across seeds, the claimed improvement is not significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (Tables I-II, §IV-D) rests on a training protocol that is never specified. §IV-B states only that after DHF1K pretraining the model is 'trained on six audio-visual datasets for 100 epochs,' and §IV-A gives a train/test split only for DIEM. If AVRSP is trained jointly on all six datasets while the CASP-Net/ViNet/TSFP-Net baselines are trained per dataset on each training split, then AVRSP sees strictly more training data per evaluation, and the reported average gains (0.97% visual, 1.58% audio-visual) could be a data-advantage artifact, not evidence for the RAVF/MS/MRG modules. The ablation study (§IV-E) confirms the authors know how to use per-dataset splits ('training on the DIEM and AVAD training sets'), but the main tables omit this detail. Without a precise statement of train/test splits and a matched-protocol rerun of all baselines, the central claim is not falsifiable from the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes AVRSP, a neural network for audio-visual video saliency prediction. The visual branch is a 3D S3D backbone with ASPP and LIP, and the audio branch is SoundNet; these are fused by a Relevance-guided Audio-Visual Fusion module (RAVF) that uses cross-attention with learned per-head relevance weights. Two additional modules, Multi-scale feature Synergy (MS) and Multi-scale Regulator Gate (MRG), enhance multi-scale visual features. The model is pre-trained on DHF1K and then trained on six audio-visual datasets. Experiments report results on DIEM, ETMD, AVAD, Coutrot1, Coutrot2, and SumMe, claiming average improvements of approximately 0.97% for the visual-only variant and 1.58% for the audio-visual variant over CASP-Net. Ablation studies on DIEM and AVAD validate the proposed modules and fusion strategy.","tokens_in":15869,"tokens_out":7223,"duration_ms":66738,"significance":"The paper addresses a real and relevant problem: audio-visual saliency prediction in the presence of irrelevant audio, such as background music. The RAVF module is a plausible mechanism for adaptively gating audio features based on cross-modal relevance, and the MS/MRG modules are reasonable extensions for integrating multi-scale visual features. The paper provides extensive experiments on six publicly available datasets, ablation studies for both the modules and alternative fusion methods, and visualizations of the learned relevance weights. If the training protocol is clarified and the dimensional issues in the fusion equations are resolved, the contribution would be a useful, if incremental, advance to the field. The current manuscript, however, does not fully support the central performance claim because of the underspecified training protocol and the ambiguities in the core fusion formulation.","major_comments":[{"comment":"The formulation of the RAVF module is dimensionally underspecified. The visual feature X0 is a 4D tensor with spatial dimensions H/32 × W/32 and temporal dimension Tv/8, while the audio feature Fa is a 1D sequence of length Ta. In Eqs. (4)-(5), v2a_attn therefore has the token count of the visual feature and a2v_attn has the token count of the audio feature. Eq. (6) concatenates these two tensors, but no pooling, reshaping, or token-count equalization is described, so the concatenation is only valid if the token counts coincide, which is not stated. Eq. (7) then multiplies Cweights with v2a_attn; if the concatenation was along the token dimension, the lengths differ and the product is undefined, while if it was along the channel dimension, the channel counts differ. The authors should specify the exact tensor shapes and operations at each step, including how the per-head weights are produced.","section":"§III-B Eqs. (1)-(7)"},{"comment":"The training protocol for the main comparison is not specified. §IV-B says the visual branch is trained on DHF1K and then 'combined it with the audio branch and trained it on six audio-visual datasets for 100 epochs,' but it does not state whether the six datasets are pooled into one joint training set or trained per dataset, nor does it give train/test splits for Coutrot1, Coutrot2, AVAD, ETMD, and SumMe (only DIEM is split in §IV-A). The ablation study in §IV-E uses only the DIEM and AVAD training sets. If the main model is trained jointly on all six datasets while the baselines in Tables I-II are trained per dataset, AVRSP has a substantial data advantage, and the reported average gains (0.97% and 1.58% over CASP-Net in §IV-D) may not reflect the proposed modules. The authors should report the exact splits and training protocol, and ideally re-evaluate the baselines under the same protocol.","section":"§IV-B, §IV-A, §IV-E"},{"comment":"The central performance claim is an average improvement of approximately 0.97% and 1.58% 'across all metrics' on the six datasets. The per-dataset results are not uniformly positive: for example, Ours(V) on ETMD has AUC-J 0.930 versus 0.938 for CASP-Net(V), and Ours(AV) on SumMe has AUC-J 0.903 versus 0.907 for CASP-Net(AV). Averaging across metrics and datasets without reporting per-metric standard deviations or significance tests makes the improvement claim difficult to interpret. The authors should report the per-metric average differences and, if possible, statistical significance (e.g., paired tests across videos or datasets) for the main comparison.","section":"§IV-D, Tables I-II"}],"minor_comments":[{"comment":"The network name appears as 'A VRSP' in the abstract and 'AVRSP' elsewhere; use a single consistent spelling throughout the manuscript.","section":"Abstract and throughout"},{"comment":"The concatenation operation 'Cat' is not defined; state explicitly whether the concatenation is along the channel or sequence dimension and provide the resulting tensor shape.","section":"§III-B, Eq. (6)"},{"comment":"The sentence 'The introduction of RetA and RetV represents the retention levels of the audio-value AV and visual-value VV' is misleading, since RetA and RetV are attention score matrices rather than scalar retention levels; rephrase for clarity.","section":"§III-B, after Eq. (1)"},{"comment":"The AUC-J metric is described only informally; provide the standard definition or a reference for the specific variant used.","section":"§IV-C"},{"comment":"The reverse window strategy is described only for the video frames; clarify how the corresponding audio segments are handled for the first 31 frames.","section":"§IV-B, Testing Details"},{"comment":"The tensor dimensions in Figure 2 are difficult to read in the preprint; ensure all dimension annotations are legible and consistent with the text.","section":"Figure 2"},{"comment":"The related work discusses MTCAM [27], but this method is not included in the experimental comparison; either add it to the experiments or remove the mention.","section":"§II-B, References [26],[27]"},{"comment":"The paper would benefit from a statement on code and model availability, as the current experimental details are insufficient for replication.","section":"§IV-B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a plausible architecture but the experimental reporting is not yet at the level expected for acceptance. The main concern is that the training protocol for the principal comparison (joint training across six datasets) differs from the ablation protocol (two datasets), and this disparity is not acknowledged or justified. The dimensional ambiguity in Eq. (6)-(7) also suggests the manuscript needs careful revision. Given the small average improvements reported, I recommend that the editor require the authors to clarify the protocol, report per-dataset splits and per-metric variations, and ideally release code or detailed configuration files to make the results verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent, incremental architecture paper with a real ambiguity about training protocol that clouds the headline numbers.\n\nWhat's actually new: the RAVF module replaces softmax with explicit retention scores (RetA/RetV) and combines cross-attention outputs with adaptive per-head weights; the MS and MRG modules are a reasonable way to propagate multi-scale visual and fused features. The ablation study in Tables III-V is the paper's strongest asset: each added module gives a consistent bump on DIEM and AVAD, and the comparison against simple fusion baselines (element-wise, bilinear, MBT) shows RAVF is doing something. That is honest work.\n\nThe soft spots are real but mostly fixable. The biggest is the training protocol in Section IV-B. The authors say they pre-train the visual branch on DHF1K, then train on 'six audio-visual datasets for 100 epochs.' They give train/test split only for DIEM. Baselines like CASP-Net and ViNet typically train per dataset on each training split. If AVRSP is trained jointly on all six datasets, it sees more training data per evaluation, and the reported ~1.58% average gain over CASP-Net could be a data-advantage artifact. The authors need to state the exact splits for all datasets and ideally rerun a matched-protocol baseline. This is not a fatal flaw, but it means the central claim is not fully falsifiable from the paper as written.\n\nSecond, Eq. 6 concatenates v2a_attn and a2v_attn, which may have different spatial/temporal resolutions. No pooling is described. Possibly a hidden reshape/pooling exists in code, but the paper should say so. Third, there are no significance tests across datasets; with gains of 1-2%, error bars would help. No code is provided, which limits reproducibility.\n\nThe circularity concern raised by the reader is not really a problem: supervised training on the same benchmark is standard practice, and the learned relevance scores are a post-hoc interpretation, not a derivation. That's fine.\n\nBottom line: this paper is for researchers working on audio-visual saliency, and it deserves a serious referee. The architecture is reasonable, the ablations are informative, and the topic is under-served. But the protocol ambiguity and the dimension mismatch need to be resolved before the comparative claim can be trusted. I would send it to review with a request for major revision.","headline":"Competent incremental architecture with informative ablations, but the training-protocol ambiguity blurs the headline improvement over CASP-Net.","tokens_in":16305,"tokens_out":3248,"would_cite":false,"duration_ms":31050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that relevance-gated audio fusion improves audio-visual video saliency prediction, with average gains of about 0.97% and 1.58% over CASP-Net on six eye-movement datasets.","keywords":["audio-visual saliency prediction","video saliency","relevance-guided fusion","cross-attention","multi-scale feature synergy","modality inconsistency","eye movement datasets","gaze prediction"],"falsifier":"Re-run the six-dataset comparison with every method trained on the same per-dataset training splits, or jointly on the same six training sets with the same DHF1K pre-training, and identical test sets; if the average gains over CASP-Net shrink to noise, the relevance-gating claim is not supported. A second check is to present videos with a deliberately mismatched audio track and inspect whether the retention maps drop toward zero and whether the saliency map remains unchanged.","tokens_in":15447,"feed_emoji":"🎯","tokens_out":8629,"duration_ms":82666,"temperature":0.7,"pith_summary":"Audio-visual video saliency prediction tries to forecast where people look while watching video with sound, and this paper argues that the standard practice of fusing audio and visual features directly is wrong when the soundtrack is semantically unrelated to what is on screen, as with background music. The proposed AVRSP network therefore gates audio features by a learned relevance score before merging them with visual features, using cross-attention retention weights, and supplements this with multi-scale visual feature synergy and a regulator gate. The authors claim this design reaches better saliency prediction than prior methods on six audio-visual eye-tracking datasets, reporting average gains of about 0.97% for the visual-only variant and 1.58% for the audio-visual variant over CASP-Net across all metrics. If true, the result offers a simple architectural principle: let the match between modalities decide how much audio should influence gaze prediction.","feed_headline":"Audio gated by relevance improves video saliency prediction","feed_subtitle":"Gating audio by its relevance to the visuals cuts background-music interference, with gains on six gaze datasets","key_machinery":"The load-bearing object is the Relevance-guided Audio-Visual feature Fusion (RAVF) block, a transformer-style cross-attention module whose retention maps $\\mathrm{Ret}_A = V_Q A_K^\\top / \\sqrt{d_k}$ and $\\mathrm{Ret}_V = A_Q V_K^\\top / \\sqrt{d_k}$ replace softmax and act as explicit gates on audio and visual values, with per-head weights $C_{\\mathrm{weights}} = \\mathrm{Softmax}(\\mathrm{MLP}(\\mathrm{Cat}(v2a_{\\mathrm{attn}}, a2v_{\\mathrm{attn}})))$. This mechanism directly controls how much audio survives into the fused representation, which is what lets the model suppress background music, while the Multi-scale feature Synergy and Multi-scale Regulator Gate modules refine and route the multi-scale visual features to the saliency decoder.","core_discovery":"The paper's central claim is that semantic relevance between audio and visual streams--not mere temporal synchronization--should govern audio-visual fusion for saliency prediction. AVRSP operationalizes this with a Relevance-guided Audio-Visual feature Fusion module in which two cross-attention retention maps, $\\mathrm{Ret}_A$ and $\\mathrm{Ret}_V$, measure how much each modality should keep of the other, replacing the softmax over attention scores so the retained amount is directly readable. The Multi-scale feature Synergy and Multi-scale Regulator Gate modules then enrich and route the fused information to the decoder. Evaluated on DIEM, Coutrot1, Coutrot2, AVAD, ETMD, and SumMe, the network outperforms the compared baselines, with the largest margins on DIEM and AVAD; the authors interpret this as evidence that relevance-guided fusion helps most on diverse, dynamic content.","pith_inferences":["A direct extension the paper does not run is to take a video with speech and swap in unrelated ambient audio, then check whether the retention maps fall and the prediction stays on the visual subject; this would isolate semantic relevance from low-level audio-visual correlation.","The RAVF gating scheme could be lifted into other multimodal tasks, such as sound-source localization or video captioning, where background audio is a known failure mode, since the retention maps give a per-frame measure of modality trust.","Because the absolute gains are under two percentage points averaged over metrics, the durable contribution is likely the design principle--relevance as an explicit gate--rather than a large accuracy margin on existing benchmarks."],"forward_implications":["Audio-visual saliency models can stop treating all synchronized audio as useful; gating by relevance should prevent background-music scenes from pulling predictions off the visual subject.","The visual-only variant also improves over visual-only baselines, indicating that the multi-scale synergy and regulator gate contribute independently of audio.","On datasets with complex dynamic content such as DIEM and AVAD, the relevance-guided fusion produces the largest gains, suggesting the benefit grows with scene diversity.","The reverse-window testing strategy lets the 32-frame model produce saliency maps for every frame, including the first 31, without retraining.","Ablation results place RAVF above element-wise addition, multiplication, concatenation, bilinear fusion, and MBT, indicating that the relevance-gating design, not multi-modality alone, drives the improvement."],"supporting_citations":[{"why":"Primary comparison baseline; the central performance claim is expressed as an average improvement over CASP-Net's visual-only and audio-visual results.","marker":"[4]"},{"why":"Strong baseline and source of the reverse-window evaluation strategy used to predict saliency for every frame of a test video.","marker":"[2]"},{"why":"Earlier audio-visual fusion baseline whose SoundNet-based audio encoding the paper builds upon.","marker":"[1]"},{"why":"Audio backbone that supplies the audio features fed into the RAVF fusion module.","marker":"[24]"},{"why":"Visual backbone providing the multi-scale spatio-temporal features that MS and MRG process.","marker":"[25]"},{"why":"Visual-only dataset used to pre-train the visual branch before audio-visual fine-tuning.","marker":"[6]"},{"why":"Recent audio-visual baseline compared in the main results tables.","marker":"[26]"},{"why":"Baseline in the comparison whose temporal-spatial feature pyramid is the closest prior use of multi-scale features.","marker":"[36]"},{"why":"Attention-bottleneck fusion baseline in the ablation study that RAVF is measured against.","marker":"[37]"},{"why":"One of the six audio-visual eye-tracking datasets used to evaluate the central claim.","marker":"[30]"}],"fun_headline_variants":["Audio relevance gates fusion for video saliency","Relevance gate stops wrong audio from hurting saliency","Audio only helps saliency when it's relevant to the scene","Relevance-gated audio fusion sharpens video saliency maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that AVRSP and the comparison methods, especially CASP-Net, were trained and tested under the same data splits and evaluation protocol, so the reported average gains of 0.97% and 1.58% come from the new modules and not from different training data or test sets.","fun_headline_variants_meta":{"raw":{"variants":["Audio relevance gates fusion for video saliency","Relevance gate stops wrong audio from hurting saliency","Audio only helps saliency when it's relevant to the scene","Relevance-gated audio fusion sharpens video saliency maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001032,"raw_usage":{"total_tokens":4338,"prompt_tokens":925,"completion_tokens":3413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":3348}},"tokens_in":541,"tokens_out":3413,"duration_ms":63984,"temperature":1.0,"reasoning_tokens":3348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:29:45.504687+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six-dataset comparison with every method trained on the same per-dataset training splits, or jointly on the same six training sets with the same DHF1K pre-training, and identical test sets; if the average gains over CASP-Net shrink to noise, the relevance-gating claim is not supported. A second check is to present videos with a deliberately mismatched audio track and inspect whether the retention maps drop toward zero and whether the saliency map remains unchanged.","supporting_citations":[{"cited_title":"Casp-net: Rethinking video saliency prediction from an audio-visual consistency perceptual perspective,","cited_arxiv_id":null,"evidence_quote":"Primary comparison baseline; the central performance claim is expressed as an average improvement over CASP-Net's visual-only and audio-visual results."},{"cited_title":"Vinet: Pushing the limits of visual modality for audio- visual saliency prediction,","cited_arxiv_id":null,"evidence_quote":"Strong baseline and source of the reverse-window evaluation strategy used to predict saliency for every frame of a test video."},{"cited_title":"Stavis: Spatio-temporal audio- visual saliency network,","cited_arxiv_id":null,"evidence_quote":"Earlier audio-visual fusion baseline whose SoundNet-based audio encoding the paper builds upon."},{"cited_title":"Soundnet: Learning sound representations from unlabeled video,","cited_arxiv_id":null,"evidence_quote":"Audio backbone that supplies the audio features fed into the RAVF fusion module."},{"cited_title":"Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,","cited_arxiv_id":null,"evidence_quote":"Visual backbone providing the multi-scale spatio-temporal features that MS and MRG process."},{"cited_title":"Revisiting video saliency: A large-scale benchmark and a new model,","cited_arxiv_id":null,"evidence_quote":"Visual-only dataset used to pre-train the visual branch before audio-visual fine-tuning."},{"cited_title":"From discrete representation to continuous modeling: A novel audio- visual saliency prediction model with implicit neural representations,","cited_arxiv_id":null,"evidence_quote":"Recent audio-visual baseline compared in the main results tables."},{"cited_title":"Clustering of gaze during dynamic scene viewing is predicted by motion. cognitive computation, 3 (1), 5–24,","cited_arxiv_id":null,"evidence_quote":"One of the six audio-visual eye-tracking datasets used to evaluate the central claim."}],"review_version":1}