Pith. sign in

REVIEW 3 major objections 8 minor 37 references

Relevance-guided Audio Visual Fusion for Video Saliency Prediction

T0 review · 3 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that relevance-gated audio fusion improves audio-visual video saliency prediction, with average gains of about 0.97% and 1.58% over CASP-Net on six eye-movement datasets.

desk verdict Competent incremental architecture with informative ablations, but the training-protocol ambiguity blurs the headline improvement over CASP-Net. read the letter →

arxiv 2411.11454 v1 pith:EAVCUYRK submitted 2024-11-18 cs.CV

Li Yu , Xuanzhe Sun , Pan Gao , Moncef Gabbouj This is my paper · ORCID
classification cs.CV
keywords audio-visualsaliencypredictionvideorelevance-guidedfusioncross-attentionmulti-scalefeaturesynergymodalityinconsistencyeyemovementdatasetsgaze
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Audio-visual video saliency prediction tries to forecast where people look while watching video with sound, and this paper argues that the standard practice of fusing audio and visual features directly is wrong when the soundtrack is semantically unrelated to what is on screen, as with background music. The proposed AVRSP network therefore gates audio features by a learned relevance score before merging them with visual features, using cross-attention retention weights, and supplements this with multi-scale visual feature synergy and a regulator gate. The authors claim this design reaches better saliency prediction than prior methods on six audio-visual eye-tracking datasets, reporting average gains of about 0.97% for the visual-only variant and 1.58% for the audio-visual variant over CASP-Net across all metrics. If true, the result offers a simple architectural principle: let the match between modalities decide how much audio should influence gaze prediction.

What carries the argument

The load-bearing object is the Relevance-guided Audio-Visual feature Fusion (RAVF) block, a transformer-style cross-attention module whose retention maps $\mathrm{Ret}_A = V_Q A_K^\top / \sqrt{d_k}$ and $\mathrm{Ret}_V = A_Q V_K^\top / \sqrt{d_k}$ replace softmax and act as explicit gates on audio and visual values, with per-head weights $C_{\mathrm{weights}} = \mathrm{Softmax}(\mathrm{MLP}(\mathrm{Cat}(v2a_{\mathrm{attn}}, a2v_{\mathrm{attn}})))$. This mechanism directly controls how much audio survives into the fused representation, which is what lets the model suppress background music, while the Multi-scale feature Synergy and Multi-scale Regulator Gate modules refine and route the multi-scale visual features to the saliency decoder.

What would settle it

Re-run the six-dataset comparison with every method trained on the same per-dataset training splits, or jointly on the same six training sets with the same DHF1K pre-training, and identical test sets; if the average gains over CASP-Net shrink to noise, the relevance-gating claim is not supported. A second check is to present videos with a deliberately mismatched audio track and inspect whether the retention maps drop toward zero and whether the saliency map remains unchanged.

Watch

Extended reading notes

Core claim

The paper's central claim is that semantic relevance between audio and visual streams--not mere temporal synchronization--should govern audio-visual fusion for saliency prediction. AVRSP operationalizes this with a Relevance-guided Audio-Visual feature Fusion module in which two cross-attention retention maps, $\mathrm{Ret}_A$ and $\mathrm{Ret}_V$, measure how much each modality should keep of the other, replacing the softmax over attention scores so the retained amount is directly readable. The Multi-scale feature Synergy and Multi-scale Regulator Gate modules then enrich and route the fused information to the decoder. Evaluated on DIEM, Coutrot1, Coutrot2, AVAD, ETMD, and SumMe, the network outperforms the compared baselines, with the largest margins on DIEM and AVAD; the authors interpret this as evidence that relevance-guided fusion helps most on diverse, dynamic content.

Load-bearing premise

The load-bearing premise is that AVRSP and the comparison methods, especially CASP-Net, were trained and tested under the same data splits and evaluation protocol, so the reported average gains of 0.97% and 1.58% come from the new modules and not from different training data or test sets.

Editorial extensions

If this is right

  • Audio-visual saliency models can stop treating all synchronized audio as useful; gating by relevance should prevent background-music scenes from pulling predictions off the visual subject.
  • The visual-only variant also improves over visual-only baselines, indicating that the multi-scale synergy and regulator gate contribute independently of audio.
  • On datasets with complex dynamic content such as DIEM and AVAD, the relevance-guided fusion produces the largest gains, suggesting the benefit grows with scene diversity.
  • The reverse-window testing strategy lets the 32-frame model produce saliency maps for every frame, including the first 31, without retraining.
  • Ablation results place RAVF above element-wise addition, multiplication, concatenation, bilinear fusion, and MBT, indicating that the relevance-gating design, not multi-modality alone, drives the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not run is to take a video with speech and swap in unrelated ambient audio, then check whether the retention maps fall and the prediction stays on the visual subject; this would isolate semantic relevance from low-level audio-visual correlation.
  • The RAVF gating scheme could be lifted into other multimodal tasks, such as sound-source localization or video captioning, where background audio is a known failure mode, since the retention maps give a per-frame measure of modality trust.
  • Because the absolute gains are under two percentage points averaged over metrics, the durable contribution is likely the design principle--relevance as an explicit gate--rather than a large accuracy margin on existing benchmarks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The manuscript proposes AVRSP, a neural network for audio-visual video saliency prediction. The visual branch is a 3D S3D backbone with ASPP and LIP, and the audio branch is SoundNet; these are fused by a Relevance-guided Audio-Visual Fusion module (RAVF) that uses cross-attention with learned per-head relevance weights. Two additional modules, Multi-scale feature Synergy (MS) and Multi-scale Regulator Gate (MRG), enhance multi-scale visual features. The model is pre-trained on DHF1K and then trained on six audio-visual datasets. Experiments report results on DIEM, ETMD, AVAD, Coutrot1, Coutrot2, and SumMe, claiming average improvements of approximately 0.97% for the visual-only variant and 1.58% for the audio-visual variant over CASP-Net. Ablation studies on DIEM and AVAD validate the proposed modules and fusion strategy.

Significance. The paper addresses a real and relevant problem: audio-visual saliency prediction in the presence of irrelevant audio, such as background music. The RAVF module is a plausible mechanism for adaptively gating audio features based on cross-modal relevance, and the MS/MRG modules are reasonable extensions for integrating multi-scale visual features. The paper provides extensive experiments on six publicly available datasets, ablation studies for both the modules and alternative fusion methods, and visualizations of the learned relevance weights. If the training protocol is clarified and the dimensional issues in the fusion equations are resolved, the contribution would be a useful, if incremental, advance to the field. The current manuscript, however, does not fully support the central performance claim because of the underspecified training protocol and the ambiguities in the core fusion formulation.

major comments (3)
  1. [§III-B Eqs. (1)-(7)] The formulation of the RAVF module is dimensionally underspecified. The visual feature X0 is a 4D tensor with spatial dimensions H/32 × W/32 and temporal dimension Tv/8, while the audio feature Fa is a 1D sequence of length Ta. In Eqs. (4)-(5), v2a_attn therefore has the token count of the visual feature and a2v_attn has the token count of the audio feature. Eq. (6) concatenates these two tensors, but no pooling, reshaping, or token-count equalization is described, so the concatenation is only valid if the token counts coincide, which is not stated. Eq. (7) then multiplies Cweights with v2a_attn; if the concatenation was along the token dimension, the lengths differ and the product is undefined, while if it was along the channel dimension, the channel counts differ. The authors should specify the exact tensor shapes and operations at each step, including how the per-head weights are produced.
  2. [§IV-B, §IV-A, §IV-E] The training protocol for the main comparison is not specified. §IV-B says the visual branch is trained on DHF1K and then 'combined it with the audio branch and trained it on six audio-visual datasets for 100 epochs,' but it does not state whether the six datasets are pooled into one joint training set or trained per dataset, nor does it give train/test splits for Coutrot1, Coutrot2, AVAD, ETMD, and SumMe (only DIEM is split in §IV-A). The ablation study in §IV-E uses only the DIEM and AVAD training sets. If the main model is trained jointly on all six datasets while the baselines in Tables I-II are trained per dataset, AVRSP has a substantial data advantage, and the reported average gains (0.97% and 1.58% over CASP-Net in §IV-D) may not reflect the proposed modules. The authors should report the exact splits and training protocol, and ideally re-evaluate the baselines under the same protocol.
  3. [§IV-D, Tables I-II] The central performance claim is an average improvement of approximately 0.97% and 1.58% 'across all metrics' on the six datasets. The per-dataset results are not uniformly positive: for example, Ours(V) on ETMD has AUC-J 0.930 versus 0.938 for CASP-Net(V), and Ours(AV) on SumMe has AUC-J 0.903 versus 0.907 for CASP-Net(AV). Averaging across metrics and datasets without reporting per-metric standard deviations or significance tests makes the improvement claim difficult to interpret. The authors should report the per-metric average differences and, if possible, statistical significance (e.g., paired tests across videos or datasets) for the main comparison.
minor comments (8)
  1. [Abstract and throughout] The network name appears as 'A VRSP' in the abstract and 'AVRSP' elsewhere; use a single consistent spelling throughout the manuscript.
  2. [§III-B, Eq. (6)] The concatenation operation 'Cat' is not defined; state explicitly whether the concatenation is along the channel or sequence dimension and provide the resulting tensor shape.
  3. [§III-B, after Eq. (1)] The sentence 'The introduction of RetA and RetV represents the retention levels of the audio-value AV and visual-value VV' is misleading, since RetA and RetV are attention score matrices rather than scalar retention levels; rephrase for clarity.
  4. [§IV-C] The AUC-J metric is described only informally; provide the standard definition or a reference for the specific variant used.
  5. [§IV-B, Testing Details] The reverse window strategy is described only for the video frames; clarify how the corresponding audio segments are handled for the first 31 frames.
  6. [Figure 2] The tensor dimensions in Figure 2 are difficult to read in the preprint; ensure all dimension annotations are legible and consistent with the text.
  7. [§II-B, References [26],[27]] The related work discusses MTCAM [27], but this method is not included in the experimental comparison; either add it to the experiments or remove the mention.
  8. [§IV-B] The paper would benefit from a statement on code and model availability, as the current experimental details are insufficient for replication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's empirical comparisons and ablations are self-contained, and the only flagged issue is an unverified training-protocol detail, which is a reproducibility concern rather than a circular derivation.

full rationale

The paper's central claim is an empirical performance comparison: AVRSP versus prior models on six audio-visual datasets. The proposed modules (RAVF, MS, MRG) are defined by explicit forward-pass equations (Eqs. 1-15) and trained with a supervised loss (Eq. 19) on ground-truth saliency maps; at test time the saliency predictions are compared with held-out eye-tracking data. Nothing in the methodology defines the predicted saliency map as the optimized value of a fitted parameter that is later reported as a prediction. The relevance weights RetA and RetV are learned intermediate activations of cross-attention, not quantities fitted to the final metric, so the interpretation of them as semantic relevance is a post-hoc description rather than a derivation. The comparison with CASP-Net and other baselines is vulnerable to a possible protocol mismatch because Section IV-B states only that the model was 'trained on six audio-visual datasets for 100 epochs' without specifying per-dataset train/test splits; this is a legitimate reproducibility and verifiability concern about whether the reported average improvement of about 1.58% is matched-protocol, but it is not circularity, because the baselines and the proposed method are both empirical systems evaluated on the same published datasets. The loss function overlaps with evaluation metrics (KL, CC, Sim), but this is standard supervised training rather than a constructional identity, since the model must still generalize to unseen test frames. There is one reference by an author (ref. [10], L. Yu), but it is used only as an example in the related-work survey and is not load-bearing for the paper's claims. No self-citation chain, uniqueness theorem, or ansatz-via-citation pattern is present, so the derivation chain is not equivalent to its inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method rests on standard transfer learning assumptions about S3D and SoundNet features, plus the design choice of unnormalized attention. The only hand-set numbers are the loss weights; the relevance scores themselves are learned from data.

free parameters (1)
  • Loss weights alpha1 and alpha2 = -0.1 each
    Chosen by hand in Eq. 19; no sensitivity analysis is provided.
assumptions (3)
  • domain assumption S3D features, fine-tuned on DHF1K, are suitable visual representations for saliency prediction.
    The visual backbone is pretrained on Kinetics and then fine-tuned on DHF1K (Section IV-B); this assumes transferability to the six audio-visual datasets.
  • domain assumption SoundNet features capture audio semantics that can be aligned with visual content for cross-attention.
    The audio backbone is SoundNet pretrained on unlabeled video (Section III-A); this assumes the learned audio representation shares a semantic space with S3D visual features.
  • ad hoc to paper Omitting softmax in cross-attention still yields meaningful relevance scores because of the rescaling and residual connections.
    Eqs. 2-5 replace softmax with raw dot-product retention scores; the paper provides no theoretical or empirical justification that these scores are well-calibrated for gating.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relevance-guided Audio Visual Fusion for Video Saliency Prediction." pith.science (2026). https://pith.science/paper/EAVCUYRK

@misc{pith2026241111454,
  author       = {Pith},
  title        = {Pith review of: Relevance-guided Audio Visual Fusion for Video Saliency Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAVCUYRK}},
  note         = {Machine review of arXiv:2411.11454}
}
read the original abstract

Audio data, often synchronized with video frames, plays a crucial role in guiding the audience's visual attention. Incorporating audio information into video saliency prediction tasks can enhance the prediction of human visual behavior. However, existing audio-visual saliency prediction methods often directly fuse audio and visual features, which ignore the possibility of inconsistency between the two modalities, such as when the audio serves as background music. To address this issue, we propose a novel relevance-guided audio-visual saliency prediction network dubbed AVRSP. Specifically, the Relevance-guided Audio-Visual feature Fusion module (RAVF) dynamically adjusts the retention of audio features based on the semantic relevance between audio and visual elements, thereby refining the integration process with visual features. Furthermore, the Multi-scale feature Synergy (MS) module integrates visual features from different encoding stages, enhancing the network's ability to represent objects at various scales. The Multi-scale Regulator Gate (MRG) could transfer crucial fusion information to visual features, thus optimizing the utilization of multi-scale visual features. Extensive experiments on six audio-visual eye movement datasets have demonstrated that our AVRSP network achieves competitive performance in audio-visual saliency prediction.

Figures

Figures reproduced from arXiv: 2411.11454 by the authors.

Figure 1
Figure 1. The visualization results of the saliency prediction in a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The proposed AVRSP, mainly consists of three main stages: (1) Audio and Visual Feature Extraction, where [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The illustration of the Relevance-Guided Audio-Visual Fusion (RAVF) method. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Multi-scale feature Synergy and Multi-scale Regulator [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Sample frame from Coutrot1, Coutrot2 and DIEM databases with their eye-tracking data, and the corresponding ground [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The visualization attention map of RetA and RetV in both relevant and irrelevant configurations. The brighter the area, the closer the value is to 1, indicating greater relevance. Conversely, darker areas indicate less relevance. simple fusion methods, element-wise add…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 30 canonical work pages

  1. [1]

    Stavis: Spatio-temporal audio- visual saliency network,

    A. Tsiami, P. Koutras, and P. Maragos, “Stavis: Spatio-temporal audio- visual saliency network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 4766–4776

    work page 2020

  2. [2]

    Vinet: Pushing the limits of visual modality for audio- visual saliency prediction,

    S. Jain, P. Yarlagadda, S. Jyoti, S. Karthik, R. Subramanian, and V . Gandhi, “Vinet: Pushing the limits of visual modality for audio- visual saliency prediction,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 3520–3527

    work page 2021

  3. [3]

    Audiovisual saliency prediction via deep learning,

    J. Chen, Q. Li, H. Ling, D. Ren, and P. Duan, “Audiovisual saliency prediction via deep learning,” Neurocomputing, vol. 428, pp. 248–258, 2021

    work page 2021

  4. [4]

    Casp-net: Rethinking video saliency prediction from an audio-visual consistency perceptual perspective,

    J. Xiong, G. Wang, P. Zhang, W. Huang, Y . Zha, and G. Zhai, “Casp-net: Rethinking video saliency prediction from an audio-visual consistency perceptual perspective,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6441–6450

    work page 2023

  5. [5]

    Actions in context,

    M. Marszalek, I. Laptev, and C. Schmid, “Actions in context,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2009, pp. 2929–2936

    work page 2009

  6. [6]

    Revisiting video saliency: A large-scale benchmark and a new model,

    W. Wang, J. Shen, F. Guo, M.-M. Cheng, and A. Borji, “Revisiting video saliency: A large-scale benchmark and a new model,” in Proceedings of the IEEE Conference on computer vision and pattern recognition , 2018, pp. 4894–4903

    work page 2018

  7. [7]

    Video saliency forecasting transformer,

    C. Ma, H. Sun, Y . Rao, J. Zhou, and J. Lu, “Video saliency forecasting transformer,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 10, pp. 6850–6862, 2022

    2022

  8. [8]

    Going from image to video saliency: Aug- menting image salience with dynamic attentional push,

    S. Gorji and J. J. Clark, “Going from image to video saliency: Aug- menting image salience with dynamic attentional push,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 7501–7511

    work page 2018

Show all 37 references
  1. [9]

    Predicting human eye fixations via an lstm-based saliency attentive model,

    M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, “Predicting human eye fixations via an lstm-based saliency attentive model,” IEEE Trans- actions on Image Processing , vol. 27, no. 10, pp. 5142–5154, 2018

    2018

  2. [10]

    Apnet: Adversarial learning assistance and perceived importance fusion network for all-day rgb- t salient object detection,

    W. Zhou, Y . Zhu, J. Lei, J. Wan, and L. Yu, “Apnet: Adversarial learning assistance and perceived importance fusion network for all-day rgb- t salient object detection,” IEEE Transactions on Emerging Topics in Computational Intelligence , vol. 6, no. 4, pp. 957–968, 2022

    2022

  3. [11]

    Transalnet: Towards perceptually relevant visual saliency prediction,

    J. Lou, H. Lin, D. Marshall, D. Saupe, and H. Liu, “Transalnet: Towards perceptually relevant visual saliency prediction,” Neurocomputing, vol. 494, pp. 455–467, 2022

    2022

  4. [12]

    Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks,

    X. Huang, C. Shen, X. Boix, and Q. Zhao, “Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 262–270

    2015

  5. [13]

    Shallow and deep convolutional networks for saliency prediction,

    J. Pan, E. Sayrol, X. Giro-i Nieto, K. McGuinness, and N. E. O’Connor, “Shallow and deep convolutional networks for saliency prediction,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 598–606

    2016

  6. [14]

    Deepfix: A fully convolutional neural network for predicting human eye fixations,

    S. S. Kruthiventi, K. Ayush, and R. V . Babu, “Deepfix: A fully convolutional neural network for predicting human eye fixations,” IEEE Transactions on Image Processing , vol. 26, no. 9, pp. 4446–4456, 2017

    2017

  7. [15]

    Psnet: Parallel symmetric network for video salient object detection,

    R. Cong, W. Song, J. Lei, G. Yue, Y . Zhao, and S. Kwong, “Psnet: Parallel symmetric network for video salient object detection,” IEEE Transactions on Emerging Topics in Computational Intelligence , vol. 7, no. 2, pp. 402–414, 2023

    2023

  8. [16]

    Simple vs complex temporal recurrences for video saliency prediction,

    P. Linardos, E. Mohedano, J. J. Nieto, N. E. O’Connor, X. Giro-i Nieto, and K. McGuinness, “Simple vs complex temporal recurrences for video saliency prediction,” arXiv preprint arXiv:1907.01869 , 2019

    1907 arXiv

  9. [17]

    Video saliency prediction using enhanced spatiotemporal alignment network,

    J. Chen, H. Song, K. Zhang, B. Liu, and Q. Liu, “Video saliency prediction using enhanced spatiotemporal alignment network,” Pattern Recognition, vol. 109, p. 107615, 2021

    2021

  10. [18]

    Deep audio-visual fusion neural network for saliency estimation,

    S. Yao, X. Min, and G. Zhai, “Deep audio-visual fusion neural network for saliency estimation,” in 2021 IEEE International Conference on Image Processing (ICIP) . IEEE, 2021, pp. 1604–1608

    2021

  11. [19]

    Tased-net: Temporally-aggregating spatial encoder-decoder network for video saliency detection,

    K. Min and J. J. Corso, “Tased-net: Temporally-aggregating spatial encoder-decoder network for video saliency detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 2394–2403

    2019

  12. [20]

    Hierarchical domain-adapted feature learning for video saliency prediction,

    G. Bellitto, F. Proietto Salanitri, S. Palazzo, F. Rundo, D. Giordano, and C. Spampinato, “Hierarchical domain-adapted feature learning for video saliency prediction,” International Journal of Computer Vision , vol. 129, pp. 3216–3232, 2021

    2021

  13. [21]

    Spatio-temporal saliency networks for dynamic saliency prediction,

    C. Bak, A. Kocak, E. Erdem, and A. Erdem, “Spatio-temporal saliency networks for dynamic saliency prediction,” IEEE Transactions on Mul- timedia, vol. 20, no. 7, pp. 1688–1698, 2018

    2018

  14. [22]

    Dave: A deep audio-visual embedding for dynamic saliency prediction,

    H. R. Tavakoli, A. Borji, E. Rahtu, and J. Kannala, “Dave: A deep audio-visual embedding for dynamic saliency prediction,” arXiv preprint arXiv:1905.10693, 2019

    1905 arXiv

  15. [23]

    Susinet: See, understand and summarize it,

    P. Koutras and P. Maragos, “Susinet: See, understand and summarize it,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0–0

    2019

  16. [24]

    Soundnet: Learning sound representations from unlabeled video,

    Y . Aytar, C. V ondrick, and A. Torralba, “Soundnet: Learning sound representations from unlabeled video,” Advances in neural information processing systems, vol. 29, 2016

    2016

  17. [25]

    Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,

    S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 318–335

    2018

  18. [26]

    From discrete representation to continuous modeling: A novel audio- visual saliency prediction model with implicit neural representations,

    D. Zhu, K. Zhang, K. Zhu, N. Zhang, W. Ding, G. Zhai, and X. Yang, “From discrete representation to continuous modeling: A novel audio- visual saliency prediction model with implicit neural representations,” IEEE Transactions on Emerging Topics in Computational Intelligence , 2024

    2024

  19. [27]

    Mtcam: A novel weakly-supervised audio-visual saliency prediction model with multi-modal transformer,

    D. Zhu, K. Zhu, W. Ding, N. Zhang, X. Min, G. Zhai, and X. Yang, “Mtcam: A novel weakly-supervised audio-visual saliency prediction model with multi-modal transformer,” IEEE Transactions on Emerging Topics in Computational Intelligence , 2024

    2024

  20. [28]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 4, pp. 834–848, 2017

    2017

  21. [29]

    Lip: Local importance-based pooling,

    Z. Gao, L. Wang, and G. Wu, “Lip: Local importance-based pooling,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3355–3364

    2019

  22. [30]

    Clustering of gaze during dynamic scene viewing is predicted by motion. cognitive computation, 3 (1), 5–24,

    P. Mital, T. Smith, R. Hill, and J. Henderson, “Clustering of gaze during dynamic scene viewing is predicted by motion. cognitive computation, 3 (1), 5–24,” 2011

    2011

  23. [31]

    How saliency, faces, and sound influence gaze in dynamic social scenes,

    A. Coutrot and N. Guyader, “How saliency, faces, and sound influence gaze in dynamic social scenes,” Journal of vision , vol. 14, no. 8, pp. 5–5, 2014

    2014

  24. [32]

    Multimodal saliency models for videos,

    ——, “Multimodal saliency models for videos,” From Human Attention to Computational Attention: A Multidisciplinary Approach , pp. 291–304, 2016

    2016

  25. [33]

    Fixation prediction through multimodal analysis,

    X. Min, G. Zhai, K. Gu, and X. Yang, “Fixation prediction through multimodal analysis,” ACM Transactions on Multimedia Computing, JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11 Communications, and Applications (TOMM) , vol. 13, no. 1, pp. 1–23, 2017

    2021

  26. [34]

    A perceptually based spatio-temporal com- putational framework for visual saliency estimation,

    P. Koutras and P. Maragos, “A perceptually based spatio-temporal com- putational framework for visual saliency estimation,” Signal Processing: Image Communication , vol. 38, pp. 15–31, 2015

    2015

  27. [35]

    Cre- ating summaries from user videos,

    M. Gygli, H. Grabner, H. Riemenschneider, and L. Van Gool, “Cre- ating summaries from user videos,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13 . Springer, 2014, pp. 505–520

    2014

  28. [36]

    Temporal-spatial feature pyramid for video saliency detection,

    Q. Chang and S. Zhu, “Temporal-spatial feature pyramid for video saliency detection,” arXiv preprint arXiv:2105.04213 , 2021

    2021 arXiv

  29. [37]

    Attention bottlenecks for multimodal fusion,

    A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” Advances in neural information processing systems , vol. 34, pp. 14 200–14 213, 2021

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.