Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TAVDiff treats video saliency prediction as a diffusion-based image generation task conditioned on text, audio, and visual inputs, and reports state-of-the-art results on six audiovisual datasets.

desk verdict A new tri-modal diffusion model for video saliency with mostly solid results, but the ETMD ablation contradicts the paper's own claim and the SITR text-condition selection uses target leakage during training. read the letter →

arxiv 2504.14267 v1 pith:V4DWITKU submitted 2025-04-19 cs.CV

Li Yu , Xuanzhe Sun , Wei Zhou , Moncef Gabbouj This is my paper · ORCID
classification cs.CV
keywords videosaliencypredictiondiffusionmodelmultimodallearningtext-audio-visualsaliency-orientedimage-textresponseSaliency-DiTcross-attentiongenerative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that video saliency prediction is better modeled as a conditional image-generation problem than as a direct regression, and that adding text as a third modality alongside audio and vision helps. It proposes TAVDiff, a diffusion model that denoises a saliency map from Gaussian noise under visual, audio, and text conditions. The text condition is produced by an automatic captioner (BLIP-2) and aligned to image regions by a Saliency-oriented Image-Text Response (SITR) module. A new denoising transformer, Saliency-DiT, injects the conditions through cross-attention rather than mixing them with the timestep embedding. On six audiovisual saliency datasets the method outperforms existing audio-visual and text-visual baselines, with average gains of 1.03% SIM, 2.35% CC, 2.71% NSS, and 0.33% AUC-J.

What carries the argument

The load-bearing machinery is a conditional diffusion formulation with two bespoke components. Saliency-oriented Image-Text Response (SITR) is a text-to-visual grounding step: it computes multi-head cross-attention score maps between CLIP text-token features and S3D visual features, weighs them by ground-truth saliency during training, and picks the map with maximal total response as the fused condition. Saliency-DiT is the denoising transformer: it modulates self-attention and MLP with timestep-derived scale/shift/gate parameters, and inserts a multi-head cross-attention layer between them to inject the fused image-text response and audio features, decoupling condition information from the noise-level estimate. Together they convert video, audio, and text into a conditioning vector that guides stepwise denoising (with DDIM sampling at inference) of a saliency map.

What would settle it

Compare TAVDiff's text-conditioned variant against the same model fed with scrambled or random captions on videos where the caption names an object nobody fixates; if the random-caption model matches the real-caption model, the reported text gains are not coming from semantic gaze guidance.

Watch

Extended reading notes

Core claim

The paper's central claim is that human gaze in video can be predicted by a generative diffusion model that reconstructs the saliency map from noise under tri-modal conditioning. The visual branch supplies spatio-temporal features; the audio branch supplies sound-source cues; the text branch supplies high-level semantics extracted from automatically generated captions. The novel text pathway, SITR, uses multi-head cross-attention between CLIP text features and S3D visual features, multiplies each attention map by the ground-truth saliency map during training, and selects the map with the largest pixel sum as the image-text response. The denoising network Saliency-DiT keeps timestep modulation separate from condition injection by placing a cross-attention layer between self-attention and MLP, which the ablations show improves over both U-Net and vanilla DiT backbones. The authors claim this is the first tri-modal text-audio-visual video saliency framework, and their experiments report consistent improvements across all six tested datasets.

Load-bearing premise

The load-bearing premise is that captions automatically generated from the video frames capture the semantic content that actually guides human gaze, and that the rule for picking which text-to-image attention map to use, learned with ground-truth saliency during training, still picks correctly when ground truth is unavailable at test time.

Editorial extensions

If this is right

  • Tri-modal conditioning is useful: adding text to audio-visual inputs improves all four metrics on all six datasets over the audio-visual-only variant.
  • The text pathway can be fed entirely by automatic captioning, so existing audio-visual datasets can be upgraded to tri-modal without new human annotations.
  • Decoupling timestep and condition pathways in a diffusion transformer is better than concatenating them, at least for dense prediction tasks like saliency.
  • Saliency prediction can be cast as a diffusion generation task and trained with a simple MSE denoising objective, avoiding bespoke discriminative architectures.
  • With only four DDIM steps at inference, the generative approach is practical enough for downstream use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that, because the captions come from the same frames used for prediction, the text channel might partly duplicate visual information; a fair test of semantic guidance would use captions that name objects absent from the frame or available only from context.
  • The SITR selection rule is trained with ground-truth saliency but applied at inference as a max-response rule; if the two diverge, a fine-tuning or distillation step that removes the ground-truth dependence could make the text pathway more robust.
  • The same condition-decoupling design could transfer to other conditional generation tasks, such as text-to-image or audio-driven editing, where timestep-condition mixing is also a concern.
  • Combining TAVDiff with human-written or subtitle-derived text, rather than BLIP-2 captions, on datasets that have such text would reveal whether the reported text gains generalize beyond generated captions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TAVDiff, a diffusion-based video saliency prediction model conditioned on textual, audio, and visual inputs. Visual features are extracted with S3D, audio with SoundNet, and text with a CLIP text encoder; captions are generated automatically by BLIP-2 from the input frames. A Saliency-oriented Image-Text Response (SITR) mechanism uses cross-attention to produce text-conditioned visual response maps, and a custom Saliency-DiT denoising network decouples timestep embedding from conditional information injection. The model is evaluated on six video saliency datasets (DIEM, ETMD, AVAD, Coutrot1, Coutrot2, SumMe) against audio-visual and text-visual baselines, and ablations are reported on AVAD and ETMD. The authors claim state-of-the-art results and an average improvement of 1.03%, 2.35%, 2.71%, and 0.33% on SIM, CC, NSS, and AUC-J respectively.

Significance. If the claims hold, the paper would be among the first to combine text with audio and visual modalities in a diffusion-based video saliency model, and the Saliency-DiT idea of separating timestep and condition injection is a reasonable design hypothesis. The paper provides a broad set of comparisons across four metrics and six datasets, and the ablation structure is clear. However, the central novelty depends on SITR, and the current evidence for text-conditioning benefits is weakened by a train/inference selection mismatch that leaks the target into the training-time condition, by the non-independence of the automatically generated captions, and by an internal contradiction in the ablation table that is supposed to validate Saliency-DiT. These issues are load-bearing for the main claims, so the paper needs substantial revision rather than minor editing.

major comments (4)
  1. [Section III-C, Eq. (1), Algorithm 1] During training, SITR selects the text-token attention map by maximizing the element-wise product of the attention map with the ground-truth saliency map, while at inference it selects the map with the largest unweighted pixel sum. The training-time condition is therefore an oracle function of the target S0, so the text-conditioning gains reported in Tables I, II, and V may reflect leakage of the target into the condition rather than a genuine text-to-saliency mapping. The paper provides no evidence that the max-sum selection rule used at inference agrees with the GT-weighted rule used in training; please add experiments with max-sum selection also used during training, with random token selection, and with a direct comparison of the two selection rules on the validation sets.
  2. [Table III, Section IV-E] The text states that Saliency-DiT achieves the best performance across all evaluation metrics on both AVAD and ETMD, but Table III shows that on ETMD Saliency-DiT attains SIM 0.489 and CC 0.636, which are below U-Net (0.554/0.669) and DiT (0.560/0.672). Since Table III is the primary evidence for the proposed denoising architecture, the ETMD row directly contradicts the claim and must be corrected or the architecture must be re-evaluated. This also raises questions about the visual comparisons in Table I, where ETMD results for the full model are reported with the same SIM and CC values as Saliency-DiT in this ablation.
  3. [Section III-A, Tables I and II] The text modality is not an independent information source: BLIP-2 generates captions from the same video frames that are used for prediction. Consequently, the reported gains of Ours(TAV) over Ours(AV) could be due to the additional visual-semantic processing provided by BLIP-2 and the captioning prior rather than to text as an independent modality. The central 'tri-modal' claim would be much stronger if experiments used human-provided captions or held-out text annotations, or if the authors demonstrated that corrupting or shuffling the generated captions degrades performance. Without such evidence, the text-conditioning contribution is not convincingly separated from extra model capacity.
  4. [Section III-D and Algorithms 1-2] The diffusion training and inference procedures are internally inconsistent as written. Section III-D first says the network produces the fully denoised map N0 = g_psi(Nt,Ct,t) and later says the final output N_t^5 is the feature map N_{t-1} after one round of denoising. Algorithm 1 trains with the objective ||g_psi(St,fa,ftv) - S0||^2, while Algorithm 2 feeds the network output Spred into a DDIM update. If the network predicts S0, the exact DDIM reparameterization must be specified; if it predicts N_{t-1}, the training objective is mis-stated. Please disambiguate the prediction target and provide the precise update rule used at inference.
minor comments (5)
  1. [Section III-A] The phrase 'pre-trained visual-verbal grand model' should read 'large model' or 'foundation model'; the typo 'grand model' appears to be a typographical error.
  2. [Section II-B and reference [13]] The related-work section refers to 'CaDiff' while the reference list entry and Table I use 'CaDiff' inconsistently with the title 'Cardiff'; please unify the name.
  3. [Section IV-D] The sentence reporting an 'average performance improvement of 2.08% compared to CASP(AV) and TSFP(AV)' is vague because no aggregation across metrics or datasets is defined; please specify the averaging procedure.
  4. [Section IV-B] The paper does not state which datasets are used for the DHF1K visual pre-training beyond the brief mention in Section IV-A, nor does it specify the fine-tuning protocol, splits, and number of runs; these details are necessary for reproducibility.
  5. [Figure 5] The axes and metric shown in Figure 5 are not clearly identified in the caption or in the text; please label the figure completely.

Circularity Check

1 steps flagged · score 6.0 of 10

SITR selects the text condition using the ground-truth saliency map during training, so the reported text-modality gains are partly an artifact of target leakage into the condition.

  1. self definitional [Section III-C, text following Eq. (1); Fig. 2; used in Algorithm 1 and Table V.]
    "During training, each of these attention maps is element-wise multiplied with the ground truth saliency map. From these multiplied maps, the one with the highest sum of pixel values is selected as the most representative Saliency-oriented Image-Text Response map (ftv). This selection strategy prioritizes the text feature that best highlights the salient regions within the image according to the ground truth. During inference, without access to the ground truth, the attention map with the highest sum of pixel values is directly selected as ftv."

    The text condition ftv is defined during training as the attention map that maximizes the sum of its element-wise product with the ground-truth saliency map S0. Thus the conditioning input to the denoiser is a function of the target S0. The model is trained to regress S0 from a condition that has been oracle-selected to align with S0, so the text modality's contribution in Tables I, II, and V reflects target leakage into the condition rather than an independent text-to-saliency mapping. At inference, ftv is chosen by an unweighted sum without S0; the paper provides no evidence that this rule approximates the GT-weighted rule, so the train/test condition distributions may differ. This makes the claimed text advantage partially circular by construction.

full rationale

The central tri-modal claim rests on the SITR mechanism, which during training selects the image-text response map ftv by maximizing the sum of its element-wise product with the ground-truth saliency map S0 (Section III-C). This makes the text condition a function of the target, so the model is trained to regress S0 from a condition that has been oracle-selected to correlate with S0. The ablation in Table V attributes the resulting improvement to SITR over simpler text fusion methods, but those alternatives do not receive the same GT-based selection. At inference, the selection rule changes to unweighted max-sum, with no evidence that it matches the training rule. The text-modality gains in Tables I, II, and V are therefore partially accounted for by target leakage, not by an independent text-to-saliency mapping. No other circular steps were found: the Saliency-DiT architecture, audio branch, and experimental comparisons are self-contained, and there are no load-bearing self-citations.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper relies on standard diffusion background, on the domain premise that gaze is jointly predictable from visual, audio, and text signals, and on two paper-specific assumptions: BLIP-2 captions are a valid proxy for attention-relevant semantics, and the SITR ground-truth-based selection transfers to inference. These are the main unpaid premises; no new physical entities are introduced.

free parameters (5)
  • Inference denoising steps = 4
    Chosen in Fig. 5 ablation as a performance/efficiency trade-off; all reported results use 4 steps.
  • Input video clip length = 16 frames
    Every video is truncated or sampled to 16 frames; no sensitivity analysis is reported.
  • Frame resolution = 224x384
    All frames are resized to 224x384 for the S3D backbone; fixed without reported comparison.
  • BLIP-2 caption prompt = 20-word prompt in Section III-A
    Hand-crafted prompt; no comparison with other prompts or human-written captions.
  • Diffusion timesteps = 1000
    Standard DDPM setting inherited from prior work; not tuned for saliency specifically.
assumptions (5)
  • standard math Diffusion forward and DDIM sampling equations are valid as background (Ho et al. 2020).
    Used in Algorithm 1 step 7 and Algorithm 2 line 9; accepted background but not verified in this paper.
  • domain assumption Human gaze in videos is predictable from visual, audio, and text cues jointly.
    The motivating premise of the paper (Introduction); not tested directly.
  • domain assumption BLIP-2-generated captions are a valid proxy for the semantic content that drives attention.
    Section III-A generates text from frames; no human text annotations or validation are provided.
  • ad hoc to paper The SITR selection rule using ground-truth saliency during training transfers to the max-sum rule at inference.
    Section III-C: training selects attention map with highest sum after multiplying by GT; inference selects map with highest sum without GT. This mismatch is assumed to be benign.
  • domain assumption Visual branch pretraining on DHF1K transfers to the six audiovisual datasets.
    Section IV-A states visual branch pretrained on DHF1K; transfer is assumed without direct validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction." pith.science (2026). https://pith.science/paper/V4DWITKU

@misc{pith2026250414267,
  author       = {Pith},
  title        = {Pith review of: Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4DWITKU}},
  note         = {Machine review of arXiv:2504.14267}
}
read the original abstract

Video saliency prediction is crucial for downstream applications, such as video compression and human-computer interaction. With the flourishing of multimodal learning, researchers started to explore multimodal video saliency prediction, including audio-visual and text-visual approaches. Auditory cues guide the gaze of viewers to sound sources, while textual cues provide semantic guidance for understanding video content. Integrating these complementary cues can improve the accuracy of saliency prediction. Therefore, we attempt to simultaneously analyze visual, auditory, and textual modalities in this paper, and propose TAVDiff, a Text-Audio-Visual-conditioned Diffusion Model for video saliency prediction. TAVDiff treats video saliency prediction as an image generation task conditioned on textual, audio, and visual inputs, and predicts saliency maps through stepwise denoising. To effectively utilize text, a large multimodal model is used to generate textual descriptions for video frames and introduce a saliency-oriented image-text response (SITR) mechanism to generate image-text response maps. It is used as conditional information to guide the model to localize the visual regions that are semantically related to the textual description. Regarding the auditory modality, it is used as another conditional information for directing the model to focus on salient regions indicated by sounds. At the same time, since the diffusion transformer (DiT) directly concatenates the conditional information with the timestep, which may affect the estimation of the noise level. To achieve effective conditional guidance, we propose Saliency-DiT, which decouples the conditional information from the timestep. Experimental results show that TAVDiff outperforms existing methods, improving 1.03\%, 2.35\%, 2.71\% and 0.33\% on SIM, CC, NSS and AUC-J metrics, respectively.

Figures

Figures reproduced from arXiv: 2504.14267 by the authors.

Figure 1
Figure 1. The framework of the proposed TAVDiff. The model extracts features from video, audio, and text inputs. A Saliency [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The structure of the Saliency-oriented Image-Text Response mechanism (SITR) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The structure of the denoising network Saliency-DiT [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of the proposed TAVDiff with state-of-the-art methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Ablation experiments on denoising steps Selected as 𝐹𝑡𝑣 “soccer” “man” “in the backyard” “A” “plays” Ground Truth Original Frame × Attention Vector Sequence Image-Text Response Sequence [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Visualization of SITR’s workflow. illustrates the impact of the number of iterative denoising steps on the final performance. Generally, a higher number of iterative denoising steps leads to better overall saliency prediction performance, but with diminishing returns. …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 18 canonical work pages

  1. [1]

    Roi-dvc: A region-of-interest based deep video coding framework,

    X. Wu, P. Wang, and X. Wang, “Roi-dvc: A region-of-interest based deep video coding framework,” in 2024 IEEE International Conference on Image Processing (ICIP) . IEEE, 2024, pp. 1967–1972

    work page 2024

  2. [2]

    A neural video codec with spatial rate-distortion control,

    N. Fathima, J. Petersen, G. Sauti `ere, A. Wiggers, and R. Pourreza, “A neural video codec with spatial rate-distortion control,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023, pp. 5365–5374

    work page 2023

  3. [3]

    Saliency prediction of traffic surveillance videos: A benchmark and a multi-task approach,

    W. Duan, M. Qiao, L. Jiang, M. Xu, X. Deng, S. Wen, and F. Li, “Saliency prediction of traffic surveillance videos: A benchmark and a multi-task approach,” in 2024 16th International Conference on Wireless Communications and Signal Processing (WCSP) . IEEE, 2024, pp. 1355–1360. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11

    work page 2024

  4. [4]

    Sp-det: Leveraging saliency prediction for voxel-based 3d ob- ject detection in sparse point cloud,

    P. An, Y . Duan, Y . Huang, J. Ma, Y . Chen, L. Wang, Y . Yang, and Q. Liu, “Sp-det: Leveraging saliency prediction for voxel-based 3d ob- ject detection in sparse point cloud,” IEEE Transactions on Multimedia , vol. 26, pp. 2795–2808, 2023

    work page 2023

  5. [5]

    A model of saliency-based visual at- tention for rapid scene analysis,

    L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual at- tention for rapid scene analysis,” IEEE Transactions on pattern analysis and machine intelligence , vol. 20, no. 11, pp. 1254–1259, 2002

    work page 2002

  6. [6]

    Saliency detection: A spectral residual approach,

    X. Hou and L. Zhang, “Saliency detection: A spectral residual approach,” in 2007 IEEE Conference on computer vision and pattern recognition . Ieee, 2007, pp. 1–8

    work page 2007

  7. [7]

    Multi-scale spatiotemporal feature fusion network for video saliency prediction,

    Y . Zhang, T. Zhang, C. Wu, and R. Tao, “Multi-scale spatiotemporal feature fusion network for video saliency prediction,” IEEE Transactions on Multimedia , 2023

    work page 2023

  8. [8]

    Transformer-based multi-scale feature integration network for video saliency prediction,

    X. Zhou, S. Wu, R. Shi, B. Zheng, S. Wang, H. Yin, J. Zhang, and C. Yan, “Transformer-based multi-scale feature integration network for video saliency prediction,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 12, pp. 7696–7707, 2023

    2023

Show all 38 references
  1. [9]

    Stavis: Spatio-temporal audio- visual saliency network,

    A. Tsiami, P. Koutras, and P. Maragos, “Stavis: Spatio-temporal audio- visual saliency network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 4766–4776

    2020

  2. [10]

    Vinet: Pushing the limits of visual modality for audio- visual saliency prediction,

    S. Jain, P. Yarlagadda, S. Jyoti, S. Karthik, R. Subramanian, and V . Gandhi, “Vinet: Pushing the limits of visual modality for audio- visual saliency prediction,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 3520–3527

    2021

  3. [11]

    Audio-visual saliency prediction with multisensory perception and integration,

    J. Xie, Z. Liu, G. Li, and Y . Song, “Audio-visual saliency prediction with multisensory perception and integration,” Image and Vision Computing , vol. 143, p. 104955, 2024

    2024

  4. [12]

    Mtcam: A novel weakly-supervised audio-visual saliency prediction model with multi-modal transformer,

    D. Zhu, K. Zhu, W. Ding, N. Zhang, X. Min, G. Zhai, and X. Yang, “Mtcam: A novel weakly-supervised audio-visual saliency prediction model with multi-modal transformer,” IEEE Transactions on Emerging Topics in Computational Intelligence , 2024

    2024

  5. [13]

    Cardiff: Video salient object ranking chain of thought reasoning for saliency prediction with diffusion,

    Y . Tang, G. Zhan, L. Yang, Y . Liao, and C. Xu, “Cardiff: Video salient object ranking chain of thought reasoning for saliency prediction with diffusion,” arXiv preprint arXiv:2408.12009 , 2024

    2024

  6. [14]

    An examination of cognitive processing of multimedia information based on viewers’ eye movements,

    H.-C. Liu and H.-H. Chuang, “An examination of cognitive processing of multimedia information based on viewers’ eye movements,” Interactive Learning Environments, vol. 19, no. 5, pp. 503–517, 2011

    2011

  7. [15]

    A cognitive approach to audio description: production and reception processes,

    J. Holsanova, “A cognitive approach to audio description: production and reception processes,” in The Routledge handbook of audio description . Routledge, 2022, pp. 57–77

    2022

  8. [16]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840– 6851, 2020

    2020

  9. [17]

    Distrifusion: Distributed parallel inference for high-resolution diffusion models,

    M. Li, T. Cai, J. Cao, Q. Zhang, H. Cai, J. Bai, Y . Jia, K. Li, and S. Han, “Distrifusion: Distributed parallel inference for high-resolution diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 7183–7193

    2024

  10. [18]

    Few-shot learner pa- rameterization by diffusion time-steps,

    Z. Yue, P. Zhou, R. Hong, H. Zhang, and Q. Sun, “Few-shot learner pa- rameterization by diffusion time-steps,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 23 263–23 272

    2024

  11. [19]

    Structure-guided ad- versarial training of diffusion models,

    L. Yang, H. Qian, Z. Zhang, J. Liu, and B. Cui, “Structure-guided ad- versarial training of diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 7256–7266

    2024

  12. [20]

    Fixation prediction through multimodal analysis,

    X. Min, G. Zhai, K. Gu, and X. Yang, “Fixation prediction through multimodal analysis,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) , vol. 13, no. 1, pp. 1–23, 2017

    2017

  13. [21]

    Soundnet: Learning sound representations from unlabeled video,

    Y . Aytar, C. V ondrick, and A. Torralba, “Soundnet: Learning sound representations from unlabeled video,” Advances in neural information processing systems, vol. 29, 2016

    2016

  14. [22]

    Casp-net: Rethinking video saliency prediction from an audio-visual consistency perceptual perspective,

    J. Xiong, G. Wang, P. Zhang, W. Huang, Y . Zha, and G. Zhai, “Casp-net: Rethinking video saliency prediction from an audio-visual consistency perceptual perspective,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6441–6450

    2023

  15. [23]

    Temporal-spatial feature pyramid for video saliency detection,

    Q. Chang and S. Zhu, “Temporal-spatial feature pyramid for video saliency detection,” arXiv preprint arXiv:2105.04213 , 2021

    2021 arXiv

  16. [24]

    From discrete representation to continuous modeling: A novel audio- visual saliency prediction model with implicit neural representations,

    D. Zhu, K. Zhang, K. Zhu, N. Zhang, W. Ding, G. Zhai, and X. Yang, “From discrete representation to continuous modeling: A novel audio- visual saliency prediction model with implicit neural representations,” IEEE Transactions on Emerging Topics in Computational Intelligence , 2024

    2024

  17. [25]

    Top-down visual saliency guided by captions,

    V . Ramanishka, A. Das, J. Zhang, and K. Saenko, “Top-down visual saliency guided by captions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 7206–7215

    2017

  18. [26]

    Visual saliency for image captioning in new multimedia services,

    M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, “Visual saliency for image captioning in new multimedia services,” in 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) . IEEE, 2017, pp. 309–314

    2017

  19. [27]

    Ecanet: Explicit cyclic attention-based network for video saliency prediction,

    H. Xue, M. Sun, and Y . Liang, “Ecanet: Explicit cyclic attention-based network for video saliency prediction,” Neurocomputing, vol. 468, pp. 233–244, 2022

    2022

  20. [28]

    Human attention in image captioning: Dataset and analysis,

    S. He, H. R. Tavakoli, A. Borji, and N. Pugeault, “Human attention in image captioning: Dataset and analysis,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 8529–8538

    2019

  21. [29]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  22. [30]

    Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,

    S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 318–335

    2018

  23. [31]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  24. [32]

    Revisiting video saliency: A large-scale benchmark and a new model,

    W. Wang, J. Shen, F. Guo, M.-M. Cheng, and A. Borji, “Revisiting video saliency: A large-scale benchmark and a new model,” in Proceedings of the IEEE Conference on computer vision and pattern recognition , 2018, pp. 4894–4903

    2018

  25. [33]

    Clustering of gaze during dynamic scene viewing is predicted by motion. cognitive computation, 3 (1), 5–24,

    P. Mital, T. Smith, R. Hill, and J. Henderson, “Clustering of gaze during dynamic scene viewing is predicted by motion. cognitive computation, 3 (1), 5–24,” 2011

    2011

  26. [34]

    How saliency, faces, and sound influence gaze in dynamic social scenes,

    A. Coutrot and N. Guyader, “How saliency, faces, and sound influence gaze in dynamic social scenes,” Journal of vision , vol. 14, no. 8, pp. 5–5, 2014

    2014

  27. [35]

    Multimodal saliency models for videos,

    ——, “Multimodal saliency models for videos,” From Human Attention to Computational Attention: A Multidisciplinary Approach , pp. 291–304, 2016

    2016

  28. [36]

    A perceptually based spatio-temporal com- putational framework for visual saliency estimation,

    P. Koutras and P. Maragos, “A perceptually based spatio-temporal com- putational framework for visual saliency estimation,” Signal Processing: Image Communication , vol. 38, pp. 15–31, 2015

    2015

  29. [37]

    Cre- ating summaries from user videos,

    M. Gygli, H. Grabner, H. Riemenschneider, and L. Van Gool, “Cre- ating summaries from user videos,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13 . Springer, 2014, pp. 505–520

    2014

  30. [38]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.