{"id":"f6a8115b-891f-475e-bc51-9fc0186b685a","arxiv_id":"2504.14267","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TAVDiff fuses text, audio, and visual features in a diffusion model to generate video saliency maps, reporting gains on six datasets over prior audio-visual and text-visual methods.","lead":"This paper presents TAVDiff, a diffusion model that predicts where people look in videos using video frames, audio, and text descriptions automatically generated from the frames. It reports improved saliency scores on six datasets, though the text condition is not independent of the visual input and its training uses the ground-truth saliency map to choose text features.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SITR trains with ground-truth saliency to select the text condition but switches to a max-sum rule at inference; if the two rules disagree, the reported text-modality gains may be an artifact of target leakage.","rationale":"Both the reader and I identify the SITR selection-rule mismatch as the load-bearing weakness. The manuscript's own Sec. III-C explicitly states that GT saliency is used to select the image-text response during training and that the max-sum rule is used at inference, without any evidence that the two are equivalent. Because this selection directly determines the conditional input to the denoiser, a mismatch can invalidate the text-modality comparison in Tables I, II, and V. The claim is testable by retraining without oracle selection. I do not see a reason to move beyond conditional acceptance on this basis alone: the concern is serious but not yet demonstrated to overturn the results. I also note a supporting inconsistency in the ablation table in Sec. IV-E: on ETMD, the Saliency-DiT row shows SIM 0.489 and CC 0.636, which are worse than U-Net (0.554/0.669) and DiT (0.560/0.672), while the text claims best performance on all metrics; this reinforces the need for a corrected and reproducible version before full acceptance.","tokens_in":15850,"tokens_out":7756,"duration_ms":72200,"concrete_test":"Retrain TAVDiff on AVAD and ETMD with the SITR selection rule changed to unweighted max-sum during training (no ground-truth multiplication), keeping architecture, optimizer, and all other settings identical. Then compare the text-modality gain (Ours(TAV) minus Ours(AV)) in Table I with the gain from the original oracle-trained model. If the gain largely disappears or reverses, the SITR oracle selection is responsible for the reported text improvement. A cheap preliminary check is to compute, on a held-out split, the agreement rate between the token chosen by the GT-weighted rule and the max-sum rule; low agreement would already indicate that the transfer assumption is unsafe.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C (Eq. 1) defines SITR: during training, the text-token attention map whose element-wise product with the ground-truth saliency map has the largest sum is selected as ftv, while at inference the map with the largest unweighted sum is chosen. The denoiser (Sec. III-D) is therefore trained with a text condition that is an oracle function of the target S0, but at deployment it receives a condition chosen by a different criterion. If the two selection rules pick different text tokens or attention maps, train and test condition distributions differ, and the text advantage in Tables I, II, and V may reflect training-time oracle selection rather than a genuine text-to-saliency mapping. The paper provides no evidence that max-sum selection approximates GT-weighted selection; it simply asserts that it represents the strongest semantic correlation. This is load-bearing because the central tri-modal contribution rests on SITR delivering useful text guidance, and a train/test mismatch in the conditioning signal can turn a capacity or overfitting effect into an apparent modality gain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TAVDiff, a diffusion-based video saliency prediction model conditioned on textual, audio, and visual inputs. Visual features are extracted with S3D, audio with SoundNet, and text with a CLIP text encoder; captions are generated automatically by BLIP-2 from the input frames. A Saliency-oriented Image-Text Response (SITR) mechanism uses cross-attention to produce text-conditioned visual response maps, and a custom Saliency-DiT denoising network decouples timestep embedding from conditional information injection. The model is evaluated on six video saliency datasets (DIEM, ETMD, AVAD, Coutrot1, Coutrot2, SumMe) against audio-visual and text-visual baselines, and ablations are reported on AVAD and ETMD. The authors claim state-of-the-art results and an average improvement of 1.03%, 2.35%, 2.71%, and 0.33% on SIM, CC, NSS, and AUC-J respectively.","tokens_in":16139,"tokens_out":6242,"duration_ms":57175,"significance":"If the claims hold, the paper would be among the first to combine text with audio and visual modalities in a diffusion-based video saliency model, and the Saliency-DiT idea of separating timestep and condition injection is a reasonable design hypothesis. The paper provides a broad set of comparisons across four metrics and six datasets, and the ablation structure is clear. However, the central novelty depends on SITR, and the current evidence for text-conditioning benefits is weakened by a train/inference selection mismatch that leaks the target into the training-time condition, by the non-independence of the automatically generated captions, and by an internal contradiction in the ablation table that is supposed to validate Saliency-DiT. These issues are load-bearing for the main claims, so the paper needs substantial revision rather than minor editing.","major_comments":[{"comment":"During training, SITR selects the text-token attention map by maximizing the element-wise product of the attention map with the ground-truth saliency map, while at inference it selects the map with the largest unweighted pixel sum. The training-time condition is therefore an oracle function of the target S0, so the text-conditioning gains reported in Tables I, II, and V may reflect leakage of the target into the condition rather than a genuine text-to-saliency mapping. The paper provides no evidence that the max-sum selection rule used at inference agrees with the GT-weighted rule used in training; please add experiments with max-sum selection also used during training, with random token selection, and with a direct comparison of the two selection rules on the validation sets.","section":"Section III-C, Eq. (1), Algorithm 1"},{"comment":"The text states that Saliency-DiT achieves the best performance across all evaluation metrics on both AVAD and ETMD, but Table III shows that on ETMD Saliency-DiT attains SIM 0.489 and CC 0.636, which are below U-Net (0.554/0.669) and DiT (0.560/0.672). Since Table III is the primary evidence for the proposed denoising architecture, the ETMD row directly contradicts the claim and must be corrected or the architecture must be re-evaluated. This also raises questions about the visual comparisons in Table I, where ETMD results for the full model are reported with the same SIM and CC values as Saliency-DiT in this ablation.","section":"Table III, Section IV-E"},{"comment":"The text modality is not an independent information source: BLIP-2 generates captions from the same video frames that are used for prediction. Consequently, the reported gains of Ours(TAV) over Ours(AV) could be due to the additional visual-semantic processing provided by BLIP-2 and the captioning prior rather than to text as an independent modality. The central 'tri-modal' claim would be much stronger if experiments used human-provided captions or held-out text annotations, or if the authors demonstrated that corrupting or shuffling the generated captions degrades performance. Without such evidence, the text-conditioning contribution is not convincingly separated from extra model capacity.","section":"Section III-A, Tables I and II"},{"comment":"The diffusion training and inference procedures are internally inconsistent as written. Section III-D first says the network produces the fully denoised map N0 = g_psi(Nt,Ct,t) and later says the final output N_t^5 is the feature map N_{t-1} after one round of denoising. Algorithm 1 trains with the objective ||g_psi(St,fa,ftv) - S0||^2, while Algorithm 2 feeds the network output Spred into a DDIM update. If the network predicts S0, the exact DDIM reparameterization must be specified; if it predicts N_{t-1}, the training objective is mis-stated. Please disambiguate the prediction target and provide the precise update rule used at inference.","section":"Section III-D and Algorithms 1-2"}],"minor_comments":[{"comment":"The phrase 'pre-trained visual-verbal grand model' should read 'large model' or 'foundation model'; the typo 'grand model' appears to be a typographical error.","section":"Section III-A"},{"comment":"The related-work section refers to 'CaDiff' while the reference list entry and Table I use 'CaDiff' inconsistently with the title 'Cardiff'; please unify the name.","section":"Section II-B and reference [13]"},{"comment":"The sentence reporting an 'average performance improvement of 2.08% compared to CASP(AV) and TSFP(AV)' is vague because no aggregation across metrics or datasets is defined; please specify the averaging procedure.","section":"Section IV-D"},{"comment":"The paper does not state which datasets are used for the DHF1K visual pre-training beyond the brief mention in Section IV-A, nor does it specify the fine-tuning protocol, splits, and number of runs; these details are necessary for reproducibility.","section":"Section IV-B"},{"comment":"The axes and metric shown in Figure 5 are not clearly identified in the caption or in the text; please label the figure completely.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The SITR train/inference mismatch is the most serious issue because it sits at the center of the claimed tri-modal contribution: the training-time condition is chosen using ground-truth saliency, so the text gains in the ablation tables could be an artifact of target leakage. The ETMD row in Table III is a clear internal contradiction that should have been caught before submission. The paper is otherwise readable and the experimental scope is appropriate, but these issues require substantive re-analysis and additional experiments before the central claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the tri-modal combination is new for video saliency prediction, and the main results mostly support the headline. But there are two soft spots worth your attention before you take the gains at face value.\n\nWhat's actually new: using a diffusion transformer as the saliency generator with decoupled timestep/conditioning, and using BLIP-2 captions plus a cross-attention selection (SITR) as text guidance alongside audio. The experiments span six audiovisual datasets and show consistent gains over several recognized baselines. That is a reasonable, incremental contribution to an applied subfield.\n\nThe problems. First, Table III undercuts the paper's own architecture claim. On ETMD, Saliency-DiT gets SIM 0.489 and CC 0.636, while the U-Net variant gets 0.554 and 0.669, and the DiT variant gets 0.560 and 0.672. The text says Saliency-DiT 'achieves the best performance across all evaluation metrics,' which is simply false on ETMD. The architecture helps on AVAD, but not on ETMD, and the paper does not explain that. Second, SITR is trained with an oracle selection: during training the attention map with the highest ground-truth-weighted sum is chosen, and at inference the map with the highest unweighted sum is chosen. The paper offers no evidence that these two selections agree, so the text condition at train time is a function of the target, and at inference it is not. That makes the reported text-modality gains hard to attribute to genuine text-to-saliency mapping. At minimum the authors should train a variant with the inference rule to see if the gain holds, or at least analyze the selection agreement.\n\nMinor issues: the text captions are generated from the same video frames, so the text modality is not independent; it is a derived feature of the visual input. That weakens the 'tri-modal' framing, though it does not invalidate the pipeline. Also no error bars, no code, and no data release are standard points a reviewer would ask for.\n\nWho it's for: anyone working on video saliency, multimodal attention, or diffusion-based dense prediction. It's not a landmark, but it is a legitimate applied paper with a stated new combination and mostly solid experiments. A serious referee should engage with it; the internal inconsistency in Table III needs to be addressed, and the SITR oracle issue deserves analysis or a changed training procedure. I would not desk-reject it.\n\nRecommendation: send to peer review, but expect revision.","headline":"A new tri-modal diffusion model for video saliency with mostly solid results, but the ETMD ablation contradicts the paper's own claim and the SITR text-condition selection uses target leakage during training.","tokens_in":16637,"tokens_out":4602,"would_cite":false,"duration_ms":39796,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TAVDiff treats video saliency prediction as a diffusion-based image generation task conditioned on text, audio, and visual inputs, and reports state-of-the-art results on six audiovisual datasets.","keywords":["video saliency prediction","diffusion model","multimodal learning","text-audio-visual","saliency-oriented image-text response","Saliency-DiT","cross-attention","generative saliency"],"falsifier":"Compare TAVDiff's text-conditioned variant against the same model fed with scrambled or random captions on videos where the caption names an object nobody fixates; if the random-caption model matches the real-caption model, the reported text gains are not coming from semantic gaze guidance.","tokens_in":15674,"feed_emoji":"🎯","tokens_out":5246,"duration_ms":40655,"temperature":0.7,"pith_summary":"This paper tries to establish that video saliency prediction is better modeled as a conditional image-generation problem than as a direct regression, and that adding text as a third modality alongside audio and vision helps. It proposes TAVDiff, a diffusion model that denoises a saliency map from Gaussian noise under visual, audio, and text conditions. The text condition is produced by an automatic captioner (BLIP-2) and aligned to image regions by a Saliency-oriented Image-Text Response (SITR) module. A new denoising transformer, Saliency-DiT, injects the conditions through cross-attention rather than mixing them with the timestep embedding. On six audiovisual saliency datasets the method outperforms existing audio-visual and text-visual baselines, with average gains of 1.03% SIM, 2.35% CC, 2.71% NSS, and 0.33% AUC-J.","feed_headline":"Tri-modal diffusion model tops video saliency on six datasets","feed_subtitle":"Adding auto-generated captions and sound to a diffusion generator lifts all four saliency metrics","key_machinery":"The load-bearing machinery is a conditional diffusion formulation with two bespoke components. Saliency-oriented Image-Text Response (SITR) is a text-to-visual grounding step: it computes multi-head cross-attention score maps between CLIP text-token features and S3D visual features, weighs them by ground-truth saliency during training, and picks the map with maximal total response as the fused condition. Saliency-DiT is the denoising transformer: it modulates self-attention and MLP with timestep-derived scale/shift/gate parameters, and inserts a multi-head cross-attention layer between them to inject the fused image-text response and audio features, decoupling condition information from the noise-level estimate. Together they convert video, audio, and text into a conditioning vector that guides stepwise denoising (with DDIM sampling at inference) of a saliency map.","core_discovery":"The paper's central claim is that human gaze in video can be predicted by a generative diffusion model that reconstructs the saliency map from noise under tri-modal conditioning. The visual branch supplies spatio-temporal features; the audio branch supplies sound-source cues; the text branch supplies high-level semantics extracted from automatically generated captions. The novel text pathway, SITR, uses multi-head cross-attention between CLIP text features and S3D visual features, multiplies each attention map by the ground-truth saliency map during training, and selects the map with the largest pixel sum as the image-text response. The denoising network Saliency-DiT keeps timestep modulation separate from condition injection by placing a cross-attention layer between self-attention and MLP, which the ablations show improves over both U-Net and vanilla DiT backbones. The authors claim this is the first tri-modal text-audio-visual video saliency framework, and their experiments report consistent improvements across all six tested datasets.","pith_inferences":["The paper leaves implicit that, because the captions come from the same frames used for prediction, the text channel might partly duplicate visual information; a fair test of semantic guidance would use captions that name objects absent from the frame or available only from context.","The SITR selection rule is trained with ground-truth saliency but applied at inference as a max-response rule; if the two diverge, a fine-tuning or distillation step that removes the ground-truth dependence could make the text pathway more robust.","The same condition-decoupling design could transfer to other conditional generation tasks, such as text-to-image or audio-driven editing, where timestep-condition mixing is also a concern.","Combining TAVDiff with human-written or subtitle-derived text, rather than BLIP-2 captions, on datasets that have such text would reveal whether the reported text gains generalize beyond generated captions."],"forward_implications":["Tri-modal conditioning is useful: adding text to audio-visual inputs improves all four metrics on all six datasets over the audio-visual-only variant.","The text pathway can be fed entirely by automatic captioning, so existing audio-visual datasets can be upgraded to tri-modal without new human annotations.","Decoupling timestep and condition pathways in a diffusion transformer is better than concatenating them, at least for dense prediction tasks like saliency.","Saliency prediction can be cast as a diffusion generation task and trained with a simple MSE denoising objective, avoiding bespoke discriminative architectures.","With only four DDIM steps at inference, the generative approach is practical enough for downstream use."],"supporting_citations":[{"why":"BLIP-2 generates the textual descriptions from video frames that create the text modality.","marker":"[29]"},{"why":"The CLIP text encoder supplies text features aligned to visual features for the SITR cross-attention.","marker":"[31]"},{"why":"S3D is the visual backbone providing spatio-temporal features for video.","marker":"[30]"},{"why":"SoundNet is the audio backbone providing sound features as a condition.","marker":"[21]"},{"why":"STA ViS supplies the audio preprocessing approach and serves as an audio-visual baseline.","marker":"[9]"},{"why":"CaDiff is the text-visual diffusion baseline that TAVDiff compares against.","marker":"[13]"},{"why":"CASP is a state-of-the-art audio-visual baseline used in the performance comparison.","marker":"[22]"},{"why":"TSFP is a state-of-the-art audio-visual baseline used in the performance comparison.","marker":"[23]"},{"why":"DHF1K is the dataset used to pretrain the visual branch before audiovisual evaluation.","marker":"[32]"},{"why":"DDPM provides the diffusion formulation that the conditional denoising framework builds on.","marker":"[16]"}],"fun_headline_variants":["Tri-modal diffusion model improves video saliency prediction","TAVDiff: text, audio, and video guide saliency prediction","Audio, text, and visuals boost diffusion-based saliency","Diffusion model fuses tri-modal cues for video saliency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that captions automatically generated from the video frames capture the semantic content that actually guides human gaze, and that the rule for picking which text-to-image attention map to use, learned with ground-truth saliency during training, still picks correctly when ground truth is unavailable at test time.","fun_headline_variants_meta":{"raw":{"variants":["Tri-modal diffusion model improves video saliency prediction","TAVDiff: text, audio, and video guide saliency prediction","Audio, text, and visuals boost diffusion-based saliency","Diffusion model fuses tri-modal cues for video saliency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1811,"prompt_tokens":1074,"completion_tokens":737,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":668}},"tokens_in":690,"tokens_out":737,"duration_ms":6143,"temperature":1.0,"reasoning_tokens":668,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:52:34.608805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare TAVDiff's text-conditioned variant against the same model fed with scrambled or random captions on videos where the caption names an object nobody fixates; if the random-caption model matches the real-caption model, the reported text gains are not coming from semantic gaze guidance.","supporting_citations":[],"review_version":1}