{"id":"2b0e37f6-3ecb-44c6-863e-76e404afb7af","arxiv_id":"2508.02516","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Large multimodal models, especially one that also processes audio, predict short-video engagement better than the prior feature-based baseline on the SnapUGC test set.","lead":"This paper applies two existing large multimodal models, VideoLLaMA2 and Qwen2.5-VL, to predict whether viewers watch short videos beyond five seconds, using keyframes, titles, and for VideoLLaMA2, background audio. The audio-aware model scored highest, and an ensemble of both won first place in the ICCV 2025 EVQA-SnapUGC challenge.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The audio-attribution claim in Section 4.5 is confounded: no ablation removes audio from VideoLLaMA2, so the gap over Qwen2.5-VL may stem from architecture, regression head, or training protocol rather than audio.","rationale":"The reader and I converge on the same load-bearing concern: the paper's headline conclusion attributes VideoLLaMA2's superiority to audio, but the comparison is confounded. My review of the full text found that the only regression-strategy ablation (Table 4) indeed shows a substantial effect, shrinking the headline gap when the feature-based head is used for Qwen2.5-VL; this strengthens the concern that the Table 1 comparison is not evidence for audio efficacy. The most direct way to settle the question is to ablate audio from VideoLLaMA2, keeping all other choices identical. Such an experiment is computationally feasible (one additional training run) and would either validate or refute the causal attribution. I also noted the minor reporting inconsistency between Section 4.7 (final score 0.714) and Table 5 (0.710), which is not central but adds to the need for careful revision. Since the existing evidence supports a weaker claim—LMMs perform well on this task—the CONDITIONAL verdict remains appropriate: the paper should be accepted only if the authors add the audio ablation and report variability. An UNCHANGED verdict means I do not see a reason to move away from the reader's recommendation.","tokens_in":14457,"tokens_out":4463,"duration_ms":45816,"concrete_test":"Run a controlled audio-ablation for VideoLLaMA2: train VideoLLaMA2 on SnapUGC using exactly the same configuration as in Table 1 (8 frames, same text metadata, same prompt template, same feature-based MLP regression head, same frozen vision encoder and fine-tuning protocol, same hyperparameters) but omit the audio tokens from the input sequence. Compare SROCC/PLCC against (a) full VideoLLaMA2 (0.691/0.701), (b) Qwen2.5-VL feature-based (0.674/0.679) and token-based (0.665/0.662). If the no-audio VideoLLaMA2 retains ~0.69 SROCC, the audio attribution fails and the gap is due to model architecture/training; if it drops to ~0.67 SROCC, audio explains most of the advantage. Repeat with 3 random seeds to also assess whether the original gap is within run-to-run noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion—that audio features are 'critical' for engagement prediction (Section 4.5 and Conclusion)—rests entirely on comparing VideoLLaMA2 (audio+visual+text) with Qwen2.5-VL (visual+text). These two models differ in multiple dimensions simultaneously: base LLM (VideoLLaMA2 vs Qwen2.5-VL), regression head (feature-based MLP in VideoLLaMA2 vs token generation in Qwen2.5-VL), training protocol (frozen vision encoder vs full fine-tuning), and input resolution. The paper's own ablation in Table 4 shows that regression strategy matters: Qwen2.5-VL feature-based reaches SROCC 0.674/PLCC 0.679, versus 0.665/0.662 for token-based. Comparing VideoLLaMA2 (0.691/0.701) to the feature-based Qwen2.5-VL reduces the gap from 0.026 to 0.017 SROCC, suggesting part of the advantage is due to the regression head. Yet no experiment removes audio from VideoLLaMA2 while holding everything else fixed. Without such an audio ablation, the statement 'We attribute this improvement to the inclusion of background audio' (Section 4.5) is an over-interpretation. A second, secondary issue is the lack of error bars or multiple seeds; the reported single-run differences (e.g., 0.691 vs 0.674 SROCC) are not shown to be statistically reliable. The ensemble result in Table 5 also contains a reporting slip: Section 4.7 says 'final score of 0.714' but the table lists 0.710, which weakens confidence in the manuscript's precision. The core empirical finding that LMMs are competitive for engagement prediction is credible, but the causal claim about audio is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the use of large multimodal models (LMMs) for cold-start short-video engagement prediction, specifically predicting Engagement Continuation Rate (ECR) on the SnapUGC dataset. Two LMMs are compared: VideoLLaMA2 (audio-visual-language) with a feature-based MLP regression head, and Qwen2.5-VL (visual-language) with token-based regression. The authors report that VideoLLaMA2 achieves SROCC 0.691 / PLCC 0.701, outperforming Qwen2.5-VL (0.665/0.662) and the prior two-stage baseline Li24 (0.657/0.665). Ablations on number of input frames, training data proportion, and regression strategy are presented, and an ensemble of four models is reported to achieve first place in the ICCV VQualA 2025 EVQA-SnapUGC Challenge. The paper concludes that audio features play a critical role in engagement prediction.","tokens_in":14913,"tokens_out":2477,"duration_ms":27938,"significance":"If the empirical findings are reliable, the paper provides a useful demonstration that end-to-end LMMs can be competitive with or superior to two-stage feature-based pipelines for short-video engagement prediction, and the ensembling result is an externally validated engineering achievement with code released. The ablations on frame count, training data, and regression head are valuable for practitioners. However, the central interpretive claim that audio is the cause of VideoLLaMA2's advantage is not supported by the experiments as designed, because the two compared LMMs differ in several other dimensions simultaneously. The paper's main positive contribution is the empirical comparison and the challenge result; the audio-attribution conclusion needs either supporting evidence or careful reframing.","major_comments":[{"comment":"The claim that VideoLLaMA2's superiority over Qwen2.5-VL 'highlights the critical role of audio features' is not supported by the presented experiments. The two models differ not only in audio input but also in base LLM architecture, regression strategy (MLP head vs. token generation), training protocol (frozen vision encoder with 5e-5 LR vs. full fine-tuning with 2e-6/1e-5 LR), and input resolution. Table 4 shows that switching Qwen2.5-VL from token-based to feature-based regression changes its final score from 0.664 to 0.676, reducing the gap to VideoLLaMA2 (0.695) from 0.031 to 0.019. There is no ablation that removes audio from VideoLLaMA2 while holding everything else fixed, nor one that adds audio to Qwen2.5-VL. Please add such an ablation (e.g., VideoLLaMA2 without the audio track) or, failing that, weaken the conclusion to state that the audio-visual-language model outperformed the visual-language model without attributing the difference specifically to audio.","section":"Section 4.5 and Conclusion"},{"comment":"All results are reported as single-run point estimates with no error bars, multiple seeds, or significance tests. Differences such as SROCC 0.691 vs. 0.674 (VideoLLaMA2 vs. feature-based Qwen2.5-VL) or 0.691 vs. 0.686 (Table 2) may be within run-to-run variation, especially given the different training budgets and hyperparameters. Please report the mean and standard deviation over at least three training runs, or apply a significance test (e.g., bootstrap on the test set or paired test across seeds), so readers can judge whether the reported ordering is statistically reliable.","section":"Tables 1-4"},{"comment":"There is an inconsistency between the text and the table: Section 4.7 states 'our method ... achieves the highest final score of 0.714', but Table 5 lists the final score as 0.710, with 0.714 being the PLCC value. Please correct this. In addition, the description 'ensembles four models, including three VideoLLaMA2 variants (from Table 2 and Table 3) and one Qwen2.5-VL model' is ambiguous, because the variants in Tables 2 and 3 differ in frame count and training proportion rather than being independently trained models; clarify whether the ensemble members are distinct trained checkpoints and how their predictions were combined.","section":"Section 4.7 and Table 5"}],"minor_comments":[{"comment":"The phrase 'the i-the keyframe' in the definition of x_i appears to be a typo; it should read 'the i-th keyframe'.","section":"Section 3.1"},{"comment":"The statement that VideoLLaMA2 'significantly outperforms' Qwen2.5-VL uses 'significantly' without statistical evidence; consider using 'consistently outperforms' or providing significance tests.","section":"Section 4.5"},{"comment":"The related work section cites Li et al. [21] for the SnapUGC dataset, but the description of the cross-modal attention architecture is brief; adding one or two sentences on how the proposed LMM-based approach differs from or improves upon this two-stage method would help position the contribution.","section":"Section 2.2"},{"comment":"The baseline 'MinimalisticVQA' is described as being retrained on SnapUGC, but the table caption does not state whether the LMM results are also from models trained on SnapUGC; specify the training condition for each row for clarity.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a challenge-participation report with a solid empirical core: the competitive results on SnapUGC and the first-place ensemble are credible engineering contributions. The main issue is that the headline scientific claim about audio is overreach given the confounded comparison, and the reporting weaknesses (single runs, no error bars, the 0.714/0.710 discrepancy) reduce confidence. The requested revisions are feasible within the scope of the manuscript: add an audio ablation or soften the claim, add uncertainty estimates, and fix the Table 5 inconsistency. I would not reject the paper on these grounds, but I would not accept it in its current form either."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:2508.02516. The paper does what it says: applies VideoLLaMA2 (audio+visual+text) and Qwen2.5-VL (visual+text) to cold-start ECR prediction on SnapUGC, and reports that the audio-visual model wins (0.695 vs 0.664 final score) and beats Li24's two-stage baseline. That's a real, useful empirical result for the engagement prediction crowd, and they released code.\n\nWhat's genuinely new: the direct end-to-end comparison of two modern LMMs on this task, plus the feature-vs-token regression ablation in Table 4. The ablations on frame count and training data are routine but clean. The ensemble first place is a credible outcome, not a surprise given the single-model numbers.\n\nThe soft spot is the audio attribution. The paper's headline claim—that audio is 'critical'—rests entirely on comparing VideoLLaMA2 to Qwen2.5-VL. Those models differ in base LLM, regression head (MLP vs token generation), training recipe (frozen vision encoder vs full fine-tuning), and input resolution. Table 4 shows that switching Qwen2.5-VL to feature-based regression improves it from 0.664 to 0.676, which already closes a third of the gap (0.031 to 0.019). No experiment removes audio from VideoLLaMA2 while holding everything else fixed. So the 'role of audio' conclusion is an over-interpretation of a single model comparison. The paper even says 'We attribute this improvement to the inclusion of background audio'—an attribution, not a demonstration.\n\nSecond issue: no error bars or multiple seeds. Some of these gaps could be noise. Also a minor reporting slip: Section 4.7 says the ensemble achieves a final score of 0.714, but Table 5 lists the final score as 0.710 (0.714 is the PLCC). Not a big deal, but it lowers confidence in the manuscript's precision.\n\nWho gets value: anyone working on video engagement prediction, LMM-based scoring, or recommendation cold-start. They'll get a solid baseline and a cautionary example about attributing performance to a single modality. The paper deserves a serious peer review. I'd send it, but the referee should demand either an audio ablation (e.g., VideoLLaMA2 with audio masked) or a revised conclusion that doesn't claim audio is the cause. As is, the empirical finding is believable; the causal claim is not.","headline":"Solid challenge report with a believable headline result, but the audio-attribution claim is not backed by an ablation.","tokens_in":15424,"tokens_out":2534,"would_cite":true,"duration_ms":27099,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large multimodal models with audio input predict short-video engagement better than visual-only models and prior feature-based pipelines.","keywords":["video engagement prediction","large multimodal models","audio-visual understanding","engagement continuation rate","SnapUGC","cold-start prediction","multimodal regression","short-form video"],"falsifier":"Run VideoLLaMA2 on SnapUGC with the audio tokens masked or removed, and run Qwen2.5-VL with an added audio branch; if their scores stay unequal or reverse, audio is not the unique cause of the gap.","tokens_in":14280,"feed_emoji":"🎧","tokens_out":7128,"duration_ms":76076,"temperature":0.7,"pith_summary":"This paper sets out to show that general-purpose large multimodal models can predict the engagement continuation rate of short videos before any user-interaction data exists. On the SnapUGC benchmark, an audio-visual-language model (VideoLLaMA2) and a vision-language model (Qwen2.5-VL), both fed keyframes plus title and description, outperform a purpose-built two-stage feature pipeline, with VideoLLaMA2 additionally taking background audio. The authors attribute that lead to background audio, with VideoLLaMA2 reaching SROCC 0.691 versus Qwen2.5-VL's 0.665. If the result holds, cold-start engagement prediction can move away from handcrafted multimodal features and toward end-to-end multimodal models.","feed_headline":"Sound-aware AI wins short-video engagement prediction","feed_subtitle":"On SnapUGC, an audio-visual LMM beats vision-only models and a feature-based baseline at predicting watch time.","key_machinery":"The object that carries the argument is the multimodal prompt: keyframes are encoded into visual tokens, the audio track into spectrogram tokens, and title and description into text, all placed inside a natural-language instruction that asks for the engagement continuation rate. VideoLLaMA2 averages the final-layer hidden states of its decoder and maps that average to a scalar through a two-layer MLP with an MSE loss, while Qwen2.5-VL learns to generate the numeric score as tokens with a cross-entropy loss. The intended comparison between the audio-inclusive and audio-exclusive branches is what supports the paper's conclusion about sound.","core_discovery":"On the SnapUGC test set, VideoLLaMA2, prompted with eight keyframes, title and description, and a spectrogram of the audio track, predicts ECR with SROCC 0.691 and PLCC 0.701, beating Qwen2.5-VL (0.665 and 0.662) and the prior two-stage baseline (0.657 and 0.665). The paper also reports that regressing the averaged hidden state through a two-layer MLP is better than generating the score as output tokens (Qwen2.5-VL feature-based 0.674/0.679 versus token-based 0.665/0.662), that eight frames outperform five, and that full training data outperform 60 percent. Ensembling four model variants produces a final score of 0.710 and first place in the EVQA-SnapUGC challenge.","pith_inferences":["The paper's core comparison does not isolate audio: the two models differ in architecture, training procedure, and regression head, so the audio attribution is a hypothesis rather than a demonstrated mechanism.","If audio is the real driver, engagement prediction for music-heavy platforms should include a dedicated audio branch, while platforms with mostly speech-over-visuals may get less benefit.","The feature-based-versus-token regression result suggests a transferable recipe: keep the LMM's decoder lightly trained and fit a small MLP head, which is cheaper than full token-level fine-tuning.","A direct audio-masking ablation on the same backbone is the natural next experiment, and its outcome would decide whether the gap is modality or model."],"forward_implications":["End-to-end large multimodal models can replace the two-stage extract-features-then-regress recipe for cold-start engagement prediction.","Including background audio is worth a measurable gain: the audio-visual branch leads the vision-language branch by roughly 0.03 in SROCC and 0.04 in PLCC.","For fine-grained score prediction, a lightweight regression head on hidden states beats having the model generate the number as text.","Using more keyframes and more training data both push engagement prediction upward.","Ensembling several LMM variants gives a further boost, lifting the final score from 0.695 to 0.710."],"supporting_citations":[{"why":"Supplies the SnapUGC dataset with ECR labels and the two-stage feature-based engagement baseline the LMMs must beat.","marker":"[21]"},{"why":"Supplies the VideoLLaMA2 audio-visual-language backbone whose hidden states are averaged and regressed to predict ECR.","marker":"[8]"},{"why":"Supplies the Qwen2.5-VL vision-language backbone used as the audio-free comparison model.","marker":"[1]"},{"why":"Supplies the MinimalisticVQA visual-quality baseline that shows pure quality-assessment models underperform on engagement.","marker":"[42]"},{"why":"Defines the challenge whose leaderboard and evaluation protocol the ensemble result is reported against.","marker":"[22]"}],"fun_headline_variants":["Audio boosts multimodal video engagement prediction","Sound-aware LMM tops short-video engagement forecast","First place: LMM ensemble predicts video engagement","Audio-visual LMM outperforms vision-only on SnapUGC","Key: audio features for engagement prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats VideoLLaMA2 and Qwen2.5-VL as identical except for audio, so the measured gap is credited to sound; no test in the paper removes audio from one model or adds it to the other.","fun_headline_variants_meta":{"raw":{"variants":["Audio boosts multimodal video engagement prediction","Sound-aware LMM tops short-video engagement forecast","First place: LMM ensemble predicts video engagement","Audio-visual LMM outperforms vision-only on SnapUGC","Key: audio features for engagement prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1309,"prompt_tokens":1019,"completion_tokens":290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":219}},"tokens_in":635,"tokens_out":290,"duration_ms":3941,"temperature":1.0,"reasoning_tokens":219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:56:51.552182+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VideoLLaMA2 on SnapUGC with the audio tokens masked or removed, and run Qwen2.5-VL with an added audio branch; if their scores stay unequal or reverse, audio is not the unique cause of the gap.","supporting_citations":[{"cited_title":"Delving deep into engagement prediction of short videos","cited_arxiv_id":null,"evidence_quote":"Supplies the SnapUGC dataset with ECR labels and the two-stage feature-based engagement baseline the LMMs must beat."},{"cited_title":"Analysis of video quality datasets via design of minimalistic video quality models","cited_arxiv_id":null,"evidence_quote":"Supplies the MinimalisticVQA visual-quality baseline that shows pure quality-assessment models underperform on engagement."}],"review_version":1}