Pith. sign in

REVIEW 3 major objections 6 minor 66 references

Cross-video spot-the-difference is a scalable proxy that teaches video models precise local spatiotemporal evidence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 11:29 UTC pith:SCAP67M6

load-bearing objection Solid, usable proxy-training recipe for local video evidence; transfer is real but modest, and the synthetic-edit limit is the main soft spot, not a collapse of the claim. the 3 major comments →

arxiv 2607.02551 v1 pith:SCAP67M6 submitted 2026-06-26 cs.CV cs.AI

DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences

Yankai Yang , Yancheng Long , Bin Wen , Fan Yang , Tingting Gao , Han Li , Shuo Yang This is my paper
classification cs.CV cs.AI
keywords video multimodal large language modelsfine-grained spatiotemporal perceptioncross-video differencesproxy taskrule-based reinforcement learningtemporal localizationspatial groundingevidence reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video multimodal language models can describe global scene meaning, yet they often miss a brief change, a small region, or a sharp temporal boundary. This paper claims that comparing two nearly identical videos—one carrying a controlled local edit—and requiring the model to name the change, its time span, and its spatial region supplies the missing supervision. The authors turn real clips into labeled A/B pairs (DELTAVID-10K), train with rule rewards on grounding and multiple-choice tasks, and introduce a diagnostic bench for difference understanding. Gains appear both on that bench and on standard video understanding tests that never ask for A/B comparison. A sympathetic reader cares because the work treats “find the difference” not as a party game but as training fuel that moves models from coarse semantics toward evidence they can point to in time and space.

Core claim

Cross-video difference comparison is both a diagnosis of fine-grained perception failure and a scalable, verifiable proxy task: when a model must find local changes between similar videos and output change type, time span, and/or spatial region under rule-based rewards, it acquires local evidence skills that transfer beyond the edit setting to general video understanding.

What carries the argument

DELTAVID: controllable spatial, temporal, and spatiotemporal perturbations on real clips yield labeled A/B pairs; those labels become Grounding rewards (temporal IoU, spatial IoU, and their product for joint cases) and MCQ exact-match rewards, optimized with group-relative policy updates and anti-shortcut randomization.

Load-bearing premise

Skills learned from programmed local edits on real clips—shape inserts, blurs, speed changes, freezes—are the same skills needed for natural fine-grained failures in open video understanding.

What would settle it

After the same post-training, measure whether gains vanish on held-out natural (non-programmatic) video difference questions and on temporal-boundary items with no synthetic overlays; if synthetic scores stay high while natural and general transfer collapse, the proxy claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Same-scale models can surpass larger models on local difference localization after this training.
  • Temporal boundary judgment is the main bottleneck and the largest measured gain.
  • Grounding and MCQ supervision are complementary; using either alone lowers both in-domain and transfer scores.
  • Spatial, temporal, and spatiotemporal edit families supply complementary training signals.
  • Local evidence learned on A/B pairs transfers to single-video QA and temporal-order benchmarks without further task-specific data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If programmatic edits already transfer, richer natural semantic differences (object swaps, causal edits) may close more of the remaining gap to human local perception.
  • The same A/B difference signal could serve as a cheap online diagnostic for when a deployed video model is relying only on global cues.
  • Difficulty-aware packing of sparse multi-part localization rewards may help other multimodal RL settings that fail early when all evidence fields must be correct at once.
  • Pairing difference training with long-context video models may particularly improve multi-event boundary tracking in long videos.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. DELTAVID proposes a verifiable proxy-task framework that improves fine-grained spatiotemporal perception in Video MLLMs by training on cross-video spot-the-difference. From real clips, the authors synthesize A/B pairs with 13 controlled spatial, temporal, and spatiotemporal perturbations, convert labels into Grounding (temporal/spatial IoU, including product reward for joint evidence) and MCQ tasks, and post-train Qwen3-VL-8B with GRPO and anti-shortcut design, releasing DELTAVID-10K and DELTAVID-Bench. On DELTAVID-Bench the 8B model gains substantially over its base (especially T-IoU), surpasses Qwen3-VL-235B overall, and approaches GPT-5; modest positive transfer is reported on MMVU, MLVU, Video-MME, TempCompass, VideoMMMU, and related general video benchmarks, with ablations supporting complementary perturbation dimensions and Grounding+MCQ.

Significance. If the proxy truly induces general local-evidence skill rather than edit-signature fitting, the work offers a scalable, preference-model-free route to strengthen temporal boundaries and spatial evidence in Video MLLMs—an important gap relative to global semantic pretraining. Strengths include fully automatic evidence labels, rule rewards tied to parseable structure (Eqs. 1–3), difficulty-aware multi-difference matching, anti-shortcut randomization, dual-task design, attention visualization consistent with T-IoU gains, and multi-benchmark transfer without task-specific fine-tuning on those suites. The contribution is practically useful for diagnosis and post-training even if natural-change coverage remains incomplete.

major comments (3)
  1. [§3.2, Table 1; §4.2–4.4, Tables 2–3; §5] Central transfer claim (Abstract; §1; §5) rests on the assumption that the 13 programmatic edits in §3.2/Table 1 induce general fine-grained evidence skill. DELTAVID-Bench is built from the same controllable pipeline (§4.2, A.5), so Table 2 mainly re-tests the training distribution. Table 3 transfer is real but modest and uneven (e.g., TempCompass +7.4, VideoMMMU +7.6 vs Video-MME +0.5, VideoHolmes +0.2; Avg 60.56→63.06). Table 4 ablations stay inside the synthetic pool. Please add either (i) evaluation on natural/semantic differences (occlusion, lighting, object state, camera cuts) or a held-out edit family, or (ii) a clearly scoped claim that the proxy improves synthetic-local and some temporal QA skills, with analysis of which general tasks benefit and why.
  2. [§3.5, Eq. (1); Appendix A.2] The joint spatiotemporal reward r_st = IoU_t · IoU_s (Eq. 1) and greedy one-to-one matching for multi-difference samples are load-bearing design choices but only lightly justified. Product rewards can collapse when either factor is near zero early in training; matching details (greedy vs optimal bipartite, unmatched penalty) affect multi-difference credit. Please report sensitivity to product vs min/mean aggregation and to matching policy, or show that early sparse-reward issues are controlled by the difficulty-aware mix in §3.4 rather than by reward form alone.
  3. [§4.1, Tables 2–3] Baseline separation of “cross-video difference training” from “general video RL” is incomplete. Video-R1 is only partially reported on general suites (Table 3 “–” entries), and there is no control that applies GRPO with non-difference verifiable rewards (e.g., standard temporal QA or format-only) on the same base. Without that, part of the Table 3 lift could be generic RL/format effects rather than local-evidence supervision. A same-budget non-difference RL control, or fuller Video-R1 numbers under the unified harness, would secure the causal story.
minor comments (6)
  1. [Figure 3, §4.4] Figure 3 attention example is qualitative and single-instance; a small quantitative attention-overlap or localization-consistency metric would better support the T-IoU narrative.
  2. [§3.2; Appendix A.1] Table 6/7 vs abstract “10K”: training has 9,433 task samples from 8,200 pairs; clarify naming (DELTAVID-10K) vs exact counts early in §3.2/§4.1.
  3. [§3.5; Appendix A.4] Reward mix is stated as 0.9/0.1 in Eq. (3) and again as (1−α)R_task+αR_format with α=0.1 in A.4; keep a single notation and note whether format partial credit for missing <think> is used in main runs.
  4. [Abstract; §4.3; Table 3] MVBench appears in Table 3 and §4.3 but is omitted from the abstract’s transfer list; align lists for consistency.
  5. [Appendix A.5] Human verification κ=0.81 and 94.6% answer agreement (A.5) are useful; briefly state annotator count and whether disagreements were resolved by majority or expert adjudication.
  6. [§2; Figure 1] Typos/formatting: “LLaV A” spacing in related work; “Video-LLaV A”; ensure consistent DELTAVID vs DeltaVid in figure captions.

Circularity Check

1 steps flagged

No load-bearing circular derivation; only a minor train/test distribution overlap on the synthetic proxy, while the central transfer claim is checked on external benchmarks.

specific steps
  1. other [§3.2 / Table 1 (perturbation synthesis); §4.2 and Appendix A.5 (DELTAVID-Bench)]
    "To make this signal scalable to train and reliable to evaluate, we further introduce DELTAVID-10K and DELTAVID-Bench, which convert controllable local differences in real videos into evidence-labeled training and test samples. ... DELTAVID-Bench is constructed through candidate curation, automatic filtering, and human verification. We first curate about 4,000 candidate A/B samples from carefully selected high-quality video pairs..."

    Training data and the in-domain diagnostic bench both come from the same controllable local-difference synthesis family (spatial/temporal/spatiotemporal perturbations on real clips). In-domain MCQ/T-IoU/S-IoU gains therefore largely re-evaluate the proxy distribution rather than an independent natural difference distribution. This is only a mild, non-load-bearing circularity-adjacent design choice: the paper’s transfer claim is separately tested on external general video benchmarks that do not use A/B synthetic edits, and rewards remain rule-based rather than fitted to those external metrics.

full rationale

DELTAVID is an empirical proxy-task / RL post-training paper, not a first-principles derivation. The method constructs controllable A/B edits, trains with rule IoU and exact-match rewards under GRPO, and reports gains on DELTAVID-Bench plus transfer to MMVU, TempCompass, Video-MME, and related suites. Rewards are not a learned scorer fitted to the target metrics; GRPO is taken from external DeepSeekMath; there is no uniqueness theorem, self-citation chain, or ansatz smuggled in as external fact. The only mild circularity-adjacent point is that DELTAVID-Bench is built from the same controllable-perturbation family as DELTAVID-10K, so in-domain T-IoU/MCQ lifts largely re-test the training distribution by design of the proxy. That does not force the transfer results, which are measured on independent single-video benchmarks that do not require A/B comparison. The paper itself flags that differences remain programmatic. Score 1 reflects that minor train/bench family overlap only; the central proxy-supervision claim is not reduced to its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

This is an empirical ML methods paper. Load-bearing content is the synthetic-edit proxy, rule rewards, and GRPO setup rather than a mathematical derivation. Free parameters are training and synthesis knobs; axioms are standard RL/ML domain assumptions plus the paper-specific claim that programmatic local edits are a valid perception proxy; invented entities are the framework, dataset, and bench constructed for the study.

free parameters (4)
  • task/format reward mix (0.9 / 0.1)
    Hand-chosen weight in Eq. (3); format term is small by design and affects parseability vs task signal.
  • GRPO group size G=8, lr=1e-6, KL β=0.02, LoRA rank 64 / α=128
    Training hyperparameters in Appendix A.6 that govern optimization stability and capacity of the post-trained policy.
  • perturbation sampling ranges (time window 15–70%, shape size 6–18%, hue set, speed factors, etc.)
    Preset ranges in A.2 define the difficulty and visual statistics of the proxy; they are design choices, not derived.
  • difficulty-aware data composition (single/dual/mixed/large-edit subset counts)
    Table 7 subset sizes organize sparse-reward training; composition is chosen rather than learned.
axioms (4)
  • domain assumption Rule-based GRPO with IoU and exact-match rewards can improve fine-grained video perception without a learned reward model.
    Assumed throughout §3.5 and justified by citation to DeepSeekMath/GRPO and related multimodal RL work.
  • ad hoc to paper Product of temporal IoU and spatial IoU is an appropriate joint reward for spatiotemporal differences.
    Eq. (1) in §3.5; enforces joint correctness by construction of the reward, not derived from external theory.
  • domain assumption Controlled local edits on real video backgrounds preserve enough natural statistics that learned skills transfer to open video understanding.
    Core premise of §3.2 and the transfer experiments; paper notes residual gap to natural/semantic changes in §5.
  • ad hoc to paper Bipartite/greedy one-to-one matching of predicted vs ground-truth differences fairly scores multi-difference samples without order constraints.
    §3.5 multi-difference reward design; unmatched differences get zero reward by design.
invented entities (3)
  • DELTAVID proxy-task framework no independent evidence
    purpose: Convert cross-video comparison into verifiable Grounding and MCQ supervision for Video MLLMs.
    Named method combining synthesis, dual tasks, anti-shortcut design, and GRPO; utility is argued via experiments, not independent physical evidence.
  • DELTAVID-10K no independent evidence
    purpose: Scalable training set of evidence-labeled A/B video (and image) difference samples from real clips.
    Constructed dataset (~9.4k task samples from 8.2k pairs); exists only as this paper's artifact unless released.
  • DELTAVID-Bench no independent evidence
    purpose: 500-question diagnostic for cross-video difference counting, type, time, and region skills.
    In-house diagnostic with human verification (κ=0.81); measures the proxy rather than an external community standard.

pith-pipeline@v1.1.0-grok45 · 25033 in / 3370 out tokens · 45088 ms · 2026-07-12T11:29:55.512457+00:00 · methodology

0 comments
read the original abstract

Video multimodal large language models have made strong progress on open-ended video understanding, but they still lack precise local spatiotemporal perception. When two videos share almost the same global semantics and differ only in a short time span or a small region, current models often fail to find the change and provide reliable evidence. We propose DELTAVID, a verifiable proxy-task framework that enhances fine-grained spatiotemporal perception with cross-video differences. The key idea is to turn cross-video spot-the-difference into a trainable perception signal, where a model identifies local changes, judges temporal boundaries, and organizes spatial evidence by comparing similar videos. To make this signal scalable to train and reliable to evaluate, we further introduce DELTAVID-10K and DELTAVID-Bench, which convert controllable local differences in real videos into evidence-labeled training and test samples. Experiments show that DELTAVID substantially improves performance on cross-video difference understanding and transfers the learned local evidence ability to general video understanding benchmarks, including MMVU, MLVU, Video-MME, VideoHolmes, VideoMMMU, LVBench, TempCompass, and LongVideoBench. These results show that cross-video differences are not only an effective way to diagnose fine-grained perception failures, but also a scalable proxy supervision that moves Video MLLMs from coarse semantic understanding toward fine-grained spatiotemporal evidence reasoning.

Figures

Figures reproduced from arXiv: 2607.02551 by Bin Wen, Fan Yang, Han Li, Shuo Yang, Tingting Gao, Yancheng Long, Yankai Yang.

Figure 1
Figure 1. Figure 1: DELTAVID data construction, rule-reward training, and anti-shortcut design. reasoning format, answer quality, and some perception skills without a human preference model. Video methods have also started to explore temporal reasoning and video decision making [10, 49, 9]. Still, existing methods often trade task specificity for generalization, or depend on human-labeled data that limits scale and diversity.… view at source ↗
Figure 2
Figure 2. Figure 2: Representative perturbation examples in DELTAVID-10K. Each panel shows an A/B video pair or key-frame sequence, covering spatial, spatiotemporal, and temporal perturbation types used to construct training data. Grounding. Grounding converts each difference into verifiable structured evidence. Spatial differ￾ences require the model to localize the changed region, temporal differences require the model to pr… view at source ↗
Figure 3
Figure 3. Figure 3: Attention visualization example. The first two rows show key frames from the reference Video A and perturbed Video B; the middle two rows show spatial attention from the base model under cross-video input; the bottom two rows show attention after DELTAVID training. Compared with the base model, DELTAVID more consistently attends to the local changed region in the perturbed video [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics and checkpoint performance. Left: MCQ reward, Grounding reward, and total reward during GRPO post-training. Right: accuracy of different checkpoints on VideoMMMU, TempCompass, and MMVU; dashed lines indicate base-model performance. start and end time in seconds. bbox is the normalized bounding box [x1, y1, x2, y2], with coordinates from 0 to 1000. The reward module only depends on the tim… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 1 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bi´...

    2022

  2. [2]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    Pith/arXiv arXiv 2025

  3. [3]

    Qwen2.5-vl technical report,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report,

  4. [4]

    URLhttps://arxiv.org/abs/2502.13923

    Pith/arXiv arXiv

  5. [5]

    Video action differencing, 2025

    James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, and Serena Yeung-Levy. Video action differencing, 2025. URL https: //arxiv.org/abs/2503.07860

    Pith/arXiv arXiv 2025

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions, 2023

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023. URL https://arxiv.org/abs/2311.12793

    Pith/arXiv arXiv 2023

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Ji...

    Pith/arXiv arXiv 2025

  8. [8]

    Video-holmes: Can mllm think like holmes for complex video reasoning?, 2025

    Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning?, 2025. URL https://arxiv.org/ abs/2505.21374

    Pith/arXiv arXiv 2025

  9. [9]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Informa- tion Processing Systems, volume 36, page...

  10. [10]

    URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 9a6a435e75419a836fe47ab6793623e6-Paper-Conference.pdf

    2023

  11. [11]

    Videozoomer: Reinforcement-learned temporal focusing for long video reasoning, 2025

    Yang Ding, Yizhen Zhang, Xin Lai, Ruihang Chu, and Yujiu Yang. Videozoomer: Reinforcement-learned temporal focusing for long video reasoning, 2025. URL https: //arxiv.org/abs/2512.22315. 10

    arXiv 2025

  12. [12]

    Video-r1: Reinforcing video reasoning in mllms, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms, 2025. URLhttps://arxiv.org/abs/2503.21776

    Pith/arXiv arXiv 2025

  13. [13]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2025

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2025. URL https://arxiv.org/abs/2306.13394

    Pith/arXiv arXiv 2025

  14. [14]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2025

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 202...

    Pith/arXiv arXiv 2025

  15. [15]

    Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, 2024

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, 2024. URLhttps://arxiv.org/abs/2310.14566

    Pith/arXiv arXiv 2024

  16. [16]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    doi:10.1038/s41586-025-09422-z 2025

  17. [17]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos,

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos,

  18. [18]

    URLhttps://arxiv.org/abs/2501.13826

    Pith/arXiv arXiv

  19. [19]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2026

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Xu Tang, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2026. URLhttps://arxiv.org/abs/2503.06749

    Pith/arXiv arXiv 2026

  20. [20]

    Learning to describe differences between pairs of similar images

    Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, 11 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4024–4034, Brussels, Belgium, October-November 2018. Association for C...

    work page doi:10.18653/v1/d18-1436 2018

  21. [21]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. URLhttps://arxiv.org/abs/1612.06890

    Pith/arXiv arXiv 2016

  22. [22]

    Llava-onevision: Easy visual task transfer,

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer,

  23. [23]

    URLhttps://arxiv.org/abs/2408.03326

    Pith/arXiv arXiv

  24. [24]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference on Machine Learning, volume 202 o...

    2023

  25. [25]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. URL https:// arxiv.org/abs/2305.06355

    Pith/arXiv arXiv 2024

  26. [26]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors,Computer Vision – ECCV 2024, pages 323–340, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72952-2

    2024

  27. [27]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore, December 2023. Association for Computational Lin- gui...

    doi:10.18653/v1/2023.emnlp-main.20 2023

  28. [28]

    Video-LLaV A: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984, Miami, Florida, USA, November

    2024

  29. [29]

    doi: 10.18653/v1/2024.emnlp-main.342

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.342. URLhttps://aclanthology.org/2024.emnlp-main.342/

    doi:10.18653/v1/2024.emnlp-main.342 2024

  30. [30]

    Visual spatial reasoning, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023. URL https: //arxiv.org/abs/2205.00363

    Pith/arXiv arXiv 2023

  31. [31]

    World model on million-length video and language with blockwise ringattention, 2025

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention, 2025. URL https://arxiv.org/abs/2402. 08268

    2025

  32. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 6dcf277ea32ce3288...

    2023

  33. [33]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors,Computer Vision – ECCV 2024, pages 216–233, Cham,

    2024

  34. [34]

    ISBN 978-3-031-72658-3

    Springer Nature Switzerland. ISBN 978-3-031-72658-3. 12

  35. [35]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772, Bangkok, Thailand, August 2024. Association for Computatio...

    doi:10.18653/v1/2024.findings-acl.517 2024

  36. [36]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 12585–12602, Bangkok, Thaila...

    2024

  37. [37]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning, 2025. URL https://arxiv.org/abs/ 2503.07365

    Pith/arXiv arXiv 2025

  38. [38]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, A...

    Pith/arXiv arXiv 2024

  39. [39]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedbac...

    2022

  40. [40]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associ...

    2023

  41. [41]

    Timechat: A time-sensitive multimodal large language model for long video understanding, 2024

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding, 2024. URL https://arxiv. org/abs/2312.02051

    Pith/arXiv arXiv 2024

  42. [42]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URLhttps://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  43. [43]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    Pith/arXiv arXiv 2024

  44. [44]

    Moviechat: From dense token to sparse memory for long video understanding, 2024

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding, 2024. URL https://arxiv.org/abs/2307.16449. 14

    Pith/arXiv arXiv 2024

  45. [45]

    Reason-rft: Reinforcement fine-tuning for visual reasoning of vision language models, 2025

    Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Xiansheng Chen, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning of vision language models, 2025. URLhttps://arxiv.org/abs/2503.20752

    arXiv 2025

  46. [46]

    Yolo Y . Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali V osoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu. Video understanding with large language models: A survey, 2025. URLhttps://arxiv.org/abs/2312.17432

    arXiv 2025

  47. [47]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024. URLhttps://arxiv.org/abs/2406.16860

    Pith/arXiv arXiv 2024

  48. [48]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. URL https://arxiv. org/abs/2401.06209

    Pith/arXiv arXiv 2024

  49. [49]

    Lvbench: An extreme long video understanding benchmark, 2025

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2025. URLhttps://arxiv.org/abs/2406.08035

    Pith/arXiv arXiv 2025

  50. [50]

    Vicrit: A verifiable reinforcement learning proxy task for visual perception in vlms, 2025

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, Ziyi Zang, Ming Li, Chung-Ching Lin, Kevin Lin, Linjie Li, Furong Huang, and Lijuan Wang. Vicrit: A verifiable reinforcement learning proxy task for visual perception in vlms, 2025. URL https://arxiv.org/abs/2506.10128

    Pith/arXiv arXiv 2025

  51. [51]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2: Scaling foundation models for multimodal video understanding. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, To...

    2024

  52. [52]

    Blaschko

    Zifu Wang, Junyi Zhu, Bo Tang, Zhiyu Li, Feiyu Xiong, Jiaqian Yu, and Matthew B. Blaschko. Jigsaw-r1: A study of rule-based visual reinforcement learning with jigsaw puzzles, 2025. URL https://arxiv.org/abs/2505.23590

    arXiv 2025

  53. [53]

    Longvideobench: A benchmark for long- context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/ 2407.15754

    Pith/arXiv arXiv 2024

  54. [54]

    Vidic: Video difference captioning, 2026

    Jiangtao Wu, Shihao Li, Zhaozhou Bian, Jialu Chen, Runzhe Wen, An Ping, Yiwen He, Jiakai Wang, Yuanxing Zhang, and Jiaheng Liu. Vidic: Video difference captioning, 2026. URL https://arxiv.org/abs/2512.03405

    arXiv 2026

  55. [55]

    Tempr1: Improving temporal understanding of mllms via temporal-aware multi-task reinforcement learning, 2026

    Tao Wu, Li Yang, Gen Zhan, Yabin Zhang, Yiting Liao, Junlin Li, Deliang Fu, Li Zhang, and Limin Wang. Tempr1: Improving temporal understanding of mllms via temporal-aware multi-task reinforcement learning, 2026. URLhttps://arxiv.org/abs/2512.03963

    Pith/arXiv arXiv 2026

  56. [56]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization, 2025

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization, 2025. URL https: //arxiv.org/abs/2503.10615

    Pith/arXiv arXiv 2025

  57. [57]

    Perception-r1: Pioneering perception policy with reinforcement learning, 2025

    En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, and Wenbing Tao. Perception-r1: Pioneering perception policy with reinforcement learning, 2025. URL https://arxiv.org/abs/2504.07954

    Pith/arXiv arXiv 2025

  58. [58]

    Coca: Contrastive captioners are image-text foundation models, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL https: //arxiv.org/abs/2205.01917. 15

    Pith/arXiv arXiv 2022

  59. [59]

    Mm-vet: Evaluating large multimodal models for integrated capabilities,

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities,

  60. [60]

    URLhttps://arxiv.org/abs/2308.02490

    Pith/arXiv arXiv

  61. [61]

    Video-LLaMA: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Yansong Feng and Els Lefever, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543–553, Singapore, December 2023. Association for Computational Linguistic...

    doi:10.18653/v1/2023.emnlp-demo.49 2023

  62. [62]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025. URLhttps://arxiv.org/abs/2503.12937

    Pith/arXiv arXiv 2025

  63. [63]

    Llava- video: Video instruction tuning with synthetic data, 2025

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava- video: Video instruction tuning with synthetic data, 2025. URL https://arxiv.org/abs/ 2410.02713

    Pith/arXiv arXiv 2025

  64. [64]

    Mmvu: Measuring expert-level multi-discipline video understanding, 2025

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. Mmvu: Measuring expert-level multi-discipline video understanding, 2025. URL https://arxiv. org/abs/2501.12380

    Pith/arXiv arXiv 2025

  65. [65]

    Mlvu: Benchmarking multi-task long video understanding, 2025

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi-task long video understanding, 2025. URLhttps://arxiv.org/abs/2406.04264

    Pith/arXiv arXiv 2025

  66. [66]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models, 2023. URL https://arxiv.org/abs/2304.10592. 16 A Appendix A.1 Source video curation and pre-processing DELTAVID-10K uses high-quality web videos that we manually search and curate for cross-video sp...

    Pith/arXiv arXiv 2023