Pith. sign in

REVIEW 5 major objections 6 minor 80 references

This paper claims that question-aware multi-round frame selection with a compact summary memory makes video question answering more accurate and more efficient.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:27 UTC pith:HF3Y5QD6

load-bearing objection REVISE is a plausible extension in sparse video QA, but the paper's central efficiency claim is undermined by an internal contradiction about whether the summary or the full frame history is the actual context. the 5 major comments →

arxiv 2602.13602 v2 pith:HF3Y5QD6 submitted 2026-02-14 cs.CV cs.LG

Towards Sparse Video Understanding and Reasoning

Chenwei Xu , Zhen Ye , Shang Wu , Weijian Li , Zihan Wang , Zhuofan Xia , Lie Lu , Pranav Maneriker
show 3 more authors
Fan Du Manling Li Han Liu
This is my paper
classification cs.CV cs.LG
keywords video question answeringsparse frame selectionmulti-round agentsummary-as-statereinforcement fine-tuningearly stoppingvision-language modelGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

REVISE tries to establish that a VLM agent does not need to see many frames to answer video questions. Instead of uniform sampling, the agent requests a few frames most relevant to the question, maintains a compact structured summary of what it has seen and what it still doubts, and stops once confident. The paper argues this sparse, stateful loop works in two settings: as a plug-and-play wrapper around a frozen VLM, and as the basis for reinforcement fine-tuning of open models. If true, long-video question answering becomes cheaper to run and less vulnerable to the overload that dense, redundant frames cause.

Core claim

On the paper's own terms, the central discovery is that a persistent summary-as-state, written as a fixed summary with fields P (previously seen), O (observations), H (belief updates), U (uncertainties), and R (reasons for the next frame request), can serve as the only information carried between rounds, and this is enough to select informative frames and answer accurately. The paper reports that a proprietary VLM wrapped in this loop raises average accuracy on a 14-category fine-grained video reasoning benchmark from 26.4 to 48.9 while using 7.99 frames per video; with reinforcement fine-tuning, a small open VLM improves from 31.7 to 51.3 on a causal and temporal QA benchmark while cutting

What carries the argument

The central mechanism is the summary-as-state: a compact text state zt = (P, O, H, U, R) that is updated each round and is the only information passed between rounds. P/O/H/U/R stand for previously seen frames, current observations, belief updates, remaining uncertainties, and reasons for the next frame request. This state conditions every new decision and provides a natural early-stopping signal. Around it sit a multi-round controller that selects at most a few frames per round, and—for open models—the EAGER reward (Evidence-Adjusted Gain for Efficient Reasoning), which assigns credit for confidence gain from new evidence, summary sufficiency at answer time, and correct-and-early stopping.

Load-bearing premise

The whole approach stands or falls on whether a fixed-format text summary can preserve every piece of visual evidence a question might need, because that summary is the only state carried between rounds.

What would settle it

A concrete check: run the same multi-round agent but carry the full set of previously seen raw frames across rounds under an equal or modestly larger token budget; if that version answers a meaningful fraction more questions correctly, the compressed summary is losing task-critical visual detail. A single question whose answer depends on a visual attribute absent from the summary would make the point directly.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Video question answering can run in a single-digit frame regime—often 3 to 8 frames per video—and still match or beat dense uniform sampling.
  • A frozen, proprietary VLM can be wrapped into a multi-round sparse reasoner without retraining; on the paper's fine-grained benchmark this adds roughly 22 accuracy points.
  • Reinforcement fine-tuning with an annotation-free reward improves both accuracy and efficiency over plug-and-play and supervised format tuning.
  • The structured summary is load-bearing: ablating state carryover or the P/O/H/U/R fields costs more than 17 accuracy points in the paper's experiments.
  • Early stopping behaves reliably under a small turn budget, with average rounds staying near 1.3 to 2.8 even when four rounds are allowed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same select-then-summarize loop should transfer to other long multimodal inputs—audio, documents, or sensor logs—by replacing the frame sampler with chunking, since the summary-as-state is not tied to vision specifically.
  • My inference: because the paper's drift analysis attributes 15 to 20 percent of errors to summary-state drift, mostly on temporal questions, adding explicit timestamps or retaining a small set of raw keyframes alongside the summary could close part of that gap.
  • My inference: the EAGER reward uses only answer labels and model scores, so the recipe may apply to other long-context question-answering settings where only final correctness is known.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes REVISE, a multi-round video-QA agent that iteratively selects a small number of informative frames, maintains a P/O/H/U/R text summary as the claimed sole cross-round state, and stops early when confident. It reports plug-and-play accuracy gains on VideoEspresso, EgoSchema, NExT-QA, and additional benchmarks, and introduces EAGER, a GRPO-compatible reward for fine-tuning open-source VLMs. The central empirical claim is that question-aware sparse frame selection with a compact summary memory improves both accuracy and efficiency (fewer frames, rounds, and prompt tokens) over dense-frame and caption-heavy baselines.

Significance. If confirmed, the contribution is practically valuable: it suggests that a lightweight agent protocol around a frozen VLM can outperform dense-frame baselines, and that a verifier-style reward can fine-tune a 3B VLM to select more informative frames and stop earlier. The paper is systematic in its ablations (state carryover, structured fields, turn budget, reward weights) and contains an honest self-analysis of summary drift. However, the manuscript currently does not release code/data, does not report variance or significance, and has an unresolved contradiction about what context is actually fed to the VLM. As a result, the headline accuracy and token-efficiency gains cannot yet be attributed to the proposed summary-as-state mechanism.

major comments (5)
  1. [§3.1 vs. Figure 3 and Eq. (3.4)] §3.1 states that '<summary> is the only information persisted to the next round' and Algorithm 1 conditions each action on (p_t, z_{t-1}, F_t) only. Figure 3's caption, however, says the VLM receives 'the entire conversation history' each round, and Eq. (3.4) imposes the budget C(S_{τ−1}) + |p_τ| ≤ K on all previously admitted frames. These descriptions are incompatible. If raw frames and prior prompts remain in context, the summary is not the only state and the measured token savings cannot be attributed to summary compression. If the implementation follows Algorithm 1, the budget should be C(F_τ) + |p_τ| (or similar) and Figure 3 is wrong. Please specify the exact prompt/context construction and token accounting, and release prompt logs or exemplar transcripts.
  2. [§4.1, Table 11] Table 11 appears to tune the EAGER hyperparameters on the NExT-QA test set. The 'Base (no RL)' row reports 31.7 accuracy / 1.74 rounds, which exactly matches the NExT-QA plug-and-play result in Table 6, and the 'Paper Setting' row reports 51.3 / 1.32, which matches the NExT-QA RFT result. Yet §4 states that ablations use VideoEspresso with Qwen-2.5-VL-7B. Either Table 11 uses a different dataset/backbone than stated, or the reported NExT-QA numbers were used to select λ_i, β, and T_stop. Please state which split was used; if NExT-QA evaluation was used for hyperparameter selection, report validation-based tuning instead.
  3. [Abstract, §1, §5; Tables 1–6] The paper repeatedly claims that REVISE reduces 'prompt tokens', but no main experiment reports token counts. Tables 1–6 give frames, rounds, and time; Table 10 reports text tokens only for the caption-vs-frame variant. Without token measurements for the headline plug-and-play and RFT results, the token-efficiency claim is unsupported. Add per-example input token counts (or total API tokens) for all main comparisons, or revise the claim to 'fewer frames and rounds'.
  4. [§4, Tables 1–4] Many baselines are quoted from other papers (e.g., VideoEspresso results 'direct quoted from Han et al.' and EgoSchema/NExT-QA baselines quoted from Awasthi et al.) rather than rerun under the same harness, and no error bars or significance tests are reported. Since REVISE's gains are large and partly driven by prompt formatting, single-run point estimates cannot rule out evaluation-protocol artifacts. Please release code, prompts, sampled trajectories, and per-video results, and report scores over at least three seeds or a bootstrap confidence interval.
  5. [§10.3] The author-provided drift analysis attributes 70–82% of errors to missing key evidence and 15–20% to summary-state drift, mainly on temporal questions. This is honest, but combined with the Figure 3 contradiction it is impossible to tell whether the drift is caused by the compact P/O/H/U/R state or by whatever raw history was actually included. Please reconcile the implementation description and, if the summary is truly the only cross-round state, give a breakdown of drift errors by question type with examples; this is load-bearing for the 'summary-as-state' premise.
minor comments (6)
  1. [Eq. (3.11)] m_t is said to be computed 'before taking action a_t' but uses S_t, while the MDP state in Eq. (3.8) is (p_t, z_{t-1}, S_{t-1}). Clarify whether the log-odds margin is computed before or after showing F_t and align the indices.
  2. [§1, §3.3] EAGER is called 'annotation-free', but it uses the correct label y*; this is answer-label-dependent, not annotation-free. Suggest saying 'no frame-level annotations'.
  3. [Algorithm 1] The symbol r_t is used both for the raw model response and for the reward. Rename to avoid ambiguity, e.g., 'resp_t' for the response.
  4. [References, Tables 3–4] Reference [2] is cited as 'LVNet' for video QA, but the reference is an echocardiography segmentation paper. The video-QA LVNet appears to be [45]. Please audit all references.
  5. [Table 10] The column header 'GPT-5.1' appears to be a typo, likely 'GPT-4o' or 'GPT-5.1' needs explanation.
  6. [Table 5] The configuration names '01_061', '02_042', etc. are unexplained. Clarify how they map to max_rounds and max_frames_per_round.

Circularity Check

0 steps flagged

No circular reduction found: the paper's claims are benchmark measurements plus a stated reward design, not derivations that reproduce their inputs.

full rationale

REVISE does not present a formal derivation chain; its headline claims (accuracy, frames, rounds, tokens) are empirical benchmark measurements against external baselines. The summary state zt=(P,O,H,U,R) is defined (Eq. 3.1) rather than fitted to the reported outcomes, and the component ablation (Table 2) and drift analysis (§10.3, 15–20% summary-state drift) show the summary-sufficiency premise is testable and tested, not an assumed conclusion. EAGER rewards (Eqs. 3.12–3.14) directly encode confidence gain, summary-only answerability, and early correct stopping, so the RL improvements are partly optimized-for by design; but this is reward shaping, not a circular reduction, and the same reward could fail on other backbones. Self-citations such as RAGEN [60] motivate the MDP/GRPO setup, but the central summary-as-state and frame-selection contribution does not reduce to those citations, and no uniqueness theorem is imported. Two concerns are flagged but do not qualify as circularity under the given definitions: (1) Figure 3's caption says the VLM receives 'the entire conversation history' while Algorithm 1 conditions on (pt, zt−1, Ft) and Eq. 3.4 enforces C(Sτ−1)+|pτ|≤K; this stated-vs-implemented inconsistency is a reproducibility/token-attribution problem, not a step where an output equals an input by construction. (2) Table 11 reports a 'Paper Setting' selected from ablations on NExT-QA, which is a statistical overfitting risk, but selecting hyperparameters on a benchmark is not a circular derivation. Therefore the central claims stand as empirical findings, and no load-bearing step reduces to its own inputs.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The method introduces no new physical entities. It depends on hand-chosen budgets, reward weights, and several domain assumptions, the strongest of which is the summary-sufficiency premise. The reward hyperparameters are not parameter-free; they are tuned through ablations on the same benchmark used to report the headline results.

free parameters (6)
  • max_rounds T = 4
    Turn budget for all main experiments; chosen by hand, not fitted to data, but it directly affects the Pareto-optimal accuracy/frame trade-off reported in Table 5.
  • max_frames_per_round = 3
    Maximum number of frames shown per round; a design choice that limits evidence per turn and shapes early stopping behavior.
  • EAGER reward weights lambda_1, lambda_2, lambda_3 = 1.0, 1.0, 0.5
    Weights for confidence gain, summary sufficiency, and correct-and-early stop in Eq. 3.15; ablated on NExT-QA in Table 11, with the reported configuration selected from those ablations.
  • Early-stop parameters beta, T_stop = 1.0, 2
    Bonus scale and turn budget for the correct-and-early stop reward; chosen from the same NExT-QA ablation (Table 11) as the reward weights.
  • Sampling hyperparameters = temperature 0.2, top-p 0.9, max response length 256
    Used for all VLM queries; hand chosen and not varied.
  • RL training hyperparameters = lr 1e-6, KL 0.001, batch size 8, epochs 200, 8000 distilled conversations
    GRPO/fine-tuning configuration listed in Table 8; the table is labeled 'PPO-based' despite the text using GRPO, adding uncertainty about which values were actually used.
axioms (5)
  • domain assumption Semantic sparsity: only a small number of frames are relevant to a given video question, so question-aware frame selection can match or beat full-context processing.
    Stated as the motivation (L1/L2 in Section 1); the entire efficiency claim depends on this premise.
  • domain assumption Summary-as-state sufficiency: the P/O/H/U/R text summary preserves all task-critical evidence from previously seen frames, so discarding raw frames is safe.
    Central to the design (Eq. 3.1, Section 3); the authors' own drift analysis (Section 10.3) concedes 15-20% of errors come from summary drift.
  • domain assumption Log-odds margin m_t is a reliable, calibrated confidence signal for early stopping and reward computation.
    EAGER's confidence-gain term and the early-stop behavior both rely on this (Eq. 3.11-3.14); no calibration analysis is provided.
  • domain assumption Quoted baseline results from other papers are directly comparable to the numbers produced by this paper's pipeline.
    Tables 3 and 4 import VideoTree, VideoAgent, LLoVi, and other baseline scores from external papers rather than rerunning them under identical conditions, despite different frame budgets and captioning pipelines.
  • standard math Standard RL/GRPO update rules apply to the token-level policy and the trajectory-level advantage in Eq. 3.16-3.18.
    The paper relies on GRPO as an established algorithm; no new convergence or statistical guarantees are claimed.

pith-pipeline@v1.3.0-alltime-deepseek · 21295 in / 13945 out tokens · 118488 ms · 2026-08-02T23:27:56.709653+00:00 · methodology

0 comments
read the original abstract

We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA). Instead of uniformly sampling frames, \revise selects a small set of informative frames, maintains a summary-as-state across rounds, and stops early when confident. It supports proprietary vision-language models (VLMs) in a ``plug-and-play'' setting and enables reinforcement fine-tuning for open-source models. For fine-tuning, we introduce EAGER (Evidence-Adjusted Gain for Efficient Reasoning), an annotation-free reward with three terms: (1) Confidence gain: after new frames are added, we reward the increase in the log-odds gap between the correct option and the strongest alternative; (2) Summary sufficiency: at answer time we re-ask using only the last committed summary and reward success; (3) Correct-and-early stop: answering correctly within a small turn budget is rewarded. Across multiple VQA benchmarks, \revise improves accuracy while reducing frames, rounds, and prompt tokens, demonstrating practical sparse video reasoning.

Figures

Figures reproduced from arXiv: 2602.13602 by Chenwei Xu, Fan Du, Han Liu, Lie Lu, Manling Li, Pranav Maneriker, Shang Wu, Weijian Li, Zhen Ye, Zhuofan Xia, Zihan Wang.

Figure 1
Figure 1. Figure 1: Summary-as-State. REVISE operates analogously to a recurrent neural network: it maintains a state that propagates information from previous turns to the VLM. To further ensure reasoning consistency and efficient multi￾round interaction, REVISE uses the persistent “summary as state” carried across rounds. Instead of reprocessing long conversation histories or large sets of previously seen frames, the agent … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of REVISE: multi-round reasoning and adaptive frame selection. Given an initial set of frames and a question, the VLM agent infers the video context to update the summary and selects relevant frames based on its reasoning. In the next round, the agent reasons over the selected frames and the updated summary to generate the final answer. advances, reinforcement learning techniques have emerged as e… view at source ↗
Figure 3
Figure 3. Figure 3: REVISE. REVISE consists of three components: multi-round conversation, a structured output protocol, and a summary-as-state. Each round, the VLM agent receives (i) the entire conversation history, (ii) the current prompt, and (iii) the chosen video frames, annotated with their timestamps and the video’s total frame count. In the first round, a formatting guideline is also provided. The VLM outputs <summary… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy–frames Pareto frontier. Each point corresponds to a different frame budget N, and the frontier is monotone. Component Ablation [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 31 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Navchetan Awasthi, Lars Vermeer, Louis S Fixsen, Richard GP Lopata, and Josien PW Pluim. Lvnet: Lightweight model for left ventricle segmentation for short axis views in echocardiographic imaging.IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 69(6): 2115–2128, 2022. 7

    2022

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities.ArXiv preprint, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.ArXiv preprint, 2023. 6

    2023

  4. [4]

    Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond, 2023. 2

    2023

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 6, 8

    Pith/arXiv arXiv 2025

  6. [6]

    Mem- ory consolidation enables long-context video understanding

    Ivana Balaževi´c, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Mem- ory consolidation enables long-context video understanding. arXiv preprint arXiv:2402.05861, 2024. 7

    Pith/arXiv arXiv 2024

  7. [7]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024. 1

    Pith/arXiv arXiv 2024

  8. [8]

    Video ChatCaptioner: Towards en- riched spatiotemporal descriptions, 2023

    Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny. Video ChatCaptioner: Towards en- riched spatiotemporal descriptions, 2023. 2

    2023

  9. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 6

    2024

  10. [10]

    Zero-shot video question answering with procedural programs.arXiv preprint arXiv:2312.00937, 2023

    Rohan Choudhury, Koichiro Niinuma, Kris M Kitani, and László A Jeni. Zero-shot video question answering with procedural programs.arXiv preprint arXiv:2312.00937, 2023. 1, 2, 7

    Pith/arXiv arXiv 2023

  11. [11]

    Videoagent: A memory-augmented multimodal agent for video understanding.arXiv preprint arXiv:2403.11481, 2024

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding.arXiv preprint arXiv:2403.11481, 2024. 2

    Pith/arXiv arXiv 2024

  12. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 2, 3, 8

    Pith/arXiv arXiv 2025

  13. [13]

    Long context tuning for video generation.arXiv preprint arXiv:2503.10589,

    Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation.arXiv preprint arXiv:2503.10589,

    Pith/arXiv arXiv

  14. [14]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26181–26191, 2025. 6, 3

    2025

  15. [15]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding, 2024

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding, 2024. 2

    2024

  16. [16]

    Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997. 1

    1997

  17. [17]

    Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens

    Sunil Hwang, Jaehong Yoon, Youngwan Lee, and Sung Ju Hwang. Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens. InInternational Conference on Machine Learning, 2024. 1

    2024

  18. [18]

    Chat-UniVi: Unified visual representation empowers large language models with image and video un- derstanding, 2024

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-UniVi: Unified visual representation empowers large language models with image and video un- derstanding, 2024. 2

    2024

  19. [19]

    Language repository for long video understanding.arXiv preprint arXiv:2403.14622, 2024

    Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding.arXiv preprint arXiv:2403.14622, 2024. 1, 2

    Pith/arXiv arXiv 2024

  20. [20]

    Segment any- thing.ArXiv preprint, 2023

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing.ArXiv preprint, 2023. 2

    2023

  21. [21]

    Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J. Kim. Large language models are temporal and causal reasoners for video question answering, 2023. 2

    2023

  22. [22]

    Text-conditioned resampler for long form video understanding, 2024

    Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zis- serman, and Federico Tombari. Text-conditioned resampler for long form video understanding, 2024. 2

    2024

  23. [23]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024. 3

    Pith/arXiv arXiv 2024

  24. [24]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 2

    2024

  25. [25]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 1, 2

    2024

  26. [26]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 6

    Pith/arXiv arXiv 2024

  27. [27]

    VideoChat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding, 2024. 2 9

    2024

  28. [28]

    MVBench: A comprehensive multi-modal video understanding benchmark, 2024

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A comprehensive multi-modal video understanding benchmark, 2024

    2024

  29. [29]

    LLMs meet long video: Advancing long video comprehension with an interactive visual adapter in LLMs, 2024

    Yunxin Li, Xinyu Chen, Baotain Hu, and Min Zhang. LLMs meet long video: Advancing long video comprehension with an interactive visual adapter in LLMs, 2024. 2

    2024

  30. [30]

    Dancing with critiques: Enhancing llm reasoning with stepwise natural language self-critique.arXiv preprint arXiv:2503.17363, 2025

    Yansi Li, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Qiuzhi Liu, Rui Wang, Zhuosheng Zhang, Zhaopeng Tu, Haitao Mi, et al. Dancing with critiques: Enhancing llm reasoning with stepwise natural language self-critique.arXiv preprint arXiv:2503.17363, 2025. 3

    Pith/arXiv arXiv 2025

  31. [31]

    Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 2

    Pith/arXiv arXiv 2023

  32. [32]

    Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023. 6

    Pith/arXiv arXiv 2023

  33. [33]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 2

    2023

  34. [34]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 1, 2

    2024

  35. [35]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 2

    2024

  36. [36]

    Let’s try again: Elic- iting multi-turn reasoning in language models via simplistic feedback, 2025

    Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li. Let’s try again: Elic- iting multi-turn reasoning in language models via simplistic feedback, 2025. 3

    2025

  37. [37]

    Open-ended multi-modal relational reasoning for video question answering

    Haozheng Luo, Ruiyang Qin, Chenwei Xu, Guo Ye, and Zening Luo. Open-ended multi-modal relational reasoning for video question answering. In2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 363–369, 2023. 2

    2023

  38. [38]

    Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024. 3

    Pith/arXiv arXiv 2024

  39. [39]

    S 2rr: Teaching llms to self-verify and self-correct via reinforcement learning.arXiv preprint arXiv:2502.12853, 2025

    Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S 2rr: Teaching llms to self-verify and self-correct via reinforcement learning.arXiv preprint arXiv:2502.12853, 2025. 3

    Pith/arXiv arXiv 2025

  40. [40]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 2

    2024

  41. [41]

    Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 1

    2023

  42. [42]

    Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36, 2024

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36, 2024. 6, 7, 3

    2024

  43. [43]

    Pg-video-llava: Pixel grounding large video- language models.arXiv preprint arXiv:2311.13435, 2023

    Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-llava: Pixel grounding large video- language models.arXiv preprint arXiv:2311.13435, 2023. 2

    Pith/arXiv arXiv 2023

  44. [44]

    GPT-4o blog, 2024

    OpenAI. GPT-4o blog, 2024. 6, 7, 8

    2024

  45. [45]

    Too many frames, not all useful: Efficient strategies for long-form video qa.arXiv preprint arXiv:2406.09396, 2024

    Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa.arXiv preprint arXiv:2406.09396, 2024. 7

    arXiv 2024

  46. [46]

    Detgpt: Detect what you need via reasoning.arXiv preprint arXiv:2305.14167, 2023

    Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Ling- peng Kong Tong Zhang. Detgpt: Detect what you need via reasoning.arXiv preprint arXiv:2305.14167, 2023. 2

    Pith/arXiv arXiv 2023

  47. [47]

    High-dimensional continuous con- trol using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jor- dan, and Pieter Abbeel. High-dimensional continuous con- trol using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015. 2

    Pith/arXiv arXiv 2015

  48. [48]

    Proximal policy optimization algo- rithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms.arXiv preprint arXiv:1707.06347, 2017. 2

    Pith/arXiv arXiv 2017

  49. [49]

    Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 6, 8, 2

    Pith/arXiv arXiv 2024

  50. [50]

    Moviechat: From dense token to sparse memory for long video understanding, 2024

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding, 2024. 1

    2024

  51. [51]

    Vipergpt: Vi- sual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Vi- sual inference via python execution for reasoning. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023. 2

    2023

  52. [52]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2, 6

    Pith/arXiv arXiv 2024

  53. [53]

    Vamos: Versatile action models for video understanding.arXiv preprint arXiv:2311.13627, 2023

    Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwon- joon Lee, and Chen Sun. Vamos: Versatile action models for video understanding.arXiv preprint arXiv:2311.13627, 2023. 2

    Pith/arXiv arXiv 2023

  54. [54]

    Lvbench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024. 1

    Pith/arXiv arXiv 2024

  55. [55]

    Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Systems, 37:121475–121499, 2024. 2

    2024

  56. [56]

    Videoagent: Long-form video understanding with large language model as agent.arXiv preprint arXiv:2403.10517,

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent.arXiv preprint arXiv:2403.10517,

    Pith/arXiv arXiv

  57. [57]

    LSTP: Language-guided spatial-temporal prompt learning for long-form video-text understanding, 2024

    Yuxuan Wang, Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, and Zilong Zheng. LSTP: Language-guided spatial-temporal prompt learning for long-form video-text understanding, 2024. 2

    2024

  58. [58]

    LifelongMem- ory: Leveraging LLMs for answering queries in long-form egocentric videos, 2024

    Ying Wang, Yanlai Yang, and Mengye Ren. LifelongMem- ory: Leveraging LLMs for answering queries in long-form egocentric videos, 2024. 2

    2024

  59. [59]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209,

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209,

    Pith/arXiv arXiv

  60. [60]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025. 2, 3, 4, 8

    2025

  61. [61]

    Longvlm: Efficient long video understanding via large language models, 2024

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models, 2024. 2

    2024

  62. [62]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InCVPR, pages 9777–9786, 2021. 6, 7, 3

    2021

  63. [63]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023. 2

    Pith/arXiv arXiv 2023

  64. [64]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision).ArXiv preprint, 2023

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision).ArXiv preprint, 2023. 2

    2023

  65. [65]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840,

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840,

    Pith/arXiv arXiv

  66. [66]

    Re-thinking temporal search for long- form video understanding, 2025

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chan- drasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long- form video understanding, 2025. 1, 2

    2025

  67. [67]

    Self-chained image-language model for video localization and question answering.Advances in Neural Information Processing Systems, 36, 2024

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering.Advances in Neural Information Processing Systems, 36, 2024. 7

    2024

  68. [68]

    CREMA: Multimodal compositional video reasoning via efficient mod- ular adaptation and fusion.arXiv preprint arXiv:2402.05889,

    Shoubin Yu, Jaehong Yoon, and Mohit Bansal. CREMA: Multimodal compositional video reasoning via efficient mod- ular adaptation and fusion.arXiv preprint arXiv:2402.05889,

    Pith/arXiv arXiv

  69. [69]

    So- cratic models: Composing zero-shot multimodal reasoning with language.arXiv preprint arXiv:2204.00598, 2022

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choro- manski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. So- cratic models: Composing zero-shot multimodal reasoning with language.arXiv preprint arXiv:2204.00598, 2022. 2

    Pith/arXiv arXiv 2022

  70. [70]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 1

    2023

  71. [71]

    A simple llm framework for long-range video question-answering.arXiv preprint arXiv:2312.17235, 2023

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering.arXiv preprint arXiv:2312.17235, 2023. 2, 7

    Pith/arXiv arXiv 2023

  72. [72]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding, 2023. 2

    2023

  73. [73]

    Gpt4roi: Instruction tuning large language model on region-of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 2

    Pith/arXiv arXiv 2023

  74. [74]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 2 11 Towards Sparse Video Understanding and Reasoning Supplementary Material Table 7. Table of notations used in REVISE and its reinforcement fine-tuning. Symbol Description V={x i}L...

    2024

  75. [75]

    Table of Notation We list the table of notations of our paper in Table 7

  76. [76]

    Extended Background of Reinforcement Learning in LLMs Reinforcement Learning (RL) enables large language models (LLMs) to improve their behavior via interaction with an environment and scalar reward feedback. Formally, given a data distribution D over prompts x and a policy πθ(y|x) 1 parameterized byθ, the RL objective maximizes the expected reward J(θ) =...

  77. [77]

    Detailed Algorithm We show the REVISEmain algorithm in Algorithm 1

  78. [78]

    Plug-and-play

    Experimental Details Computational Hardware.All experiments are conducted on platforms equipped with Tesla A100 SXM GPUs (for reinforcement fine-tuning and large-scale evaluation) and Intel Xeon Silver 4214 CPUs @ 2.20GHz. 2 Algorithm 1REVISE: Multi-Round Sparse Video Reasoning Require: Video V={x i}L−1 i=0 ; prompt p; max rounds T ; max frames per round ...

  79. [79]

    +caption

    Additional Analyses 10.1. Turn-budget Allocation Table 9.Turn-budget allocation ablation.We compare different allocations of max_rounds and max_frames_per_round under a comparable total frame budget. Balanced allocations (e.g.,5×4and4×5) outperform skewed ones (e.g.,10×2and2×10). Max Rounds Max Frames / Round Accuracy (%) Avg Rounds Avg Frames Used 10 2 6...

  80. [80]

    Instead, REVISEfacilitates more efficient 4 Table 11.EAGER reward ablation.We vary reward weights and early-stop parameters (β, Tstop) to evaluate robustness

    Broader Impacts As a new methodology for improving video understanding, we do not identify any direct negative societal impacts such as disinformation, unfair decision-making, privacy violations, or security risks. Instead, REVISEfacilitates more efficient 4 Table 11.EAGER reward ablation.We vary reward weights and early-stop parameters (β, Tstop) to eval...