Pith. sign in

REVIEW 2 major objections 5 minor 89 references

QSVideo finds a few query-relevant, diverse frames so compact vision-language models understand long and streaming videos much better under tight frame budgets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

QSVideo reformulates questions into structured queries, ranks frames by object-action-location relevance plus diversity, and applies temporal strategies to boost VLM accuracy under tight frame budgets.

T0 review reviewed 2026-07-11 challenge →

load-bearing objection Solid systems paper: query rewrite + three-axis scoring + dual temporal policies lifts 8B VLMs by ~7 pp on LVBench and hits SoTA on StreamingBench under tight frame budgets; auto-label noise is real but not load-bearing. the 2 major comments →

arxiv 2607.04559 v1 pith:6EXQOVKH submitted 2026-07-06 cs.CV

QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

Wei Ao , Lan Wang , Vishnu Naresh Boddeti This is my paper
classification cs.CV
keywords video understandingmultimodal retrievalvision-language modelsframe selectionlong videostreaming videoquery-conditioned ranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models lose accuracy as videos get longer because irrelevant moments flood their language components. The paper argues that the fix is not more frames or bigger models, but better retrieval of the right visual evidence. QSVideo first rewrites any question into a clean retrieval query and scores every frame for relevance along object, action, and location. It then selects a small, diverse set of frames while deliberately covering different time windows—jumping to key moments in long videos and favoring recent evidence in streams. Under strict limits of 8, 16 or 32 frames the method lifts an 8B model to state-of-the-art on streaming benchmarks and adds roughly seven accuracy points on hour-scale videos, remaining plug-and-play across several backbones.

Core claim

Existing multimodal retrieval for video QA is limited by biased relevance scores, near-duplicate frames, and collapse onto narrow time intervals. QSVideo shows that a query-conditioned semantic ranker (QSRanker) that reformulates questions and scores object/action/location relevance, combined with a retrieval procedure (QSRetrieval) that jointly maximises relevance, visual diversity and temporal coverage, supplies far more informative frames. The resulting selections let modest 8B video VLMs match or beat much larger systems under the same tight frame budgets on both long-video and streaming benchmarks.

What carries the argument

QSRanker plus QSRetrieval: an LLM first rewrites the raw question into a retrieval-friendly query and assigns importance weights to object, action and location; a fine-tuned 4B VLM then scores each frame on those three axes; a greedy selector next chooses frames that trade off high weighted relevance against visual L2 distance while enforcing temporal-window coverage (global-anchor-then-local for long videos, recency-first for streams).

Load-bearing premise

The automatic object/action/location scores produced by a large off-the-shelf VLM on only eight uniformly sampled frames per video are assumed to be reliable enough to supervise the smaller semantic ranker.

What would settle it

Retrain the semantic ranker on the same videos but with human-annotated object/action/location scores (or with labels from a different large VLM) and re-evaluate end-to-end accuracy on LVBench and StreamingBench; a large drop would falsify the claim that the current auto-labels are sufficient.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Under a fixed frame budget, simply replacing uniform sampling with QSVideo raises accuracy of existing 7–8B video VLMs by several points without any change to the VLM weights.
  • The same retrieval module can be dropped in front of multiple different video VLMs and still yields consistent gains, so model scaling is not required for the improvement.
  • Temporal strategies specialised for long versus streaming video measurably improve evidence recall, showing that coverage policy matters as much as relevance scoring.
  • Because the ranker is lightweight and batchable, the extra compute stays modest relative to the accuracy gain, making the pipeline practical for real-time or memory-constrained settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The structured three-axis scoring may transfer to other multimodal tasks (image-document retrieval, long-context audio) where relevance is multi-faceted rather than scalar.
  • If the auto-annotation step can be replaced by cheaper self-consistency or contrastive signals, the whole pipeline could become fully unsupervised and easier to scale to web video.
  • The recency-first rule for streams suggests a natural interface for always-on AR glasses or robotics, where the system continually re-ranks only the most recent window.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces QSVideo, a modular retrieval framework that improves video VLM performance under strict frame budgets by addressing biased relevance, limited diversity, and temporal collapse. QSRanker uses an LLM question analyzer to rewrite arbitrary questions into retrieval-friendly queries and to produce Object/Action/Location importance weights, then a fine-tuned 4B VLM semantic ranker scores frames along the same three dimensions (Eqs. 2–4). QSRetrieval greedily balances relevance and visual diversity (L2 distance on mean-pooled embeddings) inside temporal windows, with distinct traversal policies for long videos (global-anchor then local expansion) and streaming videos (recency-first). The semantic ranker is SFT-ed on the authors’ Video-Ranker-35K set whose labels are produced by Qwen-VL-30B. Under 8/16/32-frame budgets the pipeline lifts MiniCPM-o 2.6 by 6.9–7.1 pp on LVBench, reaches SoTA on StreamingBench, and yields consistent gains when plugged into three other 7–8B VLMs; latency and λ ablations are also reported.

Significance. If the reported gains hold, the work supplies a practical, model-agnostic way to raise long- and streaming-video accuracy without enlarging the VLM or the frame budget—an immediately useful engineering contribution for memory-constrained deployment. Strengths include released code, explicit plug-and-play results across backbones (Table 3), a clear relevance–diversity–coverage formulation (Eq. 1), and tailored temporal policies that match human search heuristics. The empirical claims rest on standard external benchmarks independent of the training set, so the performance numbers are falsifiable and reproducible.

major comments (2)
  1. §3.2 and the Recall@K paragraph in §5.2: the only supervision for the semantic ranker consists of discrete O/A/L scores auto-generated by Qwen-VL-30B on eight uniformly sampled frames. No human agreement study, inter-annotator reliability, or even a small held-out human-labeled subset is provided. While the end-to-end gains remain valid even under noisy labels, the claim that the ranker “explicitly enforces the correct ranking” rests on unvalidated automatic labels; a modest human validation or comparison against alternative labelers would strengthen the modular contribution of QSRanker itself.
  2. §3.3 and Tables 1–2: the temporal strategies (anchor-and-expand radius r for long video; recency-first for streaming) are presented as essential, yet no ablation isolates them from plain windowed relevance–diversity selection (Eq. 8). Without that control it is impossible to quantify how much of the 6.9–7.1 pp LVBench lift or the StreamingBench SoTA is attributable to the temporal policies versus the ranker scores alone.
minor comments (5)
  1. Table 6 (λ ablation) and §5.1: only five discrete λ values are shown; a continuous sweep or variance across random seeds would better support the claim that performance is “relatively insensitive.”
  2. §4.3 latency tables: candidate-frame count is fixed at 2048; the scaling of QSRanker cost with video length or sampling density is not characterized.
  3. Fig. 2 and the prompt appendix: the qualitative examples are helpful, but the exact prompt templates for the question analyzer and semantic ranker appear only in the appendix; a short reference in the main text would aid reproducibility.
  4. Notation: the temporal coverage term Γ in Eq. 1 is never given a concrete value or measurement procedure in the experiments.
  5. Related-work §2: a few concurrent frame-selection methods (e.g., BOLT, FlexSelect) are cited but not compared under identical frame budgets; a short discussion of why they are omitted would be useful.

Circularity Check

0 steps flagged

No significant circularity: empirical retrieval pipeline evaluated on independent external benchmarks with no self-definitional equations or fitted-as-prediction steps.

full rationale

QSVideo is an empirical systems paper whose central claims are measured accuracy lifts (e.g., +6.9–7.1 pp on LVBench, SoTA on StreamingBench under fixed 8/16/32-frame budgets) obtained by plugging a fine-tuned ranker + diversity/temporal selector into existing VLMs. The only training data is Video-Ranker-35K (sampled from LLaVA-Video-178K and auto-labeled by an off-the-shelf Qwen-VL-30B); the evaluation sets LVBench and StreamingBench are disjoint external benchmarks. No equation equates a reported quantity to a fitted parameter by construction, no uniqueness theorem is imported from the authors’ prior work, and the sole overlapping-author citation (SEAL) appears only as a baseline, not as a load-bearing premise. The derivation chain therefore reduces to ordinary supervised fine-tuning followed by independent test-set measurement; circularity score is zero.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on a small set of free design choices (lambda, temporal radius, window count, discrete 1–5 scores) and standard domain assumptions about video QA; the only invented entities are the named modules and the constructed ranking dataset. No new physical or mathematical primitives are postulated.

free parameters (4)
  • lambda (relevance-diversity trade-off) = 0.9
    Hand-chosen scalar in Eq. 6/8; ablation shows mild sensitivity, default 0.9 used for main results.
  • temporal radius r
    Controls local expansion around anchors for long videos; value not ablated.
  • number of temporal windows K = equal to selected frames (8/16/32)
    Equals the frame budget; partitions the video for coverage.
  • discrete score range {1,2,3,4,5} = 1-5
    Both importance weights and relevance scores are forced into five bins; choice is conventional but arbitrary.
axioms (4)
  • domain assumption Human video understanding balances relevance, diversity and temporal coverage; therefore an algorithmic retrieval system should do the same.
    Stated in the introduction and used to motivate the three-term objective (Eq. 1).
  • ad hoc to paper Object, Action and Location dimensions are sufficient to capture query-frame semantic relevance for video QA.
    Adopted without ablation against alternative taxonomies; drives the entire scoring function (Eq. 3–4).
  • domain assumption L2 distance on mean-pooled vision-encoder embeddings is a better diversity metric than cosine similarity.
    Claimed empirically in Section 3.3 and Appendix; used throughout QSRetrieval.
  • domain assumption Off-the-shelf large VLMs (Qwen-VL-30B) produce sufficiently accurate frame-level Object/Action/Location labels for supervised fine-tuning.
    Core of the training pipeline in Section 3.2; no human agreement study provided.
invented entities (3)
  • QSRanker (question analyzer + semantic ranker) no independent evidence
    purpose: Rewrite arbitrary questions and produce structured three-axis relevance scores.
    New modular component; independent evidence is only the downstream accuracy gains.
  • QSRetrieval (relevance-diversity temporal selection) no independent evidence
    purpose: Greedy selection that jointly optimizes score, L2 diversity and temporal coverage with long/streaming policies.
    New search procedure; evidence is empirical only.
  • Video-Ranker-35K no independent evidence
    purpose: Supervised fine-tuning set of 35k query-frame triples with auto-generated scores.
    Constructed dataset; no external validation or public release of labels beyond the construction recipe.

reviewed 2026-07-11 · how reviews work

0 comments
Cite this review

Pith. "Pith review of QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding." pith.science (2026). https://pith.science/paper/6EXQOVKH

@misc{pith2026260704559,
  author       = {Pith},
  title        = {Pith review of: QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EXQOVKH}},
  note         = {Machine review of arXiv:2607.04559}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The performance of vision-language models (VLMs) in video understanding declines with increasing video duration, as video moments unrelated to the query confuse their language components. Multimodal retrieval has emerged as a critical component of video understanding, addressing this challenge by localizing key visual evidence. However, existing multimodal retrieval methods suffer from biased relevance estimation, limited diversity, and temporal collapse. In this paper, we propose QSVideo, a unified framework that systematically addresses relevance, diversity, and temporal modeling in video retrieval. We first introduce a query-conditioned semantic ranker, QSRanker, which reformulates arbitrary questions into retrieval-friendly queries and estimates structured relevance along object, action, and location dimensions. Building upon this, we design QSRetrieval to jointly optimize relevance and diversity for more informative frame selection. Moreover, we propose temporal alignment strategies tailored for both long and streaming videos to improve evidence recall. Extensive experiments on long and streaming video benchmarks demonstrate that QSVideo greatly enhances video VLM performance under strict frame limit constraints. The code is available at https://github.com/human-analysis/QSVideo.

Figures

Figures reproduced from arXiv: 2607.04559 by Lan Wang, Vishnu Naresh Boddeti, Wei Ao.

Figure 1
Figure 1. Figure 1: QSVideo: query-conditioned semantic temporal retrieval for long and streaming videos. QSVideo includes QSRanker, QSRetrieval and a video VLM. QSRanker proposes a Question Analyzer (LLM) to understand arbitrary questions and transcribe original questions to retrieval-friendly queries. The question analyzer analyzes the questions and estimates the importance of Object, Action and Location. QSRanker designs a… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of QSVideo on LVBench. The number of selected frames is 16. We highlight frames including important supportive visual evidence. Boosting Video VLM Performance. The backbone VLM (MiniCPM-o 2.6) reports accuracy of 38.9%, 42.0% and 42.3% using 16 frames, 32 frames and 64 frames. QSVideo greatly improves accuracy by +6.9% and +7.1% using 16 frames and 32 frames. It indicates that explicitly mode… view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of Video-Ranker-35K. Comparison between different visual dis￾tance matrices: L2 distance (ours) versus cosine similarity [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 23 linked inside Pith

  1. [1]

    Anthropic: Claude 3.5 sonnet (2024)

    2024

  2. [2]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Arnab, A., Iscen, A., Caron, M., Fathi, A., Schmid, C.: Temporal chain of thought: Long-video understanding by thinking in frames. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-VL technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    In: International Conference on Computer Vision (ICCV) (2025)

    Bao, X., Xie, C., Tang, H., Weng, T., Wang, X., Zheng, Y., Wang, X.: DynImg: Key frames with visual prompts are good representation for multi-modal video understanding. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  6. [6]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Buch, S., Nagrani, A., Arnab, A., Schmid, C.: Flexible frame selection for effi- cient video reasoning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 29071–29082 (2025)

    2025

  7. [7]

    In: International Conference on Computer Vision (ICCV) (2025)

    Chatterjee, D., Remelli, E., Song, Y., Tekin, B., Mittal, A., Bhatnagar, et al.: Memory-efficient streaming VideoLLMs for real-time procedural video understand- ing. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  8. [8]

    In: International Conference on Computer Vision (ICCV) (2025)

    Chen, B., Yue, Z., Chen, S., Wang, Z., Liu, Y., Li, P., Wang, Y.: LVAgent: Long video understanding by multi-round dynamical collaboration of MLLM agents. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  9. [9]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Chen, J., Lv, Z., Wu, S., Lin, K.Q., Song, C., Gao, D., Liu, J.W., Gao, Z., Mao, D., Shou, M.Z.: VideoLLM-online: Online video large language model for streaming video. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18407–18418 (2024)

    2024

  10. [10]

    arXiv preprint arXiv:2411.18211 (2024)

    Chen, S., Lan, X., Yuan, Y., Jie, Z., Ma, L.: TimeMarker: A versatile video-LLM for long and short video understanding with superior temporal localization ability. arXiv preprint arXiv:2411.18211 (2024)

    Pith/arXiv arXiv 2024

  11. [12]

    arXiv preprint arXiv:2412.05271 (2024)

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)

    Pith/arXiv arXiv 2024

  12. [13]

    arXiv preprint arXiv:2404.16821 (2024)

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.: How far are we to GPT-4V? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404.16821 (2024)

    Pith/arXiv arXiv 2024

  13. [14]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24185–24198 (2024)

    2024

  14. [15]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025) QSVideo 17

    Diko, A., Wang, T., Swaileh, W., Sun, S., Patras, I.: ReWind: Understanding long videos with instructed learnable memory. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025) QSVideo 17

    2025

  15. [16]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Dong, Y., Liu, Z., Sun, H.L., Yang, J., Hu, W., Rao, Y., Liu, Z.: Insight-V: Ex- ploring long-chain visual reasoning with multimodal large language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  16. [17]

    In: IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR)

    Fabian Caba Heilbron, Victor Escorcia, B.G., Niebles, J.C.: ActivityNet: a large- scale video benchmark for human activity understanding. In: IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). pp. 961–970 (2015)

    2015

  17. [18]

    arXiv preprint arXiv:2501.01957 (2025)

    Fu, C., Lin, H., Wang, X., Zhang, Y.F., Shen, Y., Liu, X., Li, Y., Long, Z., Gao, H., Li, K., et al.: VITA-1.5: Towards GPT-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957 (2025)

    Pith/arXiv arXiv 2025

  18. [19]

    Gemini 3 pro model (2025)

    2025

  19. [20]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Ham- burger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4D: Around the world in 3,000 hours of egocentric video. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18995–19012 (2022)

    2022

  20. [21]

    arXiv preprint arXiv:2505.07062 (2025)

    Guo, D., Wu, F., Zhu, F., Leng, F., Shi, G., Chen, H., Fan, H., Wang, J., Jiang, J., Wang, J., et al.: Seed1.5-VL technical report. arXiv preprint arXiv:2505.07062 (2025)

    Pith/arXiv arXiv 2025

  21. [22]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Guo, W., Chen, Z., Wang, S., He, J., Xu, Y., Ye, J., Sun, Y., Xiong, H.: Logic-in- Frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  22. [23]

    International Conference on Learning Representations (ICLR)1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR)1(2), 3 (2022)

    2022

  23. [24]

    Hu, K., Gao, F., Nie, X., Zhou, P., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., et al.: M-LLM based video frame selection for efficient video understanding.In:IEEE/CVFConferenceonComputerVisionandPatternRecog- nition (CVPR). pp. 13702–13712 (2025)

    2025

  24. [25]

    arXiv preprint arXiv:2504.17447 (2025)

    Huang, D.A., Radhakrishnan, S., Yu, Z., Kautz, J.: FRAG: Frame selection aug- mented generation for long video and long document understanding. arXiv preprint arXiv:2504.17447 (2025)

    Pith/arXiv arXiv 2025

  25. [26]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Huang, Z., Li, X., Li, J., Wang, J., Zeng, X., Liang, C., Wu, T., Chen, X., Li, L., Wang, L.: Online video understanding: OVBench and VideoChat-Online. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3328–3338 (June 2025)

    2025

  26. [27]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.: GPT-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  27. [28]

    In: International Conference on Computer Vision (ICCV) (2025)

    Kang, H., Park, Y., Yoo, Y., Choi, Y., Kim, S.J.: Open-ended hierarchical stream- ing video understanding with vision language models. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  28. [29]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Kim,M.,Shim,K.,Choi,J.,Chang,S.:InfiniPot-V:Memory-constrainedKVcache compression for streaming video. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  29. [30]

    Li, B., Zhang, K., Zhang, H., Guo, D., Zhang, R., Li, F., Zhang, Y., Liu, Z., Li, C.: LLaVA-NeXT: Stronger LLMs supercharge multimodal capabilities in the wild (May 2024)

    2024

  30. [31]

    arXiv preprint arXiv:2408.03326 (2024) 18 W

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024) 18 W. Ao et al

    Pith/arXiv arXiv 2024

  31. [32]

    Science China Information Sciences 68(10), 200102 (2025)

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: VideoChat: Chat-centric video understanding. Science China Information Sciences 68(10), 200102 (2025)

    2025

  32. [33]

    arXiv preprint arXiv:2601.04720 (2026)

    Li, M., Zhang, Y., Long, D., Keqin, C., Song, S., Bai, S., Yang, Z., Xie, P., Yang, A., Liu, D., Zhou, J., Lin, J.: Qwen3-VL-Embedding and Qwen3-VL-Reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720 (2026)

    Pith/arXiv arXiv 2026

  33. [34]

    arXiv preprint arXiv:2311.17043 (2023)

    Li, Y., Wang, C., Ji, J.: LLaMA-VID: An image is worth multiple tokens in large language models. arXiv preprint arXiv:2311.17043 (2023)

    Pith/arXiv arXiv 2023

  34. [35]

    arXiv preprint arXiv:2311.10122 (2023)

    Lin, B., Zhu, B., Ye, Y., Cui, J., Ning, M., Jin, P., Li, L.Y.: Video-LLaVA: Learn- ing united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    Pith/arXiv arXiv 2023

  35. [36]

    arXiv preprint arXiv:2411.03628 (2024)

    Lin, J., Fang, Z., Chen, C., Wan, Z., Luo, F., Li, P., Liu, Y., Sun, M.: Streaming- Bench: Assessing the gap for MLLMs to achieve streaming video understanding. arXiv preprint arXiv:2411.03628 (2024)

    Pith/arXiv arXiv 2024

  36. [37]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  37. [38]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: LLaVA-NeXT: im- proved reasoning, ocr, and world knowledge (January 2024)

    2024

  38. [39]

    Annual Conference on Neural Information Processing Systems (NeurIPS)36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Annual Conference on Neural Information Processing Systems (NeurIPS)36, 34892–34916 (2023)

    2023

  39. [40]

    In: International Conference on Computer Vision (ICCV) (2025)

    Liu, R., Sun, S., Tang, H., Gao, W., Li, G.: Flow4Agent: Long-form video un- derstanding via motion prior from optical flow. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  40. [41]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Liu, S., Zhao, C., Xu, T., Ghanem, B.: BOLT: Boost large vision-language model without training for long-form video. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  41. [42]

    arXiv preprint arXiv:2412.04468 (2024)

    Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao, S., Gu, Y., Li, D., Li, X., Fang, Y., Chen, Y., Hsieh, C.Y., Huang, D.A., Cheng, A.C., Nath, V., Hu, J., Liu, S., Krishna, R., Xu, D., Wang, X., Molchanov, P., Kautz, J., Yin, H., Han, S., Lu, Y.: NVILA: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468 (2024)

    Pith/arXiv arXiv 2024

  42. [43]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Ma, Z., Gou, C., Shi, H., Sun, B., Li, S., Rezatofighi, H., Cai, J.: DrVideo: Doc- ument retrieval based long video understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  43. [44]

    In: Annual Meeting of the Association for Computational Linguistics (ACL) (2024)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-ChatGPT: Towards detailed video understanding via large language and vision models. In: Annual Meeting of the Association for Computational Linguistics (ACL) (2024)

    2024

  44. [45]

    Annual Conference on Neural Information Pro- cessing Systems (NeurIPS)35, 27730–27744 (2022)

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Annual Conference on Neural Information Pro- cessing Systems (NeurIPS)35, 27730–27744 (2022)

    2022

  45. [46]

    In: International Conference on Learning Representations (ICLR) (2026)

    Pan, J., Zhang, Q., Zhang, R., Lu, M., Wan, X., Zhang, Y., Liu, C., She, Q.: TimeSearch-R: Adaptive temporal search for long-form video understanding. In: International Conference on Learning Representations (ICLR) (2026)

    2026

  46. [47]

    MapReduce

    Pang, Z., Wang, Y.X.: Mr. Video: "MapReduce" is the principle for long video understanding. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025) QSVideo 19

    2025

  47. [48]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS)

    Qian, R.,Dong,X.,Zhang,P., Zang,Y.,Ding,S.,Lin,D., Wang,J.:Streaminglong video understanding with large language models. In: Annual Conference on Neural Information Processing Systems (NeurIPS). vol. 37, pp. 119336–119360 (2024)

    2024

  48. [49]

    In: International Conference for High Per- formance Computing, Networking, Storage and Analysis

    Rajbhandari, S., Rasley, J., Ruwase, O., He, Y.: Zero: Memory optimizations to- ward training trillion parameter models. In: International Conference for High Per- formance Computing, Networking, Storage and Analysis. pp. 1–16. IEEE (2020)

    2020

  49. [50]

    Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Shen, X., Zhang, W., Chen, J., Elhoseiny, M.: Vgent: Graph-based retrieval- reasoning-augmented generation for long video understanding. Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  50. [51]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., Zhao, B.: Video-XL: Extra-long vision language model for hour-scale video understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  51. [52]

    In: European Conference on Computer Vision (ECCV)

    Sigurdsson, G.A., Varol, G., Wang, X., Farhadi, A., Laptev, I., Gupta, A.: Hol- lywood in homes: Crowdsourcing data collection for activity understanding. In: European Conference on Computer Vision (ECCV). pp. 510–526. Springer (2016)

    2016

  52. [53]

    arXiv preprint arXiv:2601.03267 (2025)

    Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al.: Openai GPT-5 system card. arXiv preprint arXiv:2601.03267 (2025)

    Pith/arXiv arXiv 2025

  53. [54]

    In: International Conference on Computer Vision (ICCV) (2025)

    Suo, Y., Ma, F., Zhu, L., Wang, T., Rao, F., Yang, Y.: From trial to triumph: Advancing long video understanding via visual context sample scaling and self- reward alignment. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  54. [55]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Tang, X., Qiu, J., Xie, L., Tian, Y., Jiao, J., Ye, Q.: Adaptive keyframe sampling for long video understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  55. [56]

    arXiv preprint arXiv:2505.09388 (2025)

    Team, Q.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  56. [57]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Wang,L.,Chen,Y.,Tran,D.,Boddeti,V.N.,Chu,W.S.:SEAL:Semanticattention learning for long video representation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  57. [58]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  58. [59]

    arXiv preprint arXiv:2406.08035 (2024)

    Wang, W., He, Z., Hong, W., Cheng, Y., Zhang, X., Qi, J., Gu, X., Huang, S., Xu, B., Dong, Y., et al.: LVBench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035 (2024)

    Pith/arXiv arXiv 2024

  59. [60]

    In: Findings of the Association for Computational Linguistics: ACL 2025

    Wang, X., Si, Q., Zhu, S., Wu, J., Cao, L., Nie, L.: AdaReTaKe: Adaptive redun- dancy reduction to perceive longer for video-language understanding. In: Findings of the Association for Computational Linguistics: ACL 2025. pp. 5417–5432 (2025)

    2025

  60. [61]

    In: International Conference on Learning Representations (ICLR) (2023)

    Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al.: InternVid: A large-scale video-text dataset for multimodal understanding and generation. In: International Conference on Learning Representations (ICLR) (2023)

    2023

  61. [62]

    In: International Conference on Computer Vision (ICCV) (2025)

    Wang, Y., Song, Y., Xie, C., Liu, Y., Zheng, Z.: VideoLLaMB: Long-form video understanding with recurrent memory bridges. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  62. [63]

    In: Annual Conference on Neural Information Process- ing Systems (NeurIPS)

    Wu, S., Chen, J., Lin, K.Q., Wang, Q., Gao, Y., Xu, Q., Xu, T., Hu, Y., Chen, E., Shou, M.Z.: VideoLLM-MoD: Efficient video-language streaming with mixture-of- depths vision computation. In: Annual Conference on Neural Information Process- ing Systems (NeurIPS). vol. 37, pp. 109922–109947 (2024) 20 W. Ao et al

    2024

  63. [64]

    In: International Confer- ence on Learning Representations (ICLR) (2026)

    Wu, Z., Wang, X., Huang, L., Xu, T., Peng, P.: A training-free framework for long video understanding via video-query-options similarity. In: International Confer- ence on Learning Representations (ICLR) (2026)

    2026

  64. [65]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Xiao, J., Shang, X., Yao, A., Chua, T.S.: NExT-QA: Next phase of question- answering to explaining temporal actions. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9777–9786 (June 2021)

    2021

  65. [66]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., Guo, B.: Advancing high-resolution video-language representation with large-scale video transcriptions. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5036–5045 (2022)

    2022

  66. [67]

    In: An- nual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Xue, Z., Zhang, J., Xie, X., Cai, Y., Liu, Y., Li, X., Tao, D.: AdaVideoRAG: Omni- contextual adaptive retrieval-augmented efficient long video understanding. In: An- nual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  67. [68]

    In: International Conference on Computer Vision (ICCV) (2024)

    Yang, Z., Chen, D., Yu, X., Shen, M., Gan, C.: VCA: Video curious agent for long video understanding. In: International Conference on Computer Vision (ICCV) (2024)

    2024

  68. [69]

    In: International Conference on Learning Representations (ICLR) (2025)

    Yang, Z., Hu, Y., Du, Z., Xue, D., Qian, S., Wu, J., Yang, F., Dong, W., Xu, C.: SVBench: A benchmark with temporal multi-turn dialogues for streaming video understanding. In: International Conference on Learning Representations (ICLR) (2025)

    2025

  69. [70]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Yang, Z., Zhang, K., Hu, Y., Wang, B., Qian, S., Wen, B., Yang, F., Gao, T., Dong, W., Xu, C.: LiveStar: Live streaming assistant for real-world online video understanding. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  70. [71]

    arXiv preprint arXiv:2408.01800 (2024)

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al.: MiniCPM-V: A GPT-4V level MLLM on your phone. arXiv preprint arXiv:2408.01800 (2024)

    Pith/arXiv arXiv 2024

  71. [72]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Ye, J., Wang, Z., Sun, H., Chandrasegaran, K., Durante, Z., Eyzaguirre, C., Bisk, Y., Niebles, J.C., Adeli, E., Fei-Fei, L., et al.: Re-thinking temporal search for long-form video understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  72. [73]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Ye, X., Gan, Y., Ge, Y., Zhang, X.P., Tang, Y.: ATP-LLaVA: Adaptive token pruning for large vision language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  73. [74]

    Annual Conference on Neural Information Processing Systems (NeurIPS) pp

    Yu, S., Cho, J., Yadav, P., Bansal, M.: Self-chained image-language model for video localization and question answering. Annual Conference on Neural Information Processing Systems (NeurIPS) pp. 76749–76771 (2023)

    2023

  74. [75]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Zeng, X., Qiu, K., Zhang, Q., Li, X., Wang, J., Li, J., Yan, Z., Tian, K., Tian, M., Zhao, X., et al.: StreamForest: Efficient online video understanding with persistent event memory. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  75. [76]

    arXiv preprint arXiv:2306.02858 (2023)

    Zhang, H., Li, X., Bing, L.: Video-LLaMA: An instruction-tuned audio-visual lan- guage model for video understanding. arXiv preprint arXiv:2306.02858 (2023)

    Pith/arXiv arXiv 2023

  76. [77]

    In: International Conference on Computer Vision (ICCV) (2025)

    Zhang, H., Wang, Y., Tang, Y., Liu, Y., Feng, J., Jin, X.: Flash-VStream: Efficient real-time understanding for long video streams. In: International Conference on Computer Vision (ICCV) (2025)

    2025

  77. [78]

    In: International Conference on Learning Representations (ICLR) (2026) QSVideo 21

    Zhang, K., Yang, Z., Wang, B., Qian, S., Xu, C.: QueryStream: Advancing stream- ing video understanding with query-aware pruning and proactive response. In: International Conference on Learning Representations (ICLR) (2026) QSVideo 21

    2026

  78. [79]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Zhang, X., Jia, Z., Guo, Z., Li, J., Li, B., Li, H., Lu, Y.: Deep video discovery: Agen- tic search with tool use for long-form video understanding. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

  79. [80]

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: LLaVA-Video: Video instructiontuningwithsyntheticdata.TransactionsonMachineLearningResearch (June 2025)

    2025

  80. [81]

    In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    Zhang, Y., Lu, Y., Wang, T., Rao, F., Yang, Y., Zhu, L.: FlexSelect: Flexible token selection for efficient long video understanding. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2025)

    2025

Showing first 80 references.

This paper was first reviewed by grok-4.5 on July 11, 2026.