Pith. sign in

REVIEW 3 major objections 6 minor 62 references

Long-video models reason better when a fixed frame budget is split, by query, between spatial zoom-ins and temporal zoom-outs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-31 00:36 UTC pith:C2RWHQKV

load-bearing objection Solid training-free systems paper with real gains under fixed MLLM slots; the hyperframe packing makes “matched frame budget” a bit slippery, and the AKS table rows look broken. the 3 major comments →

arxiv 2607.25125 v1 pith:C2RWHQKV submitted 2026-07-27 cs.CV

LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos

Ce Zhang , Jinxi He , Katia Sycara , Yaqi Xie This is my paper
classification cs.CV
keywords long-form video understandingkeyframe samplingmulti-modal large language modelsspatio-temporal reasoningvisual promptingvideo graphtraining-free
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long multi-modal language models cannot feed whole videos into their context windows, so they must pick a small set of frames. Prior keyframe pickers treat every frame as a single, fixed-granularity unit and therefore force a hard trade-off between fine spatial detail and broad temporal coverage. LENS claims that this trade-off can be made query-dependent and training-free: a lightweight controller reads the question and divides the same frame budget between spatial zoom-ins (CLIP-attention masks that highlight query-relevant regions inside a frame) and temporal zoom-outs (graph-propagated relevance plus multi-frame “hyperframes” that widen the temporal field of view). On standard long-video QA benchmarks the resulting multi-granular evidence set consistently beats uniform sampling and prior training-free samplers under matched budgets, lifting Video-MME accuracy from 53.3% to 60.7% with an 8-frame Qwen2.5-VL backbone and producing similar gains on other models and benchmarks. A sympathetic reader cares because the method is plug-and-play, adds only modest runtime, and shows that how evidence is allocated across scale matters as much as which frames are chosen.

Core claim

Under a fixed frame budget, adaptively allocating some frames to query-conditioned spatial zoom-ins and the rest to graph-based temporal zoom-outs supplies complementary multi-granular evidence that lets off-the-shelf video MLLMs answer long-form questions more accurately than uniform sampling or prior training-free keyframe methods.

What carries the argument

LENS’s dual zoom plus adaptive allocator: spatial zoom-in (CLIP patch-attention visual prompting on watershed-selected frames) and temporal zoom-out (pairwise visual-similarity graph message-passing followed by hyperframe aggregation), with an MLLM controller that outputs the budget split ratio r from the text query alone.

Load-bearing premise

Off-the-shelf image–text similarity scores, CLIP attention maps, and pairwise visual-similarity graphs are faithful enough proxies for “what the question needs to see” that watershed selection and hyperframe packing will surface the right cues for arbitrary long-video questions.

What would settle it

On the same Video-MME / LongVideoBench / MLVU splits and identical frame budgets, replace LENS’s adaptive dual-zoom set with the best prior sampler (or with fixed 50/50 or all-spatial / all-temporal ablations) and check whether the reported overall accuracy gaps (e.g., 60.7% vs 58.7% vs 53.3% at 8 frames on Qwen2.5-VL) disappear or reverse.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Keyframe sampling for long video should treat spatial resolution and temporal coverage as a query-conditioned budget allocation problem, not only a frame-ranking problem.
  • Training-free dual-scale selection can be dropped in front of open-source and proprietary video MLLMs without weight updates and still raise accuracy under tight context limits.
  • Detail-oriented questions should receive more spatial zoom-ins; event-level or sequence questions should receive more temporal zoom-outs, and an LLM can decide the split from text alone.
  • Modest extra compute (graph message-passing and attention prompting) is enough to outperform relevance-only and multi-resolution baselines while remaining far cheaper than agentic or RAG long-video pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the allocator’s decision correlates with question type, future work could cache or distill the split ratio so the controller call itself becomes free at inference time.
  • The same spatial-vs-temporal budget idea may transfer to other token-scarce settings (long documents with figures, multi-camera streams) where “zoom in on a region” and “summarize a window” are natural dual operations.
  • Failure modes on abstract or domain-specific queries noted in the limitations suggest pairing LENS with a lightweight domain adapter or retrieval step rather than replacing the dual-zoom core.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents LENS, a training-free keyframe sampling framework for long-form video QA with MLLMs. A fixed frame budget B is split by an MLLM-predicted ratio r between (i) a spatial zoom-in branch that selects query-relevant frames via CLIP/BLIP similarity plus watershed segmentation and overlays CLIP-attention-derived saliency masks, and (ii) a temporal zoom-out branch that propagates similarity scores over a pairwise-similarity video graph (Zhou et al.-style label propagation) and packs each selected frame with τ−1 neighbors into a downsampled "hyperframe." On Video-MME, LongVideoBench, and MLVU with Qwen2.5-VL at 8/16/32 frame budgets, LENS reports consistent gains over uniform sampling and over BOLT, AKS, and Q-Frame (e.g., 53.3→60.7 overall on Video-MME at 8 frames), with transfer results on LLaVA-OneVision and GPT-5-Mini, component ablations (Table 3), hyperparameter sweeps (Table A1), and released code.

Significance. If the central comparison holds up, this is a useful, practical contribution: a training-free, plug-and-play sampler with multi-benchmark, multi-budget, multi-backbone evaluation, re-implemented baselines (a genuine plus for comparability), component-level ablations isolating each branch and the allocator, appendix hyperparameter sweeps showing stability, the allocation prompt and implementation details in the appendix, and public code. The efficiency accounting (Table 2 and the per-query breakdown) is also unusually transparent. The main caveat on significance is that the headline claim — consistent outperformance "under matched frame budgets" — is currently entangled with a slot-accounting issue (Major Comment 1): the temporal branch's hyperframes show the MLLM content from substantially more distinct source frames than the nominal budget suggests, and the paper's own Table 1 shows coverage-matched baselines meeting or beating LENS at lower nominal budgets. The τ=1 ablation (59.7 overall at 8 slots) indicates the method likely retains an edge without any extra coverage, so the contribution is probably real but its magnitude is uncalibrated.

major comments (3)
  1. [§3.3–3.4, Table 1] The central claim of 'consistent outperformance under matched frame budgets' rests on slot accounting that is not matched in terms of distinct visual content. With default τ=4 (§4.1), each temporal-branch slot concatenates 4 source frames; at r≈0.5 an 8-slot budget shows the MLLM content from up to ~4 + 4×4 = 20 distinct frames, versus 8 for uniform/BOLT/AKS/Q-Frame. Table 1's own cross-budget rows show this is load-bearing: uniform@32 (61.1) exceeds LENS@8 (60.7), Q-Frame@16 (60.7) ties it, and BOLT@32 (64.1) exceeds LENS@16 (63.1) — i.e., baselines at comparable or smaller effective distinct-frame coverage match or beat LENS at lower nominal budgets. Only LENS@32 (67.1) lacks a coverage-matched baseline. I do not think this invalidates the method — the τ=1 ablation in Table A1 (59.7 at 8 slots, still above Q-Frame@8's 58.7) shows much of the gain survives without aggregation — but the
  2. [§3.3 (Hyperframe Aggregation), §B.1] The claim that hyperframes keep 'the token budget fixed' is asserted but never substantiated with numbers. The hyperframe is 'downsampled to a lower spatial resolution' — the target resolution is not specified anywhere, and for Qwen2.5-VL, which uses dynamic-resolution ViT encoding, a 4-wide concatenated composite downsampled to an unspecified size may well consume more visual tokens than a single native frame. Please report (i) the exact downsampling resolution and (ii) measured visual-token counts per input for LENS vs. uniform at each nominal budget and backbone. This is a prerequisite for either resolution of Major Comment 1.
  3. [§3.4, §4.3 (Budget Allocation)] The adaptivity claim is under-supported relative to its prominence in the title and abstract. Adaptive allocation beats the fixed 4/4 split by only +0.5 overall (60.7 vs 60.2, Table 3), and no statistics on the controller's actual behavior are given: the prompt (§B.2) anchors r to discrete bands, but the distribution of predicted r across benchmarks, its correlation with query type, and per-query-type accuracy vs. the fixed split are all unreported. Without these, 'adaptively allocates frame budgets based on the text query' is a design description rather than an evidenced finding. A histogram of r plus a breakdown of gains on detail- vs. event-type questions (Video-MME has category labels) would close this gap cheaply.
minor comments (6)
  1. [§3.2, Eq. (2)–(3)] The citation supporting 'the final similarity is primarily governed by the deepest MSA layers' is given as [19,45], but [19] is the visual instruction tuning paper (LLaVA), which does not establish this; presumably a CLIP-decomposition reference (e.g., [11] or related interpretability work) was intended. Also, V_n^(l,h) = W_v z_n is defined but never used; s(n,l,h) should be defined explicitly in terms of it.
  2. [§4.2] Typographical/grammatical: 'reaching 60.6% on LVB and 66.7 on MLVU' (missing %); 'as more budgets becomes available'; 'fullvideosintothemodel' and similar missing-space artifacts in the introduction (likely PDF extraction, but worth checking the source).
  3. [Table 2 / §4.3] The runtime breakdown (15.8s/query vs 11.0s baseline) is welcome. Please clarify whether the 2.1s budget-allocation call uses the QA backbone itself (an extra MLLM forward pass per query) and how this cost is handled for the GPT-5-Mini API setting, where the controller call and QA call are both paid API requests.
  4. [§4.1 / §B.1] Decoding parameters are said to be 'default'; please state the temperature explicitly and, if greedy decoding is used, note that results are single-run deterministic. Otherwise some estimate of run-to-run variance (or at least per-benchmark question counts enabling significance reasoning) would strengthen the 2–3 point margins in Table 1.
  5. [Table A3 (MLVU categories)] The Anomaly/Count category (AC) jumps from 19.0 to 40.9 at 8 frames — by far the largest category gain. A sentence explaining which component drives this (presumably temporal zoom-out coverage) would be informative, especially in light of Major Comment 1.
  6. [Fig. 3] Please mark which displayed frames are hyperframes (with their τ window) versus spatially prompted frames; the green circle/hexagon legend is hard to read at print size and the downsampling of hyperframes is not visually evident.

Circularity Check

0 steps flagged

No circularity: empirical keyframe-sampling method evaluated on external benchmarks; nothing is predicted by construction from fitted inputs.

full rationale

LENS is a training-free engineering pipeline (CLIP/BLIP scoring, attention-mask visual prompting, graph message-passing, hyperframe packing, LLM budget controller) whose claimed contribution is higher VQA accuracy under a stated frame-slot budget. Reported numbers are external benchmark accuracies (Video-MME, LVB, MLVU, EgoSchema, NExT-QA), not quantities algebraically forced by a fit. Hyperparameters (σ, α, τ) are chosen and ablated in the appendix rather than laundered into “predictions.” The allocation ratio r is an LLM control signal from a fixed prompt, not a parameter fitted to the target metrics. Author self-citations in the bibliography are background and not load-bearing uniqueness theorems. The skeptic concern about hyperframes packing more source frames into each slot is a matched-budget / fairness issue, not circularity: the paper does not define success as equal to that packing, and the result is still an external accuracy number. No self-definitional loop, fitted-input-as-prediction, or ansatz-via-self-citation chain is present.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

Load-bearing content is engineering assumptions and a few hand-set scalars, not new physical entities. The claim rests on pretrained CLIP/BLIP as relevance oracles, watershed diversity, graph label propagation as temporal structure, hyperframe packing as a valid temporal zoom, and an MLLM prompt as budget controller—plus standard MLLM QA evaluation protocol.

free parameters (5)
  • Gaussian kernel bandwidth σ = 0.3
    Sets edge affinities in the video graph; chosen by ablation, default 0.3.
  • Message-passing mix α = 0.8
    Balances unary CLIP–query scores vs graph smoothing; default 0.8 from ablation.
  • Hyperframe width τ = 4
    Number of neighboring frames concatenated per temporal slot; default 4.
  • Allocation ratio r (and discrete B_spa/B_tem) = query-dependent in [0,1]
    Per-query fraction of frame budget sent to spatial vs temporal branch; produced by prompted MLLM controller, not learned end-to-end.
  • Deep-layer range L' for CLIP attribution = terminal layers (unspecified exact L')
    Which MSA layers contribute to patch relevance; taken from prior CLIP interpretation practice, not cross-validated in main text.
axioms (6)
  • domain assumption Pretrained image–text models (CLIP/BLIP) cosine similarity is a sufficient ranking signal for query-relevant frames in long videos.
    Candidate selection in §3.2 Eq. (1) and temporal priors y^(0) depend entirely on this.
  • domain assumption CLIP [CLS]-to-patch attention plus query-aligned value contributions yield masks that amplify task-critical spatial evidence without harming MLLM perception.
    Spatial zoom-in pipeline §3.2 (Eqs. 2–4 and soft-OR fusion) assumes attribution maps are actionable visual prompts.
  • domain assumption A sparse visual-similarity graph with Gaussian affinities and label propagation recovers globally coherent temporal relevance.
    Temporal branch §3.3 Eqs. (5)–(6) treats this construction as an adequate video structure model.
  • ad hoc to paper Concatenating τ neighbors and downsampling into one hyperframe trades spatial detail for usable short-range temporal context inside a single frame slot.
    Hyperframe aggregation §3.3 is a design choice specific to LENS’s budget accounting.
  • ad hoc to paper A prompted MLLM can classify whether a question needs spatial detail vs temporal coverage well enough to set r.
    Adaptive allocator §3.4 and Appendix prompt; paper shows fixed 4/4 is already strong, adaptive slightly better.
  • domain assumption Watershed partitioning of the similarity curve yields temporally diverse representatives.
    Used for both branches’ candidate selection (§3.2–3.3).
invented entities (2)
  • Hyperframe (spatially concatenated, downsampled multi-frame composite) no independent evidence
    purpose: Spend one frame budget slot on short-range temporal context for the temporal zoom-out branch.
    Defined in §3.3 as the concrete realization of temporal zoom-out under fixed token/frame counts; mechanism is engineered, not an external natural kind.
  • LENS dual-branch adaptive budget controller (spatial zoom-in vs temporal zoom-out split) no independent evidence
    purpose: Map each text query to a partition of the frame budget across the two evidence modes.
    Core system abstraction of the paper (§3.4); evaluated only inside this pipeline’s benchmarks.

pith-pipeline@v1.2.0-grok45-kimik3 · 23310 in / 3689 out tokens · 77370 ms · 2026-07-31T00:36:57.372060+00:00 · methodology

0 comments
read the original abstract

Despite rapid progress in Multi-modal Large Language Models (MLLMs), understanding long-form videos is still bottlenecked by limited context windows. While recent keyframe sampling methods attempt to mitigate this by distilling video inputs into a compact set of query-relevant frames, navigating the vast spatio-temporal search space remains challenging, as spatial detail and temporal coverage often conflict. To address this, we introduce LENS, a training-free keyframe sampling framework that dynamically decides when to zoom in for fine-grained details and when to zoom out for broader context based on the text query. Concretely, LENS adaptively allocates a limited frame budget between spatial zoom-ins, which highlight query-relevant regions within individual frames, and temporal zoom-outs, which expand the temporal scope through multi-frame aggregation, enabling the model to reason across multiple granularities while capturing both high-fidelity details and long-range context. Across diverse long-form video benchmarks, LENS consistently outperforms prior state-of-the-art keyframe sampling methods and delivers substantial gains over uniform sampling, improving Video-MME accuracy from 53.3% to 60.7% with Qwen2.5-VL.Code is available at https://github.com/zhangce01/LENS.

Figures

Figures reproduced from arXiv: 2607.25125 by Ce Zhang, Jinxi He, Katia Sycara, Yaqi Xie.

Figure 1
Figure 1. Figure 1: Illustration of LENS for multi-granular video understanding [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of LENS. (A) Spatial zoom-in selects query-relevant frames using image–text similarity and enhances fine-grained evidence via CLIP attention–based visual prompting. (B) Temporal zoom-out constructs a video graph to model inter￾frame relationships and propagates relevance scores through message passing, enabling long-range reasoning and the selection of context-rich hyperframes. (C) An adaptive bud… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison on two representative samples from the [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 6 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bi, J., Bellos, F., Guo, J., Li, Y., Huang, C., Tang, Y., Song, L., Liang, S., Zhang, Z., Corso, J.J., et al.: When to think and when to look: Uncertainty-guided lookback. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5104–5113 (2026)

    2026

  3. [3]

    arXiv preprint arXiv:2503.11557 (2025)

    Bi, J., Guo, J., Liang, S., Sun, G., Song, L., Tang, Y., He, J., Wu, J., Vosoughi, A., Chen, C., et al.: Verify: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity. arXiv preprint arXiv:2503.11557 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    arXiv preprint arXiv:2504.03151 (2025)

    Bi, J., Liang, S., Zhou, X., Liu, P., Guo, J., Tang, Y., Song, L., Huang, C., Vosoughi, A., Sun, G., et al.: Why reasoning matters? a survey of advancements in multimodal reasoning (v1). arXiv preprint arXiv:2504.03151 (2025)

    arXiv 2025

  5. [5]

    arXiv preprint arXiv:2510.20696 (2025) 16 C

    Bi, J., Sun, G., Vosoughi, A., Chen, C., Xu, C.: Diagnosing visual reasoning: Challenges, insights, and a path forward. arXiv preprint arXiv:2510.20696 (2025) 16 C. Zhang et al

    arXiv 2025

  6. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Buch, S., Nagrani, A., Arnab, A., Schmid, C.: Flexible frame selection for efficient video reasoning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 29071–29082 (2025)

    2025

  7. [7]

    In: International Conference on Learning Representations (2025),https: //openreview.net/forum?id=wCXAlfvCy6

    Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., He, Y., Yin, H., Molchanov, P., Kautz, J., Fan, L., Zhu, Y., Lu, Y., Han, S.: LongVILA: Scaling long-context visual language models for long videos. In: International Conference on Learning Representations (2025),https: //openreview.net/forum?id=wCXAlfvCy6

    2025

  8. [8]

    In: International Conference on Machine Learning

    Cheng, C., Guan, J., Wu, W., Yan, R.: Scaling video-language models to 10k frames via hierarchical differential distillation. In: International Conference on Machine Learning. pp. 9964–9981. PMLR (2025)

    2025

  9. [9]

    arXiv preprint arXiv:2406.07476 (2024)

    Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24108–24118 (2025)

    2025

  11. [11]

    In: International Conference on Learning Representations (2024),https://openreview.net/forum?id=5Ca9sSzuDp

    Gandelsman, Y., Efros, A.A., Steinhardt, J.: Interpreting CLIP’s image represen- tation via text-based decomposition. In: International Conference on Learning Representations (2024),https://openreview.net/forum?id=5Ca9sSzuDp

    2024

  12. [12]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (4), 532–550 (1987)

    Haralick, R.M., Sternberg, S.R., Zhuang, X.: Image analysis using mathematical morphology. IEEE Transactions on Pattern Analysis and Machine Intelligence (4), 532–550 (1987)

    1987

  13. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hu, K., Gao, F., Nie, X., Zhou, P., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., et al.: M-llm based video frame selection for efficient video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13702–13712 (2025)

    2025

  14. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jin, P., Takanobu, R., Zhang, W., Cao, X., Yuan, L.: Chat-univi: Unified visual representation empowers large language models with image and video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13700–13710 (2024)

    2024

  15. [15]

    Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=zKv8qULV6n

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., Li, C.: LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=zKv8qULV6n

    2025

  16. [16]

    In: International Conference on Machine Learning

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)

    2022

  17. [17]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 5971–5984 (2024)

    2024

  18. [18]

    io/blog/2024-01-30-llava-next/

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llava-next: Improved reasoning, ocr, and world knowledge (January 2024),https://llava-vl.github. io/blog/2024-01-30-llava-next/

    2024

  19. [19]

    Advances in Neural Information Processing Systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in Neural Information Processing Systems36, 34892–34916 (2023)

    2023

  20. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, S., Zhang, C.L., Zhao, C., Ghanem, B.: End-to-end temporal action detec- tion with 1b parameters across 1000 frames. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18591–18601 (2024) LENS: Adaptive Spatio-Temporal Zooming in Long-Form Videos 17

    2024

  21. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, S., Zhao, C., Xu, T., Ghanem, B.: Bolt: Boost large vision-language model with- out training for long-form video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3318–3327 (2025)

    2025

  22. [22]

    In: Advances in Neural Information Processing Systems (2025), https://openreview.net/forum?id=QaZxGWlbgO

    Luo, Y., Zheng, X., Li, G., Yin, S., Lin, H., Fu, C., Huang, J., Ji, J., Chao, F., Luo, J., Ji, R.: Video-RAG: Visually-aligned retrieval-augmented long video comprehension. In: Advances in Neural Information Processing Systems (2025), https://openreview.net/forum?id=QaZxGWlbgO

    2025

  23. [23]

    Advances in Neural Information Processing Systems36, 46212–46244 (2023)

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems36, 46212–46244 (2023)

    2023

  24. [24]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Pi, R., Han, T., Zhang, J., Xie, Y., Pan, R., Lian, Q., Dong, H., Zhang, J., Zhang, T.: Mllm-protector: Ensuring mllm’s safety without hurting performance. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 16012–16027 (2024)

    2024

  25. [25]

    In: International Conference on Machine Learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)

    2021

  26. [26]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ren, S., Yao, L., Li, S., Sun, X., Hou, L.: Timechat: A time-sensitive multi- modal large language model for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14313–14323 (2024)

    2024

  27. [27]

    In: International Conference on Machine Learning

    Santos, S., Farinhas, A., McNamee, D.C., Martins, A.:inf-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolida- tion. In: International Conference on Machine Learning. pp. 52877–52893. PMLR (2025)

    2025

  28. [28]

    In: International Conference on Machine Learning

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., et al.: Longvu: Spatiotemporal adaptive compression for long video-language understanding. In: International Conference on Machine Learning. pp. 54582–54599. PMLR (2025)

    2025

  29. [29]

    In: Advances in Neural Information Processing Systems (2025),https://openreview.net/forum? id=5xPvWat3IX

    Shen, X., Zhang, W., Chen, J., Elhoseiny, M.: Vgent: Graph-based retrieval- reasoning-augmented generation for long video understanding. In: Advances in Neural Information Processing Systems (2025),https://openreview.net/forum? id=5xPvWat3IX

    2025

  30. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., Zhao, B.: Video-xl: Extra-long vision language model for hour-scale video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26160–26169 (2025)

    2025

  31. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tang, X., Qiu, J., Xie, L., Tian, Y., Jiao, J., Ye, Q.: Adaptive keyframe sampling for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 29118–29128 (2025)

    2025

  32. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tang, Y.Y., Shimada, D., Hua, H., Huang, C., Bi, J., Feris, R., Xu, C.: Video-r4: Reinforcing text-rich video reasoning with visual rumination. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8314–8325 (2026)

    2026

  33. [33]

    IEEE Transactions on Circuits and Systems for Video Technology (2025)

    Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al.: Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology (2025)

    2025

  34. [34]

    arXiv preprint arXiv:2507.01949 (2025) 18 C

    Team, K.K., Yang, B., Wen, B., Liu, C., Chu, C., Song, C., Rao, C., Yi, C., Li, D., Zang, D., et al.: Kwai keye-vl technical report. arXiv preprint arXiv:2507.01949 (2025) 18 C. Zhang et al

    Pith/arXiv arXiv 2025

  35. [35]

    In: European Conference on Computer Vision

    Wang, X., Zhang, Y., Zohar, O., Yeung-Levy, S.: Videoagent: Long-form video understanding with large language model as agent. In: European Conference on Computer Vision. pp. 58–76. Springer (2024)

    2024

  36. [36]

    IEEE Transactions on Image Processing 13(4), 600–612 (2004)

    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004)

    2004

  37. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., Bansal, M.: Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3272–3283 (2025)

    2025

  38. [38]

    Advances in Neural Information Processing Systems37, 28828–28857 (2024)

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context in- terleaved video-language understanding. Advances in Neural Information Processing Systems37, 28828–28857 (2024)

    2024

  39. [39]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa: Next phase of question-answering to explaining temporal actions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9777–9786 (2021)

    2021

  40. [40]

    In: Conference on Language Modeling (2025),https://openreview.net/forum?id=L7jS3peM3w

    Xu, M., Gao, M., Li, S., Lu, J., Gan, Z., Lai, Z., Cao, M., Kang, K., Yang, Y., Dehghan, A.: Slowfast-LLaVA-1.5: A family of token-efficient video large language models for long-form video understanding. In: Conference on Language Modeling (2025),https://openreview.net/forum?id=L7jS3peM3w

    2025

  41. [41]

    net/forum?id=FDAI0PY9Qp

    Xue, Z., Zhang, J., Xie, X., yuxuan cai, Liu, Y., Li, X., Tao, D.: AdavideoRAG: Omni-contextualadaptive retrieval-augmented efficientlong videounderstanding.In: Advances in Neural Information Processing Systems (2025),https://openreview. net/forum?id=FDAI0PY9Qp

    2025

  42. [42]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Yan, S., Xiong, X., Nagrani, A., Arnab, A., Wang, Z., Ge, W., Ross, D., Schmid, C.: Unloc: A unified framework for video localization tasks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13623–13633 (2023)

    2023

  43. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19792– 19802 (2025)

    2025

  44. [44]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ye, J., Wang, Z., Sun, H., Chandrasegaran, K., Durante, Z., Eyzaguirre, C., Bisk, Y., Niebles, J.C., Adeli, E., Fei-Fei, L., et al.: Re-thinking temporal search for long-form video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8579–8591 (2025)

    2025

  45. [45]

    In: European Conference on Computer Vision

    Yu, R., Yu, W., Wang, X.: Attention prompting on image for large vision-language models. In: European Conference on Computer Vision. pp. 251–268. Springer (2024)

    2024

  46. [46]

    Advances in Neural Information Processing Systems36, 76749–76771 (2023)

    Yu, S., Cho, J., Yadav, P., Bansal, M.: Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems36, 76749–76771 (2023)

    2023

  47. [47]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, C., He, J., He, J., Sycara, K., Xie, Y.: Evolving contextual safety in multi- modal large language models via inference-time self-reflective memory. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 41182–41192 (2026)

    2026

  48. [48]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing

    Zhang, C., Lu, T., Islam, M.M., Wang, Z., Yu, S., Bansal, M., Bertasius, G.: A simple llm framework for long-range video question-answering. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 21715–21737 (2024)

    2024

  49. [49]

    LENS: Adaptive Spatio-Temporal Zooming in Long-Form Videos 19 Transactions on Machine Learning Research (2026),https://openreview.net/ forum?id=KZYhyilFnt

    Zhang, C., Ma, K., Fang, T., Yu, W., Zhang, H., Zhang, Z., Mi, H., Yu, D.: VScan: Rethinking visual token reduction for efficient large vision-language models. LENS: Adaptive Spatio-Temporal Zooming in Long-Form Videos 19 Transactions on Machine Learning Research (2026),https://openreview.net/ forum?id=KZYhyilFnt

    2026

  50. [50]

    Advances in Neural Information Processing Systems37, 32111–32136 (2024)

    Zhang, C., Stepputtis, S., Sycara, K., Xie, Y.: Dual prototype evolving for test- time generalization of vision-language models. Advances in Neural Information Processing Systems37, 32111–32136 (2024)

    2024

  51. [51]

    In: The Thirteenth International Conference on Learning Representations (2025),https: //openreview.net/forum?id=tTBXePRKSx

    Zhang, C., Wan, Z., Kan, Z., Ma, M.Q., Stepputtis, S., Ramanan, D., Salakhutdinov, R., Morency, L.P., Sycara, K.P., Xie, Y.: Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. In: The Thirteenth International Conference on Learning Representations (2025),https: //openreview.net/forum?id=tTBXePRKSx

    2025

  52. [52]

    arXiv preprint arXiv:2605.23898 (2026)

    Zhang, J., Li, Y., Chen, H., Lu, H., Xue, L., Wang, B., Liu, H.: Spacenum: Revisiting spatial numerical understanding in vlms. arXiv preprint arXiv:2605.23898 (2026)

    Pith/arXiv arXiv 2026

  53. [53]

    arXiv preprint arXiv:2601.15224 (2026)

    Zhang, J., Qian, C., Sun, H., Lu, H., Wang, D., Xue, L., Liu, H.: Progresslm: Towards progress reasoning in vision-language models. arXiv preprint arXiv:2601.15224 (2026)

    Pith/arXiv arXiv 2026

  54. [54]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics

    Zhang, J., Yao, D., Pi, R., Liang, P.P., Fung, Y.R.: Vlm2-bench: A closer look at how well vlms implicitly link explicit matching visual cues. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. pp. 7510–7545 (2025)

    2025

  55. [55]

    Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=30RAWQVGlx

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., Liu, Z.: Long context transfer from language to vision. Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=30RAWQVGlx

    2025

  56. [56]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, S., Yang, J., Yin, J., Luo, Z., Luan, J.: Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22056–22065 (2025)

    2025

  57. [57]

    ACM Computing Surveys58(7), 1–36 (2026)

    Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al.: Instruction tuning for large language models: A survey. ACM Computing Surveys58(7), 1–36 (2026)

    2026

  58. [58]

    Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=EElFGvt39K

    Zhang, Y., Wu, J., Li, W., Li, B., MA, Z., Liu, Z., Li, C.: LLaVA-video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=EElFGvt39K

    2025

  59. [59]

    Advances in Neural Information Processing Systems16, 321–328 (2003)

    Zhou, D., Bousquet, O., Lal, T., Weston, J., Schölkopf, B.: Learning with local and global consistency. Advances in Neural Information Processing Systems16, 321–328 (2003)

    2003

  60. [60]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al.: Mlvu: Benchmarking multi-task long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13691–13701 (2025)

    2025

  61. [61]

    In: International Conference on Learning Representations (2024),https://openreview.net/forum? id=1tZbq88f27

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision- language understanding with advanced large language models. In: International Conference on Learning Representations (2024),https://openreview.net/forum? id=1tZbq88f27

    2024

  62. [62]

    What is the main idea of the video?

    Zuo, J., Deng, Y., Kong, L., Yang, J., Jin, R., Zhang, Y., Sang, N., Pan, L., Liu, Z., Gao, C.: Videolucy: Deep memory backtracking for long video understanding. In: Advances in Neural Information Processing Systems (2025),https://openreview. net/forum?id=To7Rs2wsTd 20 C. Zhang et al. LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Fo...

    2025

This paper was first reviewed by grok-4.5 on July 31, 2026.