Pith. sign in

REVIEW 4 major objections 5 minor 66 references

A training-free frame selector, FORGE, claims to improve long-video question answering by up to 8.7 accuracy points over uniform sampling.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:41 UTC pith:ONNTJFP3

load-bearing objection FORGE is a plausible, well-scoped frame selector with strikingly consistent wins, but it never runs the one baseline that would prove its central claim: selecting the K highest-relevance-weight frames. the 4 major comments →

arxiv 2607.25266 v2 pith:ONNTJFP3 submitted 2026-07-28 cs.CV cs.LG

FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

Ghazal Kaviani , Ghassan AlRegib This is my paper
classification cs.CV cs.LG
keywords frame selectionlong-form video understandingvideo question answeringmultimodal LLMsquery-conditioned geometrytraining-freekeyframe selectionvolume maximization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FORGE is an inference-time, training-free method for choosing which frames of a long video to feed a multimodal LLM when answering a question. The paper argues that existing selectors suffer from 'scalar collapse': they reduce each frame's query-relevance to a single number, losing which dimensions of the query the frame actually covers. FORGE instead builds a query-conditioned geometry by weighting frame embeddings with a scale-adaptive relevance signal, then selects the K frames spanning the largest volume in that space, unifying relevance and diversity into one objective. On Video-MME and LongVideoBench, it reports consistent gains over five baselines and uniform sampling across eight open-source MLLMs, with up to 8.7-point accuracy gains. The authors claim this shows that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.

Core claim

The central claim is that a single volume-maximization objective in a query-warped embedding space can jointly capture relevance and diversity, because frames that are individually relevant but redundant span little volume, and diverse but irrelevant frames are collapsed near the origin. FORGE constructs this space by scaling each frame embedding by a weight derived from a scale-adaptive blend of global relevance score and local temporal contrast, projecting onto the effective-rank subspace, and greedily selecting frames that maximize the log-volume. The paper reports that this outperforms five training-free baselines and uniform sampling on two long-video benchmarks across eight MLLMs, and

What carries the argument

The central object is the query-conditioned geometry: a weighted embedding space in which each frame's vector is scaled by a relevance weight (derived from a scale-adaptive blend of global score and local temporal contrast) and then projected onto the subspace of its effective rank. The selection mechanism is greedy volume maximization: repeatedly pick the frame with the largest residual norm after orthogonalizing against already-chosen frames, which greedily maximizes the log-volume of the selected subset. Because the log-volume objective is submodular and monotone, greedy selection carries a (1-1/e) approximation guarantee.

Load-bearing premise

The load-bearing premise is that the per-frame relevance scores from a pretrained multimodal encoder correctly separate answer-bearing frames from distractors; if those scores are weak or misaligned, the query-conditioned geometry has no signal to amplify and FORGE's improvements would collapse.

What would settle it

A concrete falsifier: run FORGE with a deliberately corrupted relevance score (e.g., a random vector or an image-only embedding) on Video-MME at K=16. If accuracy still exceeds uniform sampling, the geometry itself carries the information; if it drops to or below uniform, the relevance signal is the real driver. Also, a sweep of the lambda constant that controls the global/local blend would test whether the adaptive scale selection is essential.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, FORGE can replace uniform frame sampling as a drop-in inference-time improvement for long-video question answering with any multimodal LLM.
  • The result suggests that preserving the query's high-dimensional structure in the selection geometry matters more than using scalar relevance scores.
  • FORGE is complementary to token-compression and pruning methods, so it could be combined with them for further gains.
  • It offers a cheap alternative to agentic pipelines that repeatedly call an LLM, at a fraction of the compute.
  • The consistent gains across model sizes and architectures imply that the benefit comes from the selected frames themselves, not from a particular backbone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method's gains likely hinge on the quality of the pretrained relevance scores; a systematic study varying the matching model would clarify how much of the improvement is due to the geometry versus the scorer.
  • The local-contrast term is effectively a temporal saliency prior; testing different window sizes or replacing it with other temporal cues could isolate its contribution.
  • The volume-maximization view could transfer to other subset-selection problems where both relevance and diversity matter, such as selecting frames for video captioning or evidence selection in retrieval-augmented generation.
  • One testable extension: applying FORGE to shorter videos or different frame pools (e.g., higher FPS) to see if the effective-rank projection is robust to pool density.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FORGE, a training-free and model-agnostic frame-selection method for long-form video understanding. Given a pretrained multimodal encoder that produces per-frame query-relevance scores, FORGE forms a query-conditioned embedding geometry by scaling frame embeddings with sigmoid-transformed relevance weights, projecting to an effective rank estimated from spectral entropy, and greedily selecting K frames that maximize the spanned volume via iterative orthogonalization. The authors claim that this unifies relevance and diversity into a single objective, and they report consistent gains in keyframe recall, scene hit rate, a unified selection score, and downstream VQA accuracy over several training-free baselines on Video-MME and LongVideoBench across eight MLLMs and budgets of 16, 32, and 64 frames.

Significance. If the reported results hold, FORGE would be a practically useful and low-cost improvement to inference-time frame selection for long-video MLLMs. The paper's strengths include a broad evaluation across multiple MLLM families and sizes, deterministic and reproducible-in-principle experiments, and a clear efficiency argument relative to agentic pipelines. The main empirical claims are supported by the tables, which show FORGE leading in every reported cell for the primary selection-quality and accuracy comparisons. However, the paper's central conceptual contribution — that the query-conditioned geometry, rather than the relevance weighting, drives the gains — is not cleanly isolated by the experiments as presented.

major comments (4)
  1. [§4, Tables 1–2; §4.4, Table 4] The key control is missing: selecting the top-K frames by the relevance score w_t (or s_t) alone is not reported in Tables 1–3. Table 4 includes 'Top-k (reference)' only for the property analysis, not for KFR, SHR, UKSS, or VQA accuracy. Since the query-conditioning in Eq. (4)–(5) enters only through scalar weights, a direct top-K-by-weight baseline is needed to attribute the gains to the orthogonalization/volume step. If top-K by w_t matches FORGE's numbers, the geometry contributes little; if it is much worse, the geometry matters. This baseline must be added to the main tables before the central claim can be assessed.
  2. [§3, Eqs. (3)–(4); §4, Implementation] The implementation constants c, λmin, λmax, W, τ, and ε are stated to be fixed but are not reported, and the code is not yet available. Similarly, the relevance model that produces s_t is only described as 'a pretrained multimodal encoder' without naming the exact model (e.g., CLIP or BLIP). These omissions make the experiments irreproducible and prevent readers from evaluating sensitivity of the method to the relevance signal. Please specify the encoder, list the constants, and release the implementation or a reproducible description.
  3. [§4.3, Fig. 5] The ablation removes the local-contrast term (λ=0) and removes the query warp (w_t=1), but it never removes the greedy orthogonalization/volume-selection step while keeping the relevance weighting. Therefore it does not isolate the contribution of the geometry per se. A variant that selects top-K by w_t, or a variant that uses the same weighted, projected space but selects without volume maximization, is necessary to show that the unification of relevance and diversity is what drives the improvement. The MaxInfo baseline in Table 1 is related but uses a different (unweighted) embedding, so it does not serve as this control.
  4. [§3, Eqs. (5)–(7); §1, Contributions] The claim that FORGE 'preserves the query's high-dimensional structure' is not directly supported by the formalism. The query text is never embedded into the geometry; the only query-dependent quantity is the scalar score s_t, which is transformed monotonically into w_t and applied as a diagonal scaling diag(w)·E. The SVD projection in Eq. (6) is computed on the weighted frame embeddings, not on any query vector. Thus the geometry depends on the query only through scalar relevance scores, and the paper should either temper the 'high-dimensional' claim or provide an explicit analysis of how the weighting plus spectral projection captures directional query structure.
minor comments (5)
  1. [§4.4, Table 4] The phrase 'infdicate' should be 'indicate'.
  2. [§4, Setup] In the sentence 'Downstream video understating is assessed', 'understating' should be 'understanding'.
  3. [§4, Tables 1–3] There is no report of variance or confidence intervals. The paper states all numbers were confirmed over three independent runs, but because the method and decoding are deterministic, it would be useful to state explicitly whether the reported values are exact or means over the three runs.
  4. [Fig. 5] The legend text in the figure appears mangled ('Forge Forge ( =0)'); the notation should be cleaned to match the text's FORGE (λ=0) and FORGE (no query warp).
  5. [§4, Table 3] For the LongV A and SeViLA rows, the comparison is between FORGE at 32 frames and the original method at a different frame count (e.g., 128 frames). This is informative but should be clearly labeled as a cross-budget comparison to avoid confusion.

Circularity Check

0 steps flagged

No load-bearing circularity: FORGE's selection objective is a heuristic surrogate externally validated against ground-truth keyframes and MLLM accuracy, not an identity with its inputs.

full rationale

The derivation chain in Section 3 starts from the exact identity I(V_S;A|Q)=H(A|Q)-H(A|V_S,Q) (Eq. 2), then explicitly replaces the intractable conditional entropy with a tractable surrogate: "Answer uncertainty is taken to decrease as the selected frames carry more independent query-relevant content" and "Modeling the query-conditioned embeddings as Gaussian, that entropy is an affine function of the log-determinant of the subset's Gram matrix." This is a stated modeling assumption, not a hidden redefinition: the resulting log-volume objective is not identical to the relevance scores that feed it. The weights w_t (Eq. 4) are a monotone transform of the input score s_t, but the greedy selection (Algorithm 1, Eq. 7) operates on residual norms in an SVD-projected, weighted embedding space, so the output is not top-K-by-s by construction for K>1; it is a maximum-volume-style objective. The claims are then evaluated against externally defined ground-truth keyframes (KFR), scene annotations (SHR/UKSS), and downstream VQA accuracy on fixed benchmarks, so the central predictions are not forced by the construction. No load-bearing self-citations appear: neither author's prior work is cited for any uniqueness or derivation step. The closest concern is that Tables 1-3 omit a pure Top-K-by-relevance accuracy baseline (only "Top-k (reference)" appears in Table 4 for selection properties), so the specific contribution of the orthogonalization step over the relevance ranking is not fully isolated experimentally; however, that is an experimental completeness issue, not a circularity, and the ablation in Fig. 5 at least removes the query warp and local-contrast terms. The paper's own Limitations section identifies residual gaps as coming from MLLM reasoning rather than from the selector, which is consistent with an externally validated method. Therefore no step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

FORGE's central claim rests on the unstated quality of the relevance-scoring encoder, the unvalidated Gaussian/entropy surrogate for conditional mutual information, and a paper-specific choice of projection dimension. Six named implementation constants (c, λmin, λmax, W, τ, ε) are free parameters whose values are deferred to future code, so their role in the reported numbers cannot be audited. No new physical entities are introduced.

free parameters (3)
  • λ blend constants (c, λmin, λmax) = not stated; 'fixed implementation constants'
    Eq. (3) defines λ = clip(c/(std(s)+ε), λmin, λmax); the constants are not given in the paper and are argued to be shared across datasets, but their values are deferred to the public code. The choice of λ controls how much the local-contrast term matters compared to the global relevance score.
  • temporal window half-width W = not stated
    Local contrast for frame t is s_t − mean(s_{t−W:t+W}); W is a free scale constant fixed across datasets but its value is not given.
  • sigmoid temperature τ and ε = not stated
    Eq. (4) passes the normalized blend through a mean-centered sigmoid with temperature and epsilon; both are implementation constants with unspecified values.
axioms (4)
  • domain assumption Relevance scores s_t from a pretrained multimodal encoder already contain sufficient query-answer signal, and can be computed honestly at inference time.
    Section 3 states each frame receives a relevance score through a pretrained multimodal matching model; the correctness of the entire geometry depends on the quality of these scores, but the encoder is never identified.
  • domain assumption The Gaussian/entropy approximation: answer uncertainty decreases with independent query-relevant content, and the Gaussian entropy of the selected features is an affine function of log-det of the Gram matrix.
    Section 3's derivation from Eq. (1) to the volume objective requires modeling query-conditioned embeddings as Gaussian and equating feature entropy with answer entropy; this is stated without empirical support.
  • standard math Greedy orthogonalized selection maximizes the log-volume/submodular objective within the (1−1/e) guarantee.
    Relies on Nemhauser et al. [59] and DPP references [56]; this is standard and not a source of error, assuming the log-volume objective is monotone submodular in the projected space.
  • ad hoc to paper Effective rank R from spectral entropy of the weighted embedding matrix is a good model capacity/captures query structure.
    Section 3 sets R = round(exp(H_σ)) from the singular values of the weighted matrix; this particular choice is not derived from any prior theory and is a paper-specific modeling decision.
invented entities (1)
  • Query-conditioned warped embedding geometry (Ẽ = diag(w)·E with spectral projection) no independent evidence
    purpose: A new latent space where query-relevant frames are far apart and irrelevant frames collapse, enabling single-objective relevance+diversity selection.
    This is a latent construction rather than a physical entity; its only evidence is the downstream benchmark gains in the same paper. It is a modeling artifact, not a falsifiable new entity with external handles.

pith-pipeline@v1.3.0-alltime-deepseek · 17450 in / 8932 out tokens · 73819 ms · 2026-08-03T01:41:34.892948+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible. However, the density of relevant content decreases sharply as video sequence length increases, and exposing the model to more irrelevant content measurably reduces its accuracy. In this paper, we address the problem of maximizing query-relevant information in a frame subset selected at inference time, without training. FORGE (Frame Orthogonality in Relevance Geometry) is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space, unifying relevance and diversity into a single objective. In this space, frames that cover independent query-relevant directions are far apart, and selecting the subset of maximum information captures diverse query-relevant content within the budget. Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified keyframe selection score by 11.0-15.3 points over the strongest training-free baseline and up to doubles keyframe recall (0.415 vs. 0.204 at K=64 on Video-MME). The gains extend to question answering, where accuracy improves in every evaluated setting across eight open-source MLLMs spanning 4B to 32B parameters, by up to 8.7 points over uniform sampling and 5.2 points over the strongest baseline. Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 12 linked inside Pith

  1. [1]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learn- ing united visual representation by alignment before projection. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5971–5984 (2024)

    2024

  2. [2]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 12585–12602 (2024)

    2024

  3. [3]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Science China Infor- mation Sciences67(12), 220101 (2024)

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z.,et al.: How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open-source suites. Science China Infor- mation Sciences67(12), 220101 (2024)

    2024

  5. [5]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M.,et al.: Video-mme: The first- ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108– 24118 (2025)

    2025

  6. [6]

    Advances in Neural Informa- tion Processing Systems37, 28828–28857 (2024)

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Informa- tion Processing Systems37, 28828–28857 (2024)

    2024

  7. [7]

    In: Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pp

    Li, Z., Ishida, K., Yamazaki, S., Ji, X., Liu, J.: Kfs-bench: Comprehensive evaluation of key frame sampling in long video understand- ing. In: Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pp. 5643–5652 (2026)

    2026

  8. [8]

    In: Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., Zhao, B.: Video-xl: Extra-long vision language model for hour- scale video understanding. In: Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp. 26160–26169 (2025)

    2025

  9. [9]

    arXiv preprint arXiv:2409.01071 (2024)

    Wang, Y., Song, Y., Xie, C., Liu, Y., Zheng, Z.: Videollamb: Long streaming video under- standing with recurrent memory bridges. arXiv preprint arXiv:2409.01071 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp

    Jiang, J., Li, X., Liu, Z., Li, M., Chen, G., Li, Z., Huang, D.-A., Liu, G., Yu, Z., Keutzer, K.,et al.: Storm: Token-efficient long video understanding for multimodal llms. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 5830–5841 (2025)

    2025

  11. [11]

    Transactions of the Asso- ciation for Computational Linguistics12, 157–173 (2024) 14

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Transactions of the Asso- ciation for Computational Linguistics12, 157–173 (2024) 14

    2024

  12. [12]

    In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V

    Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S.B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E.A., Peng, H.: Context length alone hurts LLM per- formance despite perfect retrieval. In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V. (eds.) Findings of the Association for Computational Lin- guistics: EMNLP 2025, pp. 23281–23298. A...

    doi:10.18653/v1/2025.findings-emnlp.1264 2025

  13. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19792–19802 (2025)

    2025

  14. [14]

    In: The Thirty- ninth Annual Conference on Neural Information Processing Systems (2025)

    Shen, L., Gong, G., He, T., Zhang, Y., liu, Zhao, S., Ding, G.: FastVID: Dynamic density pruning for fast video large language models. In: The Thirty- ninth Annual Conference on Neural Information Processing Systems (2025). https://openreview.net/forum?id=2xS4VtpApy

    2025

  15. [15]

    In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T

    Wang, X., Si, Q., Zhu, S., Wu, J., Cao, L., Nie, L.: AdaReTaKe: Adaptive redun- dancy reduction to perceive longer for video-language understanding. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Findings of the Association for Computational Linguistics: ACL 2025, pp. 5417–5432. Association for Computational Linguistics, Vienna, Austria (202...

    doi:10.18653/v1/2025.findings-acl.283 2025

  16. [16]

    In: European Conference on Computer Vision, pp

    Wang, X., Zhang, Y., Zohar, O., Yeung-Levy, S.: Videoagent: Long-form video understand- ing with large language model as agent. In: European Conference on Computer Vision, pp. 58–76 (2024). Springer

    2024

  17. [17]

    In: The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems (2025)

    Zhang, X., Jia, Z., Guo, Z., Li, J., Li, B., Li, H., Lu, Y.: Deep video discov- ery: Agentic search with tool use for long-form video understanding. In: The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems (2025). https://openreview.net/forum?id=oQYq9L1NVT

    2025

  18. [18]

    arXiv preprint arXiv:2510.04428 (2025)

    Zou, Y., Jin, S., Deng, A., Zhao, Y., Wang, J., Chen, C.: Air: Enabling adaptive, iter- ative, and reasoning-based frame selection for video question answering. arXiv preprint arXiv:2510.04428 (2025)

    arXiv 2025

  19. [19]

    https://arxiv

    Wang, Z., Zhou, H., Wang, S., Li, J., Xiong, C., Savarese, S., Bansal, M., Ryoo, M.S., Niebles, J.C.: Active Video Perception: Iter- ative Evidence Seeking for Agentic Long Video Understanding (2025). https://arxiv. org/abs/2512.05774

    Pith/arXiv arXiv 2025

  20. [20]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Tang, X., Qiu, J., Xie, L., Tian, Y., Jiao, J., Ye, Q.: Adaptive keyframe sampling for long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29118–29128 (2025)

    2025

  21. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Sun, H., Lu, S., Wang, H., Chen, Q.-G., Xu, Z., Luo, W., Zhang, K., Li, M.: Mdp3: A training-free approach for list-wise frame selection in video-llms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 24090–24101 (2025)

    2025

  22. [22]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Liu, S., Zhao, C., Xu, T., Ghanem, B.: Bolt: Boost large vision-language model without training for long-form video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3318– 3327 (2025)

    2025

  23. [23]

    arXiv preprint arXiv:2502.03183 (2025)

    Li, P., Abdullaeva, I., Gambashidze, A., Kuznetsov, A., Oseledets, I.: Maxinfo: A training-free key-frame selection method using maximum volume for enhanced video understanding. arXiv preprint arXiv:2502.03183 (2025)

    arXiv 2025

  24. [24]

    15 In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp

    Zhang, S., Yang, J., Yin, J., Luo, Z., Luan, J.: Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms. 15 In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 22056–22065 (2025)

    2025

  25. [25]

    European Conference on Computer Vision (ECCV) (2024)

    Weng, Y., Han, M., He, H., Chang, X., Zhuang, B.: Longvlm: Efficient long video understanding via large language models. European Conference on Computer Vision (ECCV) (2024)

    2024

  26. [26]

    Advances in Neural Information Processing Systems36, 76749–76771 (2023)

    Yu, S., Cho, J., Yadav, P., Bansal, M.: Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems36, 76749–76771 (2023)

    2023

  27. [27]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, ...

    Pith/arXiv arXiv 2025

  28. [28]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Yang, S., Xu, R., Cui, C., Wang, T., Lin, D., Pang, J.: Vflowopt: A token pruning framework for lmms with visual information flow-guided optimization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 23924–23934 (2025)

    2025

  29. [29]

    In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T

    Huang, X., Zhou, H., Han, K.: PruneVid: Visual token pruning for efficient video large language models. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Find- ings of the Association for Computational Linguistics: ACL 2025, pp. 19959–19973. Association for Computational Linguis- tics, Vienna, Austria (2025). https://doi. org/10.18653/v1/2025.f...

    doi:10.18653/v1/2025.findings-acl.1024 2025

  30. [30]

    In: Forty-second International Conference on Machine Learning (2025)

    Zhang, Y., Fan, C.-K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D.A., Okuno, T., Nakata, Y., Keutzer, K., Zhang, S.: SparseVLM: Visual token sparsifica- tion for efficient vision-language model inference. In: Forty-second International Conference on Machine Learning (2025). https://openreview.net/forum?id=80faIPZ67S

    2025

  31. [31]

    In: The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems (2025)

    Tong, J., Jin, W., Qin, P., Li, A., Zou, Y., Li, Y., Li, Y., Li, R.: Flowcut: Rethink- ing redundancy via information flow for efficient vision-language models. In: The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems (2025). https://openreview.net/forum?id=M6zQNbCaLl

    2025

  32. [32]

    Advances in neural information process- ing systems34, 13937–13949 (2021)

    Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.-J.: Dynamicvit: Efficient vision transformers with dynamic token sparsifica- tion. Advances in neural information process- ing systems34, 13937–13949 (2021)

    2021

  33. [33]

    In: European Conference on Computer Vision, pp

    Wang, J., Yang, X., Li, H., Liu, L., Wu, Z., Jiang, Y.-G.: Efficient video transform- ers with spatial-temporal token selection. In: European Conference on Computer Vision, pp. 69–86 (2022). Springer

    2022

  34. [34]

    In: European Conference on Com- puter Vision, pp

    Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., Li, Q.: Videoagent: A memory- augmented multimodal agent for video under- standing. In: European Conference on Com- puter Vision, pp. 75–92 (2024). Springer

    2024

  35. [35]

    arXiv preprint arXiv:2410.03226 (2024)

    Yu, S., Jin, C., Wang, H., Chen, Z., Jin, S., Zuo, Z., Xu, X., Sun, Z., Zhang, B., Wu, J., et al.: Frame-voyager: Learning to query frames for video large language models. arXiv preprint arXiv:2410.03226 (2024)

    Pith/arXiv arXiv 2024

  36. [36]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Ye, J., Wang, Z., Sun, H., Chandrasegaran, K., Durante, Z., Eyzaguirre, C., Bisk, Y., Niebles, J.C., Adeli, E., Fei-Fei, L.,et al.: Re-thinking temporal search for long-form video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8579–8591 (2025)

    2025

  37. [37]

    In: Proceedings of the Computer Vision and Pat- tern Recognition Conference, pp

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., Bansal, M.: 16 Videotree: Adaptive tree-based video repre- sentation for llm reasoning on long videos. In: Proceedings of the Computer Vision and Pat- tern Recognition Conference, pp. 3272–3283 (2025)

    2025

  38. [38]

    https://arxiv.org/abs/1811.12432

    Wu, Z., Xiong, C., Ma, C.-Y., Socher, R., Davis, L.S.: AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2019). https://arxiv.org/abs/1811.12432

    Pith/arXiv arXiv 2019

  39. [39]

    https://arxiv.org/ abs/1907.13369

    Wu, W., He, D., Tan, X., Chen, S., Wen, S.: Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019). https://arxiv.org/ abs/1907.13369

    Pith/arXiv arXiv 2019

  40. [40]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., Liu, J.: Less is more: Clipbert for video-and-language learning via sparse sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7331–7341 (2021)

    2021

  41. [41]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Buch, S., Nagrani, A., Arnab, A., Schmid, C.: Flexible frame selection for efficient video reasoning. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29071–29082 (2025)

    2025

  42. [42]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp

    Hu, K., Gao, F., Nie, X., Zhou, P., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., Chilimbi, T.: M-llm based video frame selection for efficient video under- standing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 13702–13712 (2025)

    2025

  43. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Son, J., Park, J., Kim, K.: Csta: Cnn-based spatiotemporal attention for video summa- rization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18847–18856 (2024)

    2024

  44. [44]

    IEEE Transactions on Image Processing32, 3013–3026 (2023) https://doi

    Hsu, T.-C., Liao, Y.-S., Huang, C.-R.: Video summarization with spatiotemporal vision transformer. IEEE Transactions on Image Processing32, 3013–3026 (2023) https://doi. org/10.1109/TIP.2023.3275069

    arXiv 2023

  45. [45]

    In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T

    Potapov, D., Douze, M., Harchaoui, Z., Schmid, C.: Category-specific video summa- rization. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014, pp. 540–555. Springer, Cham (2014)

    2014

  46. [46]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Gygli, M., Grabner, H., Van Gool, L.: Video summarization by learning submodular mix- tures of objectives. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3090–3098 (2015)

    2015

  47. [47]

    Advances in neural information processing systems27 (2014)

    Gong, B., Chao, W.-L., Grauman, K., Sha, F.: Diverse sequential subset selection for supervised video summarization. Advances in neural information processing systems27 (2014)

    2014

  48. [48]

    In: Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp

    Sharghi, A., Laurel, J.S., Gong, B.: Query- focused video summarization: Dataset, eval- uation, and a memory network based approach. In: Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 4788–4797 (2017)

    2017

  49. [49]

    In: Proceedings of the 2020 International Con- ference on Multimedia Retrieval, pp

    Huang, J.-H., Worring, M.: Query- controllable video summarization. In: Proceedings of the 2020 International Con- ference on Multimedia Retrieval, pp. 242–250 (2020)

    2020

  50. [50]

    In: 2023 IEEE International Conference on Image Process- ing (ICIP), pp

    Huang, J.-H., Murn, L., Mrak, M., Wor- ring, M.: Query-based video summarization with pseudo label supervision. In: 2023 IEEE International Conference on Image Process- ing (ICIP), pp. 1430–1434 (2023). IEEE

    2023

  51. [51]

    arXiv preprint arXiv:2602.08861 (2026)

    Zheng, X., Wang, Z., Peng, Y.: Tifre: Text- guided video frame reduction for efficient video multi-modal large language models. arXiv preprint arXiv:2602.08861 (2026)

    arXiv 2026

  52. [52]

    https://arxiv.org/abs/2512.04000

    Li, J., Li, B., Li, J., Lu, Y.: Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding (2025). https://arxiv.org/abs/2512.04000

    arXiv 2025

  53. [53]

    arXiv preprint arXiv:2601.11359 (2026) 17

    Tan, W., Song, R., Li, J., Ju, J., Luo, Z.: Think-clip-sample: Slow-fast frame selec- tion for video understanding. arXiv preprint arXiv:2601.11359 (2026) 17

    arXiv 2026

  54. [54]

    arXiv preprint arXiv:2602.03615 (2026)

    Song, B., Peng, J., Zhang, Y., Chen, G., Yang, F., Guo, J.: Ktv: Keyframes and key tokens selection for efficient training-free video llms. arXiv preprint arXiv:2602.03615 (2026)

    arXiv 2026

  55. [55]

    https://openreview.net/forum? id=L0dORRnUhu

    Sheng, Y., Hao, Y., Li, C., Wang, S., He, X.: SeViCES: Unifying Semantic-Visual Evi- dence Consensus for Long Video Understand- ing (2026). https://openreview.net/forum? id=L0dORRnUhu

    2026

  56. [56]

    Foun- dations and Trends®in Machine Learning 5(2–3), 123–286 (2012)

    Kulesza, A., Taskar, B.,et al.: Determinantal point processes for machine learning. Foun- dations and Trends®in Machine Learning 5(2–3), 123–286 (2012)

    2012

  57. [57]

    In: Matrix Methods: Theory, Algorithms and Applications, pp

    Goreinov, S.A., Oseledets, I.V., Savostyanov, D.V., Tyrtyshnikov, E.E., Zamarashkin, N.L.: How to find a good submatrix. In: Matrix Methods: Theory, Algorithms and Applications, pp. 247–256. World Scientific, ??? (2010)

    2010

  58. [58]

    Linear Algebra and its Applica- tions538, 187–211 (2018)

    Mikhalev, A., Oseledets, I.V.: Rectangu- lar maximum-volume submatrices and their applications. Linear Algebra and its Applica- tions538, 187–211 (2018)

    2018

  59. [59]

    Mathemat- ical programming14(1), 265–294 (1978)

    Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximiz- ing submodular set functions—i. Mathemat- ical programming14(1), 265–294 (1978)

    1978

  60. [60]

    Wiley-Interscience, ??? (2006)

    Cover, T.M., Thomas, J.A.: Ele- ments of Information Theory, 2nd edn. Wiley-Interscience, ??? (2006)

    2006

  61. [61]

    International Journal of Computer Vision30(2), 79–116 (1998)

    Lindeberg, T.: Feature detection with auto- matic scale selection. International Journal of Computer Vision30(2), 79–116 (1998)

    1998

  62. [62]

    arXiv preprint arXiv:2503.19786 (2025)

    Gemma Team: Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  63. [63]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  64. [64]

    arXiv preprint arXiv:2406.16852 (2024)

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., Liu, Z.: Long context transfer from language to vision. arXiv preprint arXiv:2406.16852 (2024)

    Pith/arXiv arXiv 2024

  65. [65]

    arXiv preprint arXiv:2501.12386 (2025)

    Wang, Y., Li, X., Yan, Z., He, Y., Yu, J., Zeng, X., Wang, C., Ma, C., Huang, H., Gao, J., Dou, M., Chen, K., Wang, W., Qiao, Y., Wang, Y., Wang, L.: Internvideo2.5: Empow- ering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386 (2025)

    Pith/arXiv arXiv 2025

  66. [66]

    arXiv preprint arXiv:2412.05271 (2024) 18

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024) 18

    Pith/arXiv arXiv 2024