Pith. sign in

REVIEW 4 major objections 5 minor 53 references

ReMem claims that long-video question answering fails not from too few frames but from frame selection that ignores the question's temporal scale, and that a training-free, granularity-adaptive selector can fix it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:13 UTC pith:XADJEL3N

load-bearing objection Promising plug-and-play keyframe selector with strong, well-ablated zero-shot gains, but Eq. (6) does the opposite of its own narrative for long videos, and the paper lacks code, variance, and clean hyperparameter independence. the 4 major comments →

arxiv 2607.24794 v1 pith:XADJEL3N submitted 2026-06-30 cs.AI cs.CV

Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding

Linghao Meng , Qiankun Li , Junyuan Mao , Pujin Liao , Zhicheng He , Enbo Zhang , Kun Wang , Yang Liu
show 2 more authors
Huazhu Fu Yueming Jin
This is my paper
classification cs.AI cs.CV
keywords training-freekeyframe selectionlong video understandingtemporal granularityvideo question answeringmemory mechanismframe routingzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ReMem claims that the main bottleneck in long-video question answering is not context length but frame selection: uniform or static query-guided sampling ignores the temporal scale a question actually requires. To fix this, ReMem uses a reasoning LLM's long-term memory to estimate a continuous temporal granularity for each question and to extract semantic entities, then builds a temporal-semantic memory graph over video frames to align frames with the query. A granularity-modulated fusion weight blends instantaneous visual similarity with temporal-structural scores, and a structure-aware router clusters frames into events and distributes the limited frame budget. On four long-video QA benchmarks and three video-LLM backbones, ReMem reports state-of-the-art zero-shot accuracy with no training, including double-digit improvements on the most demanding benchmarks.

Core claim

ReMem's central claim is that the right frames for a long-video question depend on the question's temporal granularity: a question about a single localized event needs dense frames from a narrow window, while a 'how many times over the whole video' question needs sparse coverage across the full timeline. The paper operationalizes this by asking an LLM to output a granularity score g in [0,1] and extract concrete visual entities, then uses those to guide a two-part frame scorer. One part measures static visual-semantic similarity to the query; the other propagates the query signal through a temporal-semantic memory graph built from chronologically ordered cluster anchors, capturing long-range

What carries the argument

Central object: the temporal-semantic memory graph — a K=√M-node graph whose nodes are the actual video frames closest to K-Means cluster centroids, ordered chronologically, with edges encoding semantic similarity and local temporal adjacency. A random walk with restart seeded by the query-anchored embedding diffuses activation over this graph and yields a temporal-semantic score per frame. This is combined with a static pretrained-encoder similarity via a granularity-gated weight α, and the fused scores drive structure-aware dynamic routing: frames are clustered under a semantic-plus-temporal cost, each event receives a frame quota proportional to its softmax information capacity, and the s

Load-bearing premise

The framework's success hinges on the reasoning LLM correctly estimating each question's temporal granularity and extracting useful semantic entities; a wrong estimate mis-calibrates the fusion weight α and the frame budget, and the paper does not analyze how sensitive the results are to such parsing errors.

What would settle it

Force the granularity score to a wrong value on a controlled subset of questions (e.g., set g=0.1 for questions that truly need whole-video reasoning) and measure the accuracy drop. If the drop is small, the granularity mechanism is not the driving force; alternatively, replace LLM-extracted entities with random nouns and see if accuracy holds.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Training-free, query-aware keyframe selection can match or exceed supervised selection methods, suggesting that explicit temporal-structure modeling is more sample-efficient than learning selection on curated data.
  • Granularity-adaptive fusion outperforms a fixed binary short/long split, so continuous granularity estimation is a meaningful ingredient for long-video reasoning.
  • The candidate-pool size matters: a moderate pool (around 150 frames) balances recall and noise; too large a pool degrades accuracy, giving a concrete design knob for practitioners.
  • Because the selected keyframes are backbone-agnostic, they can be computed once and reused across different video-LLMs, amortizing selection cost in multi-model deployments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the granularity estimate is systematically wrong for a class of questions, ReMem's advantage should shrink or reverse; an adversarial test that feeds mislabeled granularity scores would isolate how much of the gain comes from this estimate versus from the memory graph itself.
  • The memory-graph construction is content-agnostic and could be extended to audio or subtitle streams, where the same 'temporal-semantic memory' could route tokens rather than frames.
  • The method's sensitivity to entity-extraction quality suggests a testable extension: replacing LLM entity extraction with ground-truth entities (or random entities) would bound the value of that stage.
  • The granularity-gated fusion weight could be made video-aware by using the video's own structure, e.g., estimating granularity from detected scene-change rate, rather than relying solely on the LLM's prior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReMem, a training-free keyframe-selection framework for long-video question answering. ReMem operates in two stages: (1) query-level adaptation, where an LLM (GPT-4o) is prompted to estimate the query's temporal granularity g and extract semantic entities, and (2) video-level adaptation, where CLIP embeddings of uniformly sampled frames are scored by a static visual-semantic similarity S_v and a temporal-semantic memory-graph score S_t, combined via a granularity-modulated weight α. The selected candidate pool is then partitioned into event clusters, and a frame budget is distributed across clusters based on informativeness. The method is evaluated on LVBench, MLVU, LongVideoBench, and Video-MME with three MLLMs, reporting state-of-the-art zero-shot results, e.g., LLaVA-Video + ReMem reaches 54.5% on LVBench and 67.1% on LongVideoBench. The authors also provide ablations, sensitivity analyses for pool size N and decay λ, efficiency comparisons, and qualitative case studies.

Significance. If the empirical gains are reproducible, ReMem would be a practically useful plug-and-play selection strategy for long-video QA, particularly because it is training-free and model-agnostic. The paper's strengths include extensive evaluations across four benchmarks and three backbones, component ablations on multiple models, and an efficiency/latency analysis. The central adaptive mechanism, however, is compromised by an internal inconsistency in the fusion-weight formula (Eq. (6)): the stated dependence on video duration is opposite to what the equation actually enforces. Additionally, key hyperparameters (N, λ) are tuned on the same test benchmarks used for the headline results, which inflates apparent performance and limits the convincingness of the SOTA claims. The lack of error bars or multiple-seed reporting also makes it hard to judge the significance of the reported gaps. The core idea is promising, but the manuscript requires substantial revision before the claims can be considered reliable.

major comments (4)
  1. [§3.2, Eq. (6)] The fusion weight is defined as α = 1 − 0.5·g·exp(−λt). The text states that “for a larger g or longer video, α smoothly decays, elevating S_t.” While larger g indeed decreases α, longer duration t has the opposite effect: since λ > 0, exp(−λt) decreases with t, so α increases toward 1 as t grows. For example, with g=0.9 and λ=1e-4, α≈0.551 at t=30 s but α≈0.717 at t=3600 s. Thus the longest videos receive the least temporal weight, directly contradicting the stated design and the reported gains on Video-MME Long. This is not a notational slip: Eq. (6) is the mechanism that claims to make the method “temporal granularity-adaptive.” The authors must correct the formula (likely α = 1 − 0.5·g·(1 − exp(−λt)) or an equivalent that decays with t) and re-examine the sensitivity analysis in §4.5, whose description of λ’s effect (e.g., “premature decay forces α to 0.5”) also does not match the cu
  2. [§4.4–4.5 and Table 1] The candidate pool size N=150 and decay constant λ=1e-4 are selected by varying them and measuring accuracy on Video-MME and LongVideoBench (Table 3, Fig. 4). These are the same two benchmarks on which the headline results are reported in Table 1. This constitutes tuning on the test sets, which can substantially inflate the reported gains. No held-out validation set or nested cross-validation is described. The claim in §1 that ReMem “requires no parameter tuning” is therefore misleading. The authors should either add a validation split or evaluate the sensitivity in a way that does not cherry-pick hyperparameters on the evaluation benchmarks, and report the variability across reasonable settings.
  3. [Eqs. (2), (4), (6)] The symbol α is overloaded. In Eq. (2), α is a coefficient weighting two static-similarity terms; in Eq. (4), α balances semantic and temporal transition matrices in the random walk; and in Eq. (6), α is redefined as a function of g and t and used to fuse S_v and S_t. Similarly, β appears in Eq. (2) and again in Eq. (7) for temporal cost, without clarification of whether these are the same parameters. If any of these share the same α, the definition in Eq. (6) would overwrite the earlier ones and the system would be over-constrained; if they are distinct, the notation is ambiguous. This needs to be cleaned up for the method to be reproducible.
  4. [§4.1, Table 1] All main results are reported as single numbers without error bars or multiple runs. The selection process uses GPT-4o (stochastic) and K-Means (initialization-dependent), so repeated runs could vary. Given that some improvements over baselines are small (e.g., 69.2 vs 68.9 for N=180/200 in Table 3), the absence of variance or significance information weakens the SOTA claims. The authors should report at least three repeated runs with mean and standard deviation, or argue determinism of the pipeline if applicable.
minor comments (5)
  1. [Abstract and §4.1] The code link is given as the literal word “ReMem” rather than a URL or repository identifier. Please provide an accessible link.
  2. [§3.1 and §4.1] The granularity g is produced by GPT-4o, but no analysis is provided of how robust g and the extracted entities are to prompt phrasing or LLM parsing errors. Since α depends directly on g, a few sentences on this limitation would help.
  3. [Table 3] The arrow notation (↑/↓) in Table 3 is confusing; it appears to indicate changes relative to the previous row, but this is not stated. Please clarify in the caption or use a more conventional formatting.
  4. [§3.2, Eq. (2)] The second term in Eq. (2) lacks explicit normalization (it has a 1/√D factor but no 2-norm in the denominator), unlike the first term. The text says the formula controls scale and higher-order correlation; please define all terms precisely.
  5. [§4.2] The phrase “without any parameter tuning” in the Conclusion conflicts with the empirical choice of N and λ in §4.4–4.5. Rephrase to avoid this contradiction.

Circularity Check

1 steps flagged

Headline SOTA numbers on Video-MME and LongVideoBench are partly produced by tuning hyperparameters on those same test benchmarks; no derivation-level circularity.

specific steps
  1. fitted input called prediction [Section 4.4 (Candidate Pool Capacity Scaling) and Section 4.5 (Sensitivity Analysis of Decay Constant), Table 3 and Fig. 4, vs. Table 1 and Abstract.]
    "To determine the optimal decay constant λ, we analyze its impact on reasoning accuracy. As illustrated in Fig. 4, accuracy on both VideoMME [9] and LongVideoBench [38] exhibits a clear inverted-V trend as λ increases from 10^-5 to 10^-2. Performance peaks at λ=10^-4 (achieving 69.2% and 67.1% respectively)... Consequently, N=150 emerges as the optimal size."

    The decay constant λ and candidate-pool size N are selected by maximizing accuracy on Video-MME and LongVideoBench, and the same two benchmarks are then reported in Table 1 and the Abstract as zero-shot SOTA results (69.2% on Video-MME, 67.1% on LongVideoBench, +8.2%). The headline gains on those datasets are therefore not out-of-sample predictions; they are partly the output of optimizing hyperparameters on the test sets. This is a fitted-input-called-prediction issue, though it does not make the framework's core mechanism definitionally circular.

full rationale

ReMem is an empirically evaluated keyframe-selection system rather than a formal derivation; most of its machinery (CLIP embeddings, k-means memory graphs, random-walk diffusion, budget routing) is defined independently of the target accuracies. I found no self-definitional equation in which a claimed prediction is identical to an input by construction, and no load-bearing self-citation chain: the cited works are external Video-LLMs and sampling baselines. The meaningful circularity concern is evaluative: Section 4.4 tunes N on Video-MME/LongVideoBench by watching accuracy, and Section 4.5 tunes λ on the same two datasets, after which Table 1 and the Abstract advertise those exact datasets as zero-shot SOTA. This inflates the reported improvements and makes the LongVideoBench/Video-MME results partially selected rather than predicted. The effect is partial because the framework also shows gains on LVBench and MLVU, and because the remaining components are not fitted to the test labels. I also note a separate non-circularity issue flagged in the skeptical reading: Eq. (6), α = 1 − 0.5·g·exp(−λt), makes α increase with video duration t, contradicting the prose claim that 'for a larger g or longer video, α smoothly decays' and elevating S_t. That is an internal inconsistency/correctness bug, not a circularity, so it is not scored as a circular step here.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The method relies on domain assumptions about CLIP discriminativeness and LLM reliability, plus several hand-tuned hyperparameters. The most load-bearing choices (N, λ) are tuned directly on the evaluation benchmarks, which is a source of potential overfitting.

free parameters (7)
  • Candidate pool size N = 150
    Selected via candidate pool scaling experiments (Table 3) as the accuracy/efficiency optimum on Video-MME and LongVideoBench.
  • Decay constant λ = 1e-4
    Selected via sensitivity analysis (Fig. 4) showing inverted-V accuracy trend on Video-MME and LongVideoBench.
  • Weights α, β in static similarity (Eq. 2) = unspecified
    Coefficients in Eq. (2) controlling scale and higher-order correlation; values not reported, presumably hand-tuned.
  • Random walk parameters γ, α (Eq. 4) and number of diffusion steps = 2 steps, γ/α unspecified
    Diffusion hyperparameters; values not reported.
  • Clustering/routing hyperparameters L, β, ε, η, θ = unspecified
    Hyperparameters in Eqs. (7)-(8); not reported or justified.
  • Frame budget B = 64 for LLaVA-Video, 32 for Qwen models
    Design choice following base model conventions.
  • Initial sampling rate = 1 fps
    Uniform pre-sampling rate; empirically chosen.
axioms (5)
  • domain assumption CLIP text and visual embeddings capture sufficient semantics for frame selection.
    All alignment scores (Eqs. 1-5) rely on CLIP feature space.
  • domain assumption LLM long-term memory reliably maps questions to temporal granularity g and entities.
    Section 3.1; central to adaptive fusion and anchor enhancement.
  • domain assumption K-means on CLIP features forms coherent event clusters.
    Section 3.2 memory graph construction; Section 3.3 event clustering.
  • domain assumption Random walk on the memory graph encodes long-range temporal dependencies.
    Eq. (4)-(5); no theoretical justification that two diffusion steps suffice.
  • domain assumption Evaluation benchmarks and protocols are unbiased.
    Comparisons rely on public benchmark answer accuracy.

pith-pipeline@v1.3.0-alltime-deepseek · 13666 in / 9116 out tokens · 77642 ms · 2026-08-02T09:13:13.282335+00:00 · methodology

0 comments
read the original abstract

While Multimodal Large Language Models (MLLMs) demonstrate superior generalization in fundamental video tasks, restricted context windows limit their long video understanding. To accommodate this constraint, models typically resort to keyframe selection. However, uniform sampling or static query-guided selection often overlooks critical temporal context, failing to adapt to the varying query temporal granularities. In this paper, we propose ReMem, a temporal granularity-adaptive keyframe selection framework for training-free LongVideoQA. ReMem introduces a dual-level memory-augmented adaptation. At the query level, Memory-Driven Question Parsing leverages LLM long-term memory to decode question temporal granularity and extract semantic entities. At the video level, Synergistic Dual-Semantic Frame Alignment exploits intrinsic structural memory to align frames with query semantics, guiding Structure-Aware Dynamic Frame Routing to cluster events and optimally distribute sampling budgets. By explicitly preserving temporal information with memory mechanisms, ReMem suppresses redundancy and empowers MLLMs to perform robust multi-granular video reasoning. Evaluations across four popular LongVideoQA benchmarks using three MLLMs demonstrate highly efficient, state-of-the-art zero-shot performance; notably, LLaVA-Video with ReMem reaches 54.5% (+12.3%) on LVBench and 67.1% (+8.2%) on LongVideoBench.

Figures

Figures reproduced from arXiv: 2607.24794 by Enbo Zhang, Huazhu Fu, Junyuan Mao, Kun Wang, Linghao Meng, Pujin Liao, Qiankun Li, Yang Liu, Yueming Jin, Zhicheng He.

Figure 1
Figure 1. Figure 1: Query-adaptive methods evaluate frames in isolation and suffer from the absence of temporal understanding. Our ReMem overcomes this bottleneck with memory-augmented temporal granularity-adaptive sampling, achieving superior accu￾racy and efficiency for long video reasoning. 1 Introduction Recently, Multimodal Large Language Models (MLLMs) have achieved remark￾able breakthroughs in cross-modal reasoning by … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of ReMem. ReMem performs dual-level adaptation for training￾free LongVideoQA: ❶ query-level adaptation, where an LLM evaluates the ques￾tion’s temporal granularity and extracts entities; ❷ video-level adaptation, where memory-augmented dual-semantic evidence assists in selecting keyframes. The pipeline consists of three modules: Memory-Driven Question Parsing, Synergistic Dual-Semantic Frame Align… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation results of Qwen2-VL using ReMem. (a) compares components of our proposed framework on Qwen video model. (b) details the impact of removing TSA. modeling visual-semantic-temporal correlations mathematically is more efficient and accurate than implicitly learning them through brute-force training. Robustness Across Diverse Temporal Granularities. Our method’s su￾periority is vividly demonstrated on … view at source ↗
Figure 4
Figure 4. Figure 4: Model performance on two bench￾marks with varying decay constant [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of LLaVA-Video reasoning accuracy and latency time with dif￾ferent frame selection methods on Video-MME (left) and LongVideoBench (right). 4.6 Effect of Dynamic Time Granularity To validate the necessity of the continuous Time Granularity, we compare our dynamic weighting mechanism S = αSv+(1−α)St against rigid binary classifica￾tion (short-term α = 0.9, long-term α = 0.5). As illustrated in … view at source ↗
Figure 7
Figure 7. Figure 7: Case analysis from Video-MME. Unlike Uniform Sampling and AKS which miss crucial temporal context, our method accurately routes the relevant frames following the key event (red dot) to answer correctly. 4.8 Qualitative Analysis To intuitively demonstrate the superiority of our frame selection strategy, we present a qualitative comparison in [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Case analysis from MLVU. Variants without key components of our proposed ReMem fail. Ultimately, the complete ReMem pipeline precisely extracts frames across the timeline, depicting all four anchor events in correct chronological order. This vi￾sualization underscores how our modules work synergistically to guarantee both visual-semantic relevance and temporal structural integrity. 5 Conclusion In this wor… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 22 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    arXiv preprint arXiv:2309.16609 (2023)

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv e-prints pp. arXiv–2502 (2025)

    2025

  5. [5]

    arXiv preprint arXiv:2408.10188 (2024)

    Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., et al.: Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188 (2024)

    Pith/arXiv arXiv 2024

  6. [6]

    arXiv preprint arXiv:2406.07476 (2024)

    Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    arXiv preprint arXiv:2408.14023 (2024)

    Fei, J., Li, D., Deng, Z., Wang, Z., Liu, G., Wang, H.: Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    arXiv preprint arXiv:2503.21776 (2025)

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24108–24118 (2025)

    2025

  10. [10]

    arXiv preprint arXiv:2506.00318 (2025)

    Ghazanfari, S., Croce, F., Flammarion, N., Krishnamurthy, P., Khorrami, F., Garg, S.: Chain-of-frames: Advancing video understanding in multimodal llms via frame- aware reasoning. arXiv preprint arXiv:2506.00318 (2025)

    Pith/arXiv arXiv 2025

  11. [11]

    arXiv preprint arXiv:2503.13139 (2025)

    Guo, W., Chen, Z., Wang, S., He, J., Xu, Y., Ye, J., Sun, Y., Xiong, H.: Logic-in- frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. arXiv preprint arXiv:2503.13139 (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Hu, K., Gao, F., Nie, X., Zhou, P., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., et al.: M-llm based video frame selection for efficient video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 13702–13712 (2025)

    2025

  13. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jin, P., Takanobu, R., Zhang, W., Cao, X., Yuan, L.: Chat-univi: Unified visual rep- resentation empowers large language models with image and video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13700–13710 (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    arXiv preprint arXiv:2510.03584 (2025) Reasoning with Memory 17

    Li, C., Li, T., Tao, F., Zhao, Z., Wu, Z., Zhao, M., Song, J., Niu, C., Fazli, P.: Frameoracle: Learning what to see and how much to see in videos. arXiv preprint arXiv:2510.03584 (2025) Reasoning with Memory 17

    arXiv 2025

  16. [16]

    Science China Information Sciences 68(10), 200102 (2025)

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. Science China Information Sciences 68(10), 200102 (2025)

    2025

  17. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22195–22206 (2024)

    2024

  18. [18]

    arXiv preprint arXiv:2504.06958 (2025)

    Li,X.,Yan,Z.,Meng,D.,Dong,L.,Zeng,X.,He,Y.,Wang,Y.,Qiao,Y.,Wang,Y., Wang, L.: Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    In: European Conference on Computer Vision

    Li, Y., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large lan- guage models. In: European Conference on Computer Vision. pp. 323–340.Springer (2024)

    2024

  20. [20]

    arXiv preprint arXiv:2407.03104 (2024)

    Liang, H., Li, J., Bai, T., Huang, X., Sun, L., Wang, Z., He, C., Cui, B., Chen, C., Zhang, W.: Keyvideollm: Towards large-scale video keyframe selection. arXiv preprint arXiv:2407.03104 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    In: Proceedings of the 2024 conference on empirical methods in natural language processing

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 5971– 5984 (2024)

    2024

  22. [22]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llavanext: Improved reasoning, ocr, and world knowledge (2024)

    2024

  23. [23]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  24. [24]

    In: Proceedings of the Com- puter Vision and Pattern Recognition Conference

    Liu, S., Zhao, C., Xu, T., Ghanem, B.: Bolt: Boost large vision-language model without training for long-form video understanding. In: Proceedings of the Com- puter Vision and Pattern Recognition Conference. pp. 3318–3327 (2025)

    2025

  25. [25]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12585–12602 (2024)

    2024

  26. [26]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ren, S., Yao, L., Li, S., Sun, X., Hou, L.: Timechat: A time-sensitive multi- modal large language model for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14313– 14323 (2024)

    2024

  28. [28]

    arXiv preprint arXiv:2410.17434 (2024)

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., et al.: Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., Zhao, B.: Video-xl: Extra-long vision language model for hour-scale video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 26160–26169 (2025)

    2025

  30. [30]

    arXiv preprint arXiv:2602.03615 (2026) 18 L

    Song, B., Peng, J., Zhang, Y., Chen, G., Yang, F., Guo, J.: Ktv: Keyframes and key tokens selection for efficient training-free video llms. arXiv preprint arXiv:2602.03615 (2026) 18 L. Meng, Q. Li et al

    arXiv 2026

  31. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al.: Moviechat: From dense token to sparse memory for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18221–18232 (2024)

    2024

  32. [32]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Sun, H., Lu, S., Wang, H., Chen, Q.G., Xu, Z., Luo, W., Zhang, K., Li, M.: Mdp3: A training-free approach for list-wise frame selection in video-llms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 24090–24101 (2025)

    2025

  33. [33]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Tang, X., Qiu, J., Xie, L., Tian, Y., Jiao, J., Ye, Q.: Adaptive keyframe sampling for long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29118–29128 (2025)

    2025

  34. [34]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  35. [35]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang,W.,He,Z.,Hong,W.,Cheng,Y.,Zhang,X.,Qi,J.,Ding,M.,Gu,X.,Huang, S., Xu, B., et al.: Lvbench: An extreme long video understanding benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22958–22967 (2025)

    2025

  36. [36]

    arXiv preprint arXiv:2307.06942 (2023)

    Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al.: Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 (2023)

    Pith/arXiv arXiv 2023

  37. [37]

    In: European conference on computer vision

    Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al.: Internvideo2: Scaling foundation models for multimodal video understanding. In: European conference on computer vision. pp. 396–416. Springer (2024)

    2024

  38. [38]

    Advances in Neural Information Pro- cessing Systems37, 28828–28857 (2024)

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Pro- cessing Systems37, 28828–28857 (2024)

    2024

  39. [39]

    In: Findings of the Association for Computational Linguistics: ACL 2025

    Yao, L., Wu, H., Ouyang, K., Zhang, Y., Xiong, C., Chen, B., Sun, X., Li, J.: Gen- erative frame sampler for long video understanding. In: Findings of the Association for Computational Linguistics: ACL 2025. pp. 17900–17917 (2025)

    2025

  40. [40]

    Advances in Neural Information Processing Systems36, 76749–76771 (2023)

    Yu, S., Cho, J., Yadav, P., Bansal, M.: Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems36, 76749–76771 (2023)

    2023

  41. [41]

    arXiv preprint arXiv:2410.03226 (2024)

    Yu, S., Jin, C., Wang, H., Chen, Z., Jin, S., Zuo, Z., Xu, X., Sun, Z., Zhang, B., Wu, J., et al.: Frame-voyager: Learning to query frames for video large language models. arXiv preprint arXiv:2410.03226 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    arXiv preprint arXiv:2501.13106 (2025)

    Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X., et al.: Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 (2025)

    Pith/arXiv arXiv 2025

  43. [43]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Zhang, C., Lu, T., Islam, M.M., Wang, Z., Yu, S., Bansal, M., Bertasius, G.: A simple llm framework for long-range video question-answering. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 21715–21737 (2024)

    2024

  44. [44]

    IEEE Transactions on Infor- mation Forensics and Security (2025)

    Zhang, J., Wan, Z., Hu, L., Lin, S., Wu, S., Shan, S.: Collaboratively self-supervised video representation learning for action recognition. IEEE Transactions on Infor- mation Forensics and Security (2025)

    2025

  45. [45]

    IEEE Transactions on Circuits and Systems for Video Technology (2026) Reasoning with Memory 19

    Zhang, J., Wan, Z., Nie, S., Li, C., Shan, S., Chen, X.: Bimm: Brain inspired masked modeling for video representation learning. IEEE Transactions on Circuits and Systems for Video Technology (2026) Reasoning with Memory 19

    2026

  46. [46]

    arXiv preprint arXiv:2406.16852 (2024)

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., Liu, Z.: Long context transfer from language to vision. arXiv preprint arXiv:2406.16852 (2024)

    Pith/arXiv arXiv 2024

  47. [47]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, S., Yang, J., Yin, J., Luo, Z., Luan, J.: Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22056–22065 (2025)

    2025

  48. [48]

    arXiv preprint arXiv:2410.02713 (2024)

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024)

    Pith/arXiv arXiv 2024

  49. [49]

    arXiv preprint arXiv:2506.00993 (2025)

    Zhang, Y., Lu, Y., Wang, T., Rao, F., Yang, Y., Zhu, L.: Flexselect: Flexible token selection for efficient long video understanding. arXiv preprint arXiv:2506.00993 (2025)

    Pith/arXiv arXiv 2025

  50. [50]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhao,M.,Yu,Y.,Wang,X.,Yang,L.,Niu,D.:Search-map-search:aframeselection paradigm for action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10627–10636 (2023)

    2023

  51. [51]

    arXiv preprint arXiv:2406.042642(5), 6 (2024)

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang, X., Xiong, Y., Zhang, B., Huang, T., Liu, Z.: Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.042642(5), 6 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    arXiv preprint arXiv:2304.10592 (2023)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)

    Pith/arXiv arXiv 2023

  53. [53]

    arXiv preprint arXiv:2510.27280 (2025)

    Zhu, Z., Xu, H., Luo, Y., Liu, Y., Sarkar, K., Yang, Z., You, Y.: Focus: Efficient keyframe selection for long video understanding. arXiv preprint arXiv:2510.27280 (2025)

    arXiv 2025