Pith. sign in

REVIEW 4 major objections 5 minor 47 references

SEAL: Semantic Attention Learning for Long Video Representation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SEAL claims that a compact set of semantic tokens—scenes, objects, and actions—selected for query relevance and diversity, gives a long-video representation that beats far larger models on QA and temporal grounding.

desk verdict Useful integration of semantic tokens and query-aware selection, but the LVBench headline may be inflated by answer-option leakage in the selection step. read the letter →

arxiv 2412.01798 v3 pith:A6TGWMIN submitted 2024-12-02 cs.CV

Lan Wang , Yujia Chen , Du Tran , Vishnu Naresh Boddeti , Wen-Sheng Chu This is my paper · ORCID
classification cs.CV
keywords longvideounderstandingsemanticdecompositionattentionlearningsubsetselectionquestionansweringtemporalgroundingsceneobjectactiontokensstreamingrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEAL claims that long videos can be reduced to a compact set of semantic tokens—scenes, objects, and actions—and that an attention-learning step that selects a diverse, query-relevant subset of these tokens yields a unified representation strong enough to beat far larger models. On LVBench (hour-long videos), SEAL with a 34B LLM reaches 45.9% overall accuracy, ahead of Qwen2-VL-72B (41.3%) and InternVL2-34B (39.6%). The same representation, with different prediction heads, improves video QA on MovieChat-1K and temporal grounding on Ego4D-NLQ. If this holds, it would show that careful semantic-structured sampling can substitute for raw computation and model scale in long-video understanding.

What carries the argument

The central machinery is the pairing of semantic decomposition with attention learning. Decomposition turns a long video into three token types: scene tokens (uniformly sampled frames), object tokens (SAM/SAM-2 mask crops from key frames), and action tokens (tracklets from a class-agnostic tracker, e.g., YOLOv10-X with BoT-SORT or SAM-2). Attention learning is a subset-selection optimization that balances query relevance $R(\cdot)$ and token diversity $S(\cdot)$; the scalar $\alpha$ controls the trade-off, defaulting to 0.9. This step is what lets the representation drop redundant tokens while keeping content useful for a given query, and it operates in either a global mode (whole video at once) or a streaming mode (sliding window over the selected subset so far).

What would settle it

Re-run SEAL on LVBench with a modified query that contains only the question stem (options removed), and compare the overall accuracy; if accuracy drops materially from 45.9%, the reported gain is inflated by answer leakage through the relevance term.

Watch

Extended reading notes

Core claim

The paper's central claim is that temporal redundancy, not model capacity, is the main obstacle to long-video understanding, and that replacing dense frame sampling with a small set of semantic entity tokens—static scenes, static objects, and dynamic actions—plus a subset-selection step that maximizes query relevance while preserving token diversity, yields a representation that transfers across tasks. The subset selection is formulated as optimizing $F_s(T_s|T_G, q) = \alpha \sum_{t_s \in T_s} R(t_s, q) + (1-\alpha) \sum_{t_i,t_j \in T_s, i\ne j} 1/S(t_i, t_j)$ over candidate token sets, with $R$ computed by BLIP-2 cosine similarity and $S$ by token-pair similarity. SEAL then plugs the selected tokens into either an MLLM head for QA or a vision head with a moment decoder for temporal grounding. The paper reports that this combination outperforms state-of-the-art methods on LVBench, MovieChat-1K, and Ego4D-NLQ, including models with more than twice its LLM size.

Load-bearing premise

The load-bearing premise is that the query q used in the relevance scorer does not leak the correct answer on multiple-choice tasks; on LVBench the query includes the question plus the four answer options, and if those options are not stripped, the token-selection step can partly select tokens simply for resembling the right answer text.

Editorial extensions

If this is right

  • A unified long-video representation can be shared across video QA, open-ended generative QA, and temporal grounding without task-specific video encoders.
  • A 34B-LLM pipeline with SEAL tokens can outperform 72B models that process densely sampled frames, suggesting token quality matters more than model scale.
  • Streaming mode offers a path to arbitrarily long videos: the model updates its representation incrementally and still beats full-video baselines on temporally intensive tasks.
  • The attention-learning objective's balance between relevance and diversity, with $\alpha=0.9$, outperforms either term alone, confirming that both focus and coverage matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication: if answer options are stripped from the query during relevance scoring on LVBench, accuracy should drop; reporting that ablation would clarify how much of the gain depends on query-image matching versus genuine video reasoning.
  • The subset-selection step is not end-to-end trainable (the paper states a memory constraint on the QP solver); making it differentiable via a continuous relaxation could let the token types and the selection co-adapt with the LLM head, possibly closing part of the remaining gap to human performance.
  • The same token-decomposition idea could be applied to other modalities, e.g., audio or egocentric sensor streams, wherever redundancy is high and queries are sparse.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces SEAL, a representation for long video understanding that decomposes a video into scene, object, and action tokens and then selects a fixed-size subset of these tokens by maximizing an objective that combines query relevance (via BLIP-2 cosine similarity) and token diversity. The selected tokens are passed either to a vision decoder for temporal grounding (Ego4D-NLQ) or to an MLLM for video QA (MovieChat-1K, LVBench). The authors report state-of-the-art results on LVBench with a 34B LLM (45.9% overall, surpassing Qwen2-VL-72B at 41.3%), competitive results on MovieChat-1K, and consistent improvements over SnAG on Ego4D-NLQ under token constraints. The paper includes ablations of the three token types, of the relevance/diversity trade-off, and of global vs. streaming modes.

Significance. If the reported numbers hold, the core idea of semantic decomposition plus query-aware diverse token selection is a useful and practical contribution to long video understanding: it is task-agnostic, supports streaming, and reduces token count substantially. The ablation study isolating each token type is informative, and the partial-observation experiments are a valuable addition. The main caveat is that the headline LVBench result depends on an unresolved specification of what query q is used for relevance scoring; the paper must rule out answer-option leakage before the superiority claim over larger models can be accepted. The paper would be strengthened by reproducible details of the subset-selection optimizer and by error bars on the headline numbers.

major comments (4)
  1. [Section 3.2 and Section D.2; Figure 3] The paper never states whether the query q used in the BLIP-2 relevance scoring R(t_s, q) contains the multiple-choice options on LVBench. The LVBench prompt in Section D.2 includes the question and the four options, and Figure 3 displays the options alongside the selected tokens. If the same q is used for relevance scoring, the subset selector becomes answer-aware: tokens matching the correct option text (e.g., 'blue and gold') would be up-ranked, inflating the reported 45.9% overall accuracy. Please state explicitly whether options are stripped from q during attention learning, and run an ablation on LVBench with options removed from q (and with q consisting of the question only) to quantify the effect. This is load-bearing for the claim that SEAL outperforms Qwen2-VL-72B by 4.6%.
  2. [Section 3.2 and Section 5 (Limitation)] The subset-selection objective is presented only as an arg-max problem; no algorithm is given for solving it, nor are its initialization, complexity, or convergence properties discussed. The Limitation paragraph in Section 5 states that Attention Learning is 'bounded by the memory constraint for the QP solver,' but no QP formulation or solver details appear in the paper or supplement. Without this information, the central mechanism is not reproducible and its computational cost cannot be assessed. Please provide the exact optimization procedure (or a reference), including how the pairwise diversity term is handled and how the fixed-size constraint is enforced.
  3. [Tables 1, 4, and 6; Section 4.3] All accuracy numbers are reported as point estimates without error bars, confidence intervals, or repeated-run statistics. The paper itself acknowledges 'high variance in the Summarization accuracy due to the small number of questions in this category and the randomness of LLM.' Given that the headline 45.9% overall accuracy is the basis for claiming state-of-the-art performance, the absence of variability information makes it impossible to determine whether the 4.6-point gap over Qwen2-VL-72B is statistically meaningful. Please report means and standard deviations over at least three runs, or bootstrap confidence intervals, especially for the comparisons in Tables 1 and 4.
  4. [Table 2] The Ego4D-NLQ comparison is made only against SnAG, and the reported gains are very small at the 450-token setting (R@1 0.3: 13.78 vs. 13.44; R@1 0.5: 9.26 vs. 9.23). The paper claims SEAL 'consistently outperforms' SnAG, which is supported in direction, but without statistical significance tests or multiple runs the margin could be noise. Please report variance or significance tests for these numbers, or soften the claim accordingly.
minor comments (5)
  1. [Throughout] There are several typos and inconsistencies: 'excell' should be 'excel', 'foundamental' should be 'fundamental', 'Attention Leaning' in Section 3.3 should be 'Attention Learning', and 'Moviechat'/'Ego4D-NLP'/'InterVL2' appear alongside 'MovieChat'/'Ego4D-NLQ'/'InternVL2'.
  2. [Table 6] The row labeled 'rep Yolo w/ SAM2' is unclear; the text suggests it refers to replacing YOLO with SAM2 for action-token extraction, but the label and caption should be more explicit.
  3. [Figure 2] The caption says 'Cross-Model Fusion' while the text says 'cross-modal fusion' in Section 3.4; please unify the terminology.
  4. [Section D.2] The LVBench prompt shown in Section D.2 includes the options, but it is not stated whether this prompt is also used as q for relevance scoring in Section 3.2; this is related to the major comment and should be clarified directly in that section.
  5. [Section 4.1] The sentence 'Scene, action, and object tokens are extracted at 8, 10, 1 FPS for MovieChat-1K, Ego4D-NLQ and LVbench' is ambiguous because the order of datasets does not match the order of the FPS values; please make the mapping explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SEAL is an empirical query-conditioned token-selection pipeline; the potential LVBench answer-option ambiguity is a benchmark-validity risk, not a demonstrated by-construction equivalence.

full rationale

SEAL does not claim a first-principles derivation; it proposes an empirical representation built from off-the-shelf components (SAM2, YOLOv10+BoT-SORT, BLIP-2, a QP-based subset selection, and an MLLM head). The selection objective in Section 3.2 maximizes R(t_s,q) plus a diversity term, where R is computed as BLIP-2 cosine similarity; this is a query-conditioned retrieval mechanism, not a quantity fitted to the target answer. The reported LVBench, MovieChat-1K, and Ego4D-NLQ results are evaluated against independent external benchmarks, and no fitted parameter is renamed as a prediction. The only plausible circularity concern is whether LVBench multiple-choice options are included in q during BLIP-2 relevance scoring; if they were, token selection could be partly answer-aware. However, the paper never specifies the composition of q for the attention-learning stage — Section 3.1 only says q 'may take different forms depending on the task, such as natural language text for video question and answering,' and Supplementary D.2 shows options only in the MLLM prompt. Without a specification demonstrating that options are fed to the BLIP-2 scorer, this remains a potential leakage/validity risk rather than a by-construction equivalence. No load-bearing self-citations or imported uniqueness theorems appear. Therefore no circular step is established under the quoted-evidence standard.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the semantic decomposition, the quality of BLIP-2 relevance scores, the cosine-similarity diversity metric, and per-dataset hyperparameters. The most consequential assumption is that the query used for selection does not contain the answer; this is not stated for LVBench. No new physical entities are introduced; the scene/object/action tokens are a representational abstraction, not an entity requiring independent falsification.

free parameters (4)
  • alpha (relevance/diversity balance) = 0.9
    The only hyperparameter in the selection objective; chosen by ablation on LVBench (Figure 5).
  • selected token count k = 16 (LVBench), 256 (MovieChat), 200/450 (Ego4D)
    Number of tokens passed to the prediction head; set per dataset in Section D.1 of the supplement.
  • action tracklet length thresholds Lmin/Lmax = 8/16 (MovieChat), 16/32 (Ego4D)
    Tracklets shorter than Lmin are discarded and longer than Lmax are split; values chosen per dataset in Section D.1.
  • SAM mask filtering thresholds = pred_iou_thresh=0.88, stability_score_thresh=0.92, box_nms_thresh=0.7
    Quality filters for object masks from the SAM2 automatic mask generator; set in Section D.1.
assumptions (5)
  • domain assumption A long video can be adequately represented by scene, object, and action tokens.
    Section 3.1 introduces the decomposition; the paper does not prove sufficiency of this token set.
  • domain assumption BLIP-2 cosine similarity between a visual token and the query is a valid relevance measure for downstream tasks.
    Used as R(ts,q) in the selection objective (Sec. 3.2); no analysis of failure modes.
  • domain assumption Pairwise token dissimilarity as measured by cosine similarity corresponds to information diversity.
    Used as S(ti,tj) in the objective; only ablative support is provided.
  • ad hoc to paper The query q used in relevance scoring does not contain the answer for multiple-choice tasks.
    On LVBench the paper does not state whether q includes the four answer options; if it does, the selection is conditioned on the answer.
  • standard math The subset selection problem in Sec. 3.2 has a computable solution within the given memory budget.
    The paper only mentions a memory-bound QP solver in the Limitations section, without specifying the algorithm or its approximation guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEAL: Semantic Attention Learning for Long Video Representation." pith.science (2026). https://pith.science/paper/A6TGWMIN

@misc{pith2026241201798,
  author       = {Pith},
  title        = {Pith review of: SEAL: Semantic Attention Learning for Long Video Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6TGWMIN}},
  note         = {Machine review of arXiv:2412.01798}
}
read the original abstract

Long video understanding presents challenges due to the inherent high computational complexity and redundant temporal information. An effective representation for long videos must efficiently process such redundancy while preserving essential contents for downstream tasks. This paper introduces SEmantic Attention Learning (SEAL), a novel unified representation for long videos. To reduce computational complexity, long videos are decomposed into three distinct types of semantic entities: scenes, objects, and actions, allowing models to operate on a compact set of entities rather than a large number of frames or pixels. To further address redundancy, we propose an attention learning module that balances token relevance with diversity, formulated as a subset selection optimization problem. Our representation is versatile and applicable across various long video understanding tasks. Extensive experiments demonstrate that SEAL significantly outperforms state-of-the-art methods in video question answering and temporal grounding tasks across diverse benchmarks, including LVBench, MovieChat-1K, and Ego4D.

Figures

Figures reproduced from arXiv: 2412.01798 by the authors.

Figure 1
Figure 1. Long Video Representation with Semantic Attention Learning (SEAL): (a) Conventional uniform sampling results in redun￾dant and cluttered visual information, making it difficult for both AI models and human brains to process efficiently. (b) Decomposing long videos into semantic entities such as scenes, objects, and actions reduces temporal redundancy, thus making model training and inference more efficient. In this … view at source ↗
Figure 2
Figure 2. SEAL Overview. During semantic decomposition, a long video V is decomposed into semantic tokens representing scenes, objects, and actions. Then, during attention learning, these tokens and the query q, are optimized for query relevance R(·) and token diversity S(·). The resulting attended token subset is then passed to a vision or an MLLM head for predictions. segmentation approach enables comprehensive object infor… view at source ↗
Figure 3
Figure 3. Qualitative results on LVBench. Two long videos visualized with questions, multiple choice options, and SEAL predicted answers. SEAL attends to relevant entities such as “royal family” and “stool” (Q1.a), different “meals” and “drinks” (Q2.a), “scene” and “location” (Q2.b) and correctly answers these questions. Although attending to relevant “push-up” activity (Q2.c), SEAL fails to predict the right answer due to th… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Accuracy vs. efficiency trade-off on LVBench. SEAL runs 2-3x faster than InternVL2 at the same accuracy, and is more accurate when compared at the same FPS. liance on densely sampled tokens and uses subject-level and motion aware representations/tokens to improve effic…
Figure 5
Figure 5. Figure 5: Ablation studies of different values of α on LVBench. α = 0.9 achieves the best performance across different tasks except for temporal grounding (TG) [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Distribution of answers generated by different models. The answers from InterVL2-40B and our method are the closest to the ground truth distribution. tial challenges in handling stylized or abstract visual con￾tent. While our method shows clear improvements over existi…
Figure 7
Figure 7. Figure 7: Additional qualitative results on LVBench.SEAL attends to relevant entities such as “tattoo” and “man’s arm” (Q3.a), different “rain forest plants” and “rain forest leaves” (Q3.b), “tall hat woman”, “dog”, and “performing” activity (Q4.a), “ black and white dog” and it…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 29 canonical work pages

  1. [1]

    Bot- sort: Robust associations multi-pedestrian tracking

    Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot- sort: Robust associations multi-pedestrian tracking. ArXiv, abs/2206.14651, 2022. 5, 4

    arXiv 2022

  2. [2]

    Frame-wise action representations for long videos via se- quence contrastive learning

    Minghao Chen, Fangyun Wei, Chong Li, and Deng Cai. Frame-wise action representations for long videos via se- quence contrastive learning. In CVPR, 2022. 7

    work page 2022

  3. [3]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 6, 2, 4

    arXiv 2024

  4. [4]

    Visual attention as an important visual function: an outline of manifestations, diagnosis and management of impaired vi- sual attention

    Meghomala Das, David M Bennett, and Gordon N Dutton. Visual attention as an important visual function: an outline of manifestations, diagnosis and management of impaired vi- sual attention. British Journal of Ophthalmology , 91(11): 1556–1560, 2007. 1

    work page 2007

  5. [5]

    Videoagent: A memory-augmented mul- timodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented mul- timodal agent for video understanding. In ECCV, pages 75–

    work page

  6. [6]

    Eva: Exploring the limits of masked visual representa- tion learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representa- tion learning at scale. In CVPR, 2023. 2

    work page 2023

  7. [7]

    Gueter Josmy Faure, Jia-Fong Yeh, Min-Hung Chen, Hung- Ting Su, Shang-Hong Lai, and Winston H. Hsu. Hermes: temporal-coherent long-form understanding with episodes and semantics. arXiv preprint arXiv:2408.17443, 2024. 6

    work page Pith review arXiv 2024

  8. [8]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ICCV, 2017. 1

    work page 2017

Show all 47 references
  1. [9]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Mar- tin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Meng...

    2022

  2. [10]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In CVPR, 2024. 2, 4

    2024

  3. [11]

    ActivityNet: A large-scale video benchmark for human activity understanding

    Fabian Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In CVPR,

  4. [12]

    Cone: An efficient coarse-to-fine alignment frame- work for long video temporal grounding

    Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan. Cone: An efficient coarse-to-fine alignment frame- work for long video temporal grounding. arXiv preprint arXiv:2209.10918, 2022. 3

    2022 arXiv

  5. [13]

    Zamir, Yu-Gang Jiang, Alex Gor- ban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah

    Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gor- ban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos ”in the wild”. CoRR, abs/1604.06182, 2016. 1

    2016 arXiv

  6. [14]

    The kinetics human action video dataset

    Will Kay, Jo ˜ao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017. 1

    2017 arXiv

  7. [15]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chlo´e Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B. Girshick. Segment anything. In ICCV, 2023. 3

    2023

  8. [16]

    Text-conditioned resampler for long form video understanding

    Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zis- serman, and Federico Tombari. Text-conditioned resampler for long form video understanding. In ECCV, pages 271–

  9. [17]

    From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In CVPR, 2022. 8

    2022

  10. [18]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML. PMLR, 2023. 4, 5

    2023

  11. [19]

    Videochat: Chat-centric video understanding

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wen Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 6

    2023 arXiv

  12. [20]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Sol- dan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. arXiv preprint arXiv:2206.01670, 2022. 5, 2 9

    2022 arXiv

  13. [21]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Ji- wen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 6

    2024 arXiv

  14. [22]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 5

    2019

  15. [23]

    Luck and Michelle Ford

    Steven J. Luck and Michelle Ford. On the role of selective attention in visual perception. Proceedings of the National Academy of Sciences of the United States of America, 95(3): 825–830, 1998. 1

    1998

  16. [24]

    Khan, and Fahad Shahbaz Khan

    Muhammad Maaz, Hanoona Abdul Rasheed, Salman H. Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In AMACL, 2023. 6

    2023

  17. [25]

    Morevqa: Exploring modular reason- ing models for video question answering

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reason- ing models for video question answering. In CVPR, pages 13235–13245, 2024. 2

    2024

  18. [26]

    SnAG: Scalable and accurate video grounding

    Fangzhou Mu, Sicheng Mo, and Yin Li. SnAG: Scalable and accurate video grounding. In CVPR, 2024. 5, 6

    2024

  19. [27]

    Snag: Scalable and accurate video grounding

    Fangzhou Mu, Sicheng Mo, and Yin Li. Snag: Scalable and accurate video grounding. In CVPR, pages 18930–18940,

  20. [28]

    Gpt-3.5, 2023

    OpenAI. Gpt-3.5, 2023. 5

    2023

  21. [29]

    Scanning only once: An end-to-end framework for fast temporal grounding in long videos

    Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao. Scanning only once: An end-to-end framework for fast temporal grounding in long videos. In ICCV, pages 13767–13777, 2023. 3

    2023

  22. [30]

    Egovideo: Exploring egocentric founda- tion model and downstream adaptation

    Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric founda- tion model and downstream adaptation. arXiv preprint arXiv:2406.18070, 2024. 2

    2024 arXiv

  23. [31]

    Streaming long video understanding with large language models

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuan- grui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405.16009, 2024. 2

    2024 arXiv

  24. [32]

    SAM 2: Segment anything in images and videos,

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

  25. [33]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InCVPR, pages 14313– 14323, 2024. 3, 4

    2024

  26. [34]

    E. Song, W. Chai, G. Wang, Y . Zhang, H. Zhou, F. Wu, X. Guo, T. Ye, Y . Lu, JN. Hwang, and G. Wang. MovieChat: From dense token to sparse memory for long video under- standing. In CVPR, 2024. 2, 3, 5, 6, 4

    2024

  27. [35]

    Moviechat+: Question-aware sparse memory for long video question answering

    Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024. 2, 3

    2024 arXiv

  28. [36]

    Hallucination mitigation prompts long-term video understanding

    Yiwei Sun, Zhihang Liu, Chuanbin Liu, Bowei Pu, Zhi- han Zhang, and Hongtao Xie. Hallucination mitigation prompts long-term video understanding. arXiv preprint arXiv:2406.11333, 2024. 3, 6

    2024 arXiv

  29. [37]

    Koala: Key frame-conditioned long video-llm

    Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In CVPR, 2024. 7

    2024

  30. [38]

    Yolov10: Real-time end-to- end object detection

    Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jun- gong Han, and Guiguang Ding. Yolov10: Real-time end-to- end object detection. ArXiv, abs/2405.14458, 2024. 5, 4

    2024 arXiv

  31. [39]

    Protege: Untrimmed pretraining for video temporal grounding by video temporal grounding

    Lan Wang, Gaurav Mittal, Sandra Sajeev, Ye Yu, Matthew Hall, Vishnu Naresh Boddeti, and Mei Chen. Protege: Untrimmed pretraining for video temporal grounding by video temporal grounding. In CVPR, 2023. 1

    2023

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  33. [41]

    W. Wang, Z. He, W. Hong, Y . Cheng, X. Zhang, J. Qi, S. Huang, B. Xu, Y . Dong, M. Ding, and J. Tang. LVBench: An extreme long video understanding benchmark, 2024. 1, 2, 3, 5, 6

    2024

  34. [42]

    Longvlm: Efficient long video understand- ing via large language models

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understand- ing via large language models. In ECCV, pages 453–470. Springer, 2024. 1, 3, 7

    2024

  35. [43]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 3, 6

    2024 arXiv

  36. [44]

    Yi: Open foundation models by 01.ai

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yu...

    2024 arXiv

  37. [45]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. In EMNLP, 2023. 3, 6

    2023

  38. [46]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 6

    2024

  39. [47]

    tattoo” and “man’s arm

    Guangze Zheng, Shijie Lin, Haobo Zuo, Changhong Fu, and Jia Pan. NetTrack: Tracking Highly Dynamic Objects with a Net. In CVPR, 2024. 4 10 SEAL: SEmantic Attention Learning for Long Video Representation Supplementary Material A. Additional Ablations Streaming Window Size . Tab...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.