Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Recursively zooming from coarse segments to fine boundaries lets a single vision-language model ground events in hour-long videos, beating prior specialized methods on MAD and VidChapters-7M.

desk verdict Novel recursive coarse-to-fine VLM for hour-long temporal grounding with table-consistent SOTA on MAD, but unmeasured coarse-stage recall and several internal inconsistencies mean the efficiency/accuracy trade-off is not yet pinned down. read the letter →

arxiv 2411.14901 v1 pith:KSU3HVFR submitted 2024-11-22 cs.CV cs.CL

Tanveer Hannan , Md Mohaiminul Islam , Jindong Gu , Thomas Seidl , Gedas Bertasius This is my paper · ORCID
classification cs.CVcs.CL
keywords temporalgroundinglong-videounderstandingvision-languagemodelsrecursivehierarchycoarse-to-finelocalizationconfidencecalibrationcontrastivetrainingtext-to-videoretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a vision-language model can locate a text-described event inside an hour-long video, something earlier VLMs could not do because they could only consume a small number of frames. Its solution, ReVisionLLM, mimics human search: scan the whole video with compact segment-level features to pick broad regions of interest, then zoom into those regions with frame-level features to predict exact start and end boundaries. If the claim holds, one instruction-following model could replace the multi-network pipelines used by specialized long-video grounding methods, and the same search-then-refine machinery transfers to text-to-video retrieval. The paper reports state-of-the-art results on MAD and VidChapters-7M, including a +2.6 point R1@0.1 gain over the previous best method on MAD.

What carries the argument

The load-bearing mechanism is the two-level temporal feature representation produced by the Hierarchical Adapter. Dense features keep every sampled frame's CLS token, while sparse features are produced by cross-attending each long segment to the query text and then condensing it with self-attention into one learnable embedding per segment; the LLM consumes sparse features at the upper hierarchies to select segments and dense features at the bottom hierarchy to refine boundaries. A progressive training schedule first teaches the model to localize events in short clips with contrastive absent-event segments, then freezes the adapter and fine-tunes a second LoRA on hour-long videos using sparse features. At inference, predictions are ranked by the inverse of the mean entropy of the generated words, a calibration step that replaces CLIP-similarity ranking.

What would settle it

A direct test is to measure, on the MAD test split, how often the ground-truth moment falls inside at least one of the coarse segments selected by the top hierarchy; if that recall is close to 100 percent, the residual error is in refinement, while if it is far below the final R1@0.1, the coarse stage is the bottleneck and the recursive argument needs a better first scan.

Watch

Extended reading notes

Core claim

The central claim is that the frame-limit barrier for VLM temporal grounding can be removed by recursive hierarchical perception. Given an hour-long video, the model's top hierarchy uses sparse temporal features, each condensing a long window into one embedding, to identify which broad segments are relevant to the query; lower hierarchies then feed dense, frame-aligned features from only those segments into the language model, which outputs start and end times or "Not Present." The authors show cumulative ablations where each component contributes, with the recursive process delivering the largest gain, and report that the full model outperforms the prior state of the art on both MAD and VidChapters-7M while processing fewer frames than the VTimeLLM baseline.

Load-bearing premise

The recursive pipeline depends on the first coarse scan having high recall: if the true event is not among the broad segments passed to the finer levels, no amount of refinement can recover it, and the paper does not report this coarse-stage recall.

Editorial extensions

If this is right

  • If ReVisionLLM's central claim is correct, a single instruction-following VLM can replace the multi-network, post-processing pipelines used by prior long-video grounding methods.
  • The hierarchy lets the model adapt to any video length by changing the number of levels, so the same checkpoint handles short clips, half-hour footage, and multi-hour movies.
  • Because the top levels process sparse segment embeddings, the model sees fewer frames than a full-video baseline and still improves accuracy, suggesting recursive zooming is an efficient scaling strategy for long video.
  • The contrastive training and entropy-based confidence ranking reduce false positives, and the reported ECE drop indicates the model's confidence scores are more trustworthy for selecting top-k predictions.
  • The same model, applied to concatenated videos, also solves text-to-video retrieval, indicating the grounding capability transfers to a related retrieval task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the top-level coarse recall is the pipeline's ceiling; measuring it directly would show how much of the remaining error comes from misses at the first scan rather than from boundary refinement.
  • One extension the authors leave implicit is applying the same hierarchy to dense video captioning or highlight detection, which share the coarse-to-fine search structure and could inherit the frame-efficiency gains.
  • The inverse-entropy confidence score could serve as a dynamic stopping rule for the recursion, skipping additional levels when the model is already certain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces ReVisionLLM, a recursive vision-language model for temporal grounding in hour-long videos. The model builds on a frozen CLIP encoder, a hierarchical adapter that produces dense and sparse temporal features, and a Vicuna-7B LLM with LoRA that predicts start/end boundaries progressively from coarse to fine. Training proceeds in stages: first on short contrastive segments, then on long videos with sparse features; inference ranks predictions by an inverse-entropy confidence score. The paper reports state-of-the-art claims on MAD and VidChapters-7M, ablations of the proposed modules and model variants, an ECE calibration comparison in the supplement, and a text-to-video retrieval generalization experiment on MSRVTT, with code released.

Significance. If the reported numbers are correct, ReVisionLLM is a practically relevant contribution: it demonstrates that a single instruction-following VLM can perform temporal grounding on hour-long inputs, and its recursive inference offers an accuracy/efficiency trade-off (57% of frames on MAD for the default variant). The paper has clear strengths: the staged ablations in Tables 2-4 isolate the contributions of contrastive segments, confidence-based ranking, and recursion; the supplementary ECE analysis directly addresses the calibration motivation; and the MAD headline number (+2.6% R1@0.1 over RGNet) is internally consistent with Table 1. However, several central claims and protocol details need additional support or correction before the results can be taken at face value, in particular the VidChapters-7M 'state-of-the-art' claim and the unmeasured recall of the top-level coarse selection.

major comments (3)
  1. [§5, Table 1] The text states that ReVisionLLM 'sets a new state-of-the-art' on VidChapters-7M and 'significantly outperforms' M-DETR, but the comparison in Table 1 is not uniformly favorable: at R1@0.3 the model scores 33.8 versus M-DETR's 37.4, and at R1@0.5 the two are essentially tied (27.4 versus 27.3). The claim rests on the average and on the high-IoU thresholds R1@0.7 and R1@0.9. Please specify which metric is the primary endpoint, report the aggregated recall consistently, and state whether the 'new state-of-the-art' claim is based on the average rather than on the standard lower-IoU thresholds. Without this clarification, the VidChapters-7M claim is stronger than the table supports.
  2. [§3.2, Fig. 3, Table 3] The recursive pipeline is structurally load-bearing: the top hierarchy consumes only sparse temporal features, and the lower hierarchies receive dense features only for segments inside the previously predicted interval. Any event not contained in the top-level interval is irrecoverable, yet the paper never reports the recall of this coarse stage. Table 3 makes the concern concrete: the top-down default ReVisionLLM uses 57% of frames and reaches R1@0.1 = 15.0, while the bottom-up ReVisionLLM-I uses 100% of frames and reaches R1@0.1 = 17.4. The 2.4-point gap may be exactly the cost of coarse misses, but the paper does not decompose it. Please report coarse-stage recall or per-hierarchy cumulative recall, and connect it to the efficiency claim of processing 43% fewer frames than VTimeLLM; otherwise the efficiency is purchased at an unquantified risk of discarding true events.
  3. [§3.2 and Supplementary S1] The hierarchy count and indexing are inconsistent. Section 3.2 defines I^(1) as the dense bottom level and higher levels l = 2,...,L as sparse, while Section 3.4 refers to 'the bottom hierarchy (l = 0)'. Table 4 ablates 0 to 3 hierarchies, but the supplement says 'we employ two hierarchies with long videos' and then describes a top level, a second level, and a final hierarchy. These inconsistencies prevent readers from reproducing the exact inference protocol for the default model and the ReVisionLLM-I variant. Please align the definitions, state exactly how many levels are used for MAD and VidChapters-7M, and specify which levels consume dense versus sparse features.
minor comments (7)
  1. [§5] The sentence on ReVisionLLM-I states that it outperforms RGNet by '+4.9% in R1@.1 and +6.3% in R1@.3', but Table 1 gives R1@.3 = 12.7 for ReVisionLLM-I versus 9.5 for RGNet, a difference of +3.2, not +6.3. Please correct the number.
  2. [Table 1 and References] Table 1 cites VTimeLLM* as [69], but reference [69] is not VTimeLLM; the correct reference is [18]. Similarly, the text refers to the CLIP encoder as [60], while [60] is CLIP-ViP; the CLIP paper is [45]. Please fix these reference mismatches.
  3. [§3.3, §4.2, Supp. S1] The LoRA training protocol is described inconsistently: Section 3.3 Stage 2 says 'fine-tuning only the same LoRA module used in Stage 1', Section 4.2 says the default model trains two LoRAs (one for the bottom hierarchy, one for higher levels), and the supplement says two separate LoRA modules are used. Please clarify which LoRA is updated in each stage and which one is used at each hierarchy.
  4. [§5.3 and Supp. S3] For the MSRVTT text-to-video retrieval experiment, the paper says all videos are concatenated into a single long video and the model predicts the relevant video index. Please specify how many videos are concatenated per test example, how the input length is handled for the 1k-Test set, and whether the 'hour-long' framing applies to this setting.
  5. [§5.1, Fig. 5] The video-length ablation in Figure 5 extends videos by repeating them multiple times. Please state explicitly whether the reported long-video behavior is measured on repeated content, since this is not equivalent to naturally longer videos with new scene structure.
  6. [§5.1, Table 3] The sentence on ReVisionLLM-U says it uses 'fewer trainable parameters (363M vs. 159M)', but Table 3 lists ReVisionLLM-U as 159M and ReVisionLLM as 363M. The numbers in parentheses appear to be in the wrong order.
  7. [§4.2.1] The implementation details paragraph contains the phrase 'Training is conducted on the using a total batch size of 128'; please remove the stray words.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is an empirical benchmark comparison on external datasets, and the sole same-group citation (RGNet) is a baseline, not a load-bearing premise.

full rationale

ReVisionLLM's derivation chain is a system-building and empirical-evaluation argument, not a formal derivation from a self-defined quantity. The recursive model is specified (Sec. 3.2) as a concrete encoder-adapter-LLM pipeline; the training objective (Eq. 3) is standard next-token likelihood; and the confidence score in Sec. 3.4 is an entropy-based ranking defined explicitly from the LLM's output distribution. No equation in the paper is defined in terms of the quantity it is said to predict, and no fitted parameter is renamed as a prediction. The main accuracy claims are evaluated with held-out R@IoU metrics on the external MAD, VidChapters-7M, and MSRVTT benchmarks, so they are falsifiable outside the paper's own fitted values. The only same-group reference, RGNet [15], appears as a comparison baseline; it is not invoked to justify the recursive architecture, to supply a uniqueness theorem, or to forbid alternative designs. The paper's own ablations (Tables 3 and 4) and the ReVisionLLM-I variant provide internal checks on the recursive pipeline. The unmeasured recall of the top-level coarse stage is a legitimate correctness and evaluation concern, because a miss at the coarse level is irrecoverable by design, but this is a risk about an architectural bottleneck rather than a circularity in which a prediction equals its input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the sufficiency of CLIP CLS features, the LLM's ability to output numeric boundaries, and the recall of the coarse search stage. These are domain assumptions not proven by the paper. The method introduces one new learned entity (the sparse segment token) without independent evidence. The remaining numerical choices are segmentation and training hyperparameters.

free parameters (6)
  • Sliding window length Lw = 125 s (MAD); 500 s (VidChapters-7M)
    Hand-chosen segment duration that defines the granularity of the hierarchical search; directly controls how many segments are produced from an hour-long video.
  • Sliding window stride = 25 s (MAD); 100 s (VidChapters-7M)
    Hand-chosen overlap between segments; determines the number of candidate segments and the positional resolution of the coarse stage.
  • Frames uniformly sampled per segment = 250 frames
    Hand-chosen temporal sampling density; for a 125 s window this yields about 2 frames per second, which limits the finest achievable boundary precision.
  • Number of hierarchies used at inference = 3 in Table 4; 2 stated in Supplementary S1
    The depth of the recursive refinement is a chosen configuration, and the paper is internally inconsistent about whether the default uses two or three levels.
  • Concurrent segment batch sizes at upper hierarchies = 100 segments at top level; 33 segments at second level
    Hand-chosen batch limits for LLM inference that also affect how many candidates can be considered in parallel.
  • LoRA rank and alpha = r=64, alpha=128
    Standard hyperparameters for the low-rank adapters; chosen by hand and not swept.
assumptions (5)
  • domain assumption Per-frame CLIP CLS tokens contain enough visual information to localize brief events (average 4.1 s) in hour-long videos.
    The Multimodal Encoder (Sec. 3.2) discards spatial tokens and uses only the CLS token per frame, so the entire visual signal for grounding is a global feature vector per sampled frame. If these features cannot distinguish visually similar scenes, the central localization claim fails.
  • domain assumption The pretrained Vicuna-7B LLM, after LoRA tuning, can emit reliable 'From s to e' frame-index responses.
    The LLM is tasked with producing numeric boundaries (Sec. 3.2). This assumes the base model plus adapters can learn and generalize a numeric-index output format that is uncommon in its pretraining.
  • domain assumption The coarse stage has high recall: the true event segment is among the segments selected at the top hierarchy.
    The recursive mechanism only zooms into segments flagged at coarser levels (Sec. 3.2 and S1). No recall of the coarse candidate set is measured, so a miss at the top level is unrecoverable and directly bounds the final accuracy.
  • domain assumption Ground-truth start/end annotations in MAD and VidChapters-7M are accurate and comparable across methods.
    All evaluations in Sec. 5 rely on these annotations and on the R@k at IoU protocol; any noise or protocol mismatch would affect the reported margins.
  • domain assumption The next-token prediction objective in Eq. 3 is a valid training signal for temporal grounding.
    The model is trained to maximize p(T | P), where T is the text phrase containing the boundary output; the paper does not justify that this objective correlates with IoU accuracy beyond empirical results.
invented entities (1)
  • Sparse temporal feature token S_i
    purpose: Condenses an entire video segment (e.g., 125 s) into a single learnable vector for coarse-level processing, reducing LLM token count.
    This learned query token is introduced by the paper (Sec. 3.2, Eq. 2) and its utility is supported only by the paper's own ablations; there is no external benchmark, theoretical guarantee, or independent replication showing that such a compression preserves event-localizing information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos." pith.science (2026). https://pith.science/paper/KSU3HVFR

@misc{pith2026241114901,
  author       = {Pith},
  title        = {Pith review of: ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSU3HVFR}},
  note         = {Machine review of arXiv:2411.14901}
}
read the original abstract

Large language models (LLMs) excel at retrieving information from lengthy text, but their vision-language counterparts (VLMs) face difficulties with hour-long videos, especially for temporal grounding. Specifically, these VLMs are constrained by frame limitations, often losing essential temporal details needed for accurate event localization in extended video content. We propose ReVisionLLM, a recursive vision-language model designed to locate events in hour-long videos. Inspired by human search strategies, our model initially targets broad segments of interest, progressively revising its focus to pinpoint exact temporal boundaries. Our model can seamlessly handle videos of vastly different lengths, from minutes to hours. We also introduce a hierarchical training strategy that starts with short clips to capture distinct events and progressively extends to longer videos. To our knowledge, ReVisionLLM is the first VLM capable of temporal grounding in hour-long videos, outperforming previous state-of-the-art methods across multiple datasets by a significant margin (+2.6% R1@0.1 on MAD). The code is available at https://github.com/Tanveer81/ReVisionLLM.

Figures

Figures reproduced from arXiv: 2411.14901 by the authors.

Figure 1
Figure 1. Existing vision-language models (VLMs) such as [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. , we show the operating principle of our model. At the top hierarchy, the model operates broadly, identifying rele￾vant segments (e.g., 5 minutes) from a 2-hour-long video. As it moves down the intermediate hierarchies, it narrows its focus to increasingly fine-grained temporal segments at the lowest hierarchy, pinpointing precise event boundaries (e.g., 3.5 seconds). Such a recursive processing structure of our mod… view at source ↗
Figure 3
Figure 3. The ReVisionLLM model. (Left) First, we detect segments (e.g., a few minutes) from an hour-long video using sparse temporal features produced by the Hierarchical Adapter. (Right) Then ReVisionLLM produces a precise temporal boundary using dense temporal features within the predicted segments. Note that the green box represents the same event boundary in both sub-figures, zooming in from left to right. The multimodal… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Progressive Training Method. Our model is trained progressively: first on short video segments and then on hour-long videos. (Left) In the first stage, the model learns to detect whether an event is present in the input video and, if so, predicts its precise start and …
Figure 5
Figure 5. Figure 5: Ablation on Video Length. Our recursive approach maintains strong performance even with videos up to 10 hours long, while the baseline method fails entirely in these cases. Video Length [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on MAD. ReVisionLLM accurately locates precise event boundaries that involve intricate actions (top) and complex visual details (bottom) within hour-long movies. In contrast, our VLM baseline fails entirely to capture these events. formance for long…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Video Temporal Grounding with Multimodal Large Language Model

    cs.CV 2025-08 unverdicted novelty 3.0 of 10

    A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.

Reference graph

Works this paper leans on

71 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [69]

    Learning 2d temporal adjacent networks for moment local- ization with natural language

    Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment local- ization with natural language. In AAAI, pages 12870–12877,

    work page

  2. [1]

    needle in a haystack

    Greg kamradt on x: ”pressure testing gpt-4-128k with long context recall 128k tokens of context is awesome - but what’s performance like? i wanted to find out so i did a “needle in a haystack” analysis some ex- pected (and unexpected) results here’s what i found: Findings: * gpt-4’s recall https://t.co/nhmokmfhw5” / x. https://twitter.com/GregKamradt/stat...

    work page 2024

  3. [2]

    needle in a haystack

    Greg kamradt on x: ”claude 2.1 (200k tokens) - pressure testing long context recall we all love increasing context lengths - but what’s performance like? anthropic reached out with early access to claude 2.1 so i repeated the “needle in a haystack” analysis i did on gpt-4 here’s what i found: https://t.co/b36knjtjme” / x. https://twitter.com/ GregKamradt/...

    arXiv 2024

  4. [3]

    https://sharegpt.com/

    Sharegpt: Share your wildest chatgpt conversations with one click. https://sharegpt.com/ . (Accessed on 11/19/2024). 1

    work page 2024

  5. [4]

    https://github.com/gkamradt/LLMTest_ NeedleInAHaystack

    gkamradt/llmtest needleinahaystack: Doing simple retrieval from llm models at various context lengths to measure accu- racy. https://github.com/gkamradt/LLMTest_ NeedleInAHaystack. (Accessed on 11/14/2024). 1

    work page 2024

  6. [5]

    Lo- calizing moments in long video via multimodal guidance

    Wayner Barrios, Mattia Soldan, Alberto Mario Ceballos- Arroyo, Fabian Caba Heilbron, and Bernard Ghanem. Lo- calizing moments in long video via multimodal guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13667–13678, 2023. 3, 6

    work page 2023

  7. [6]

    Functional brain organization of preparatory attentional control in visual search

    Patrick Bourke, Steven Brown, Elton Ngan, and Mario Li- otti. Functional brain organization of preparatory attentional control in visual search. Brain research, 1530:32–43, 2013. 2

    work page 2013

  8. [7]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 3

    2020

Show all 71 references
  1. [8]

    Videollm: Modeling video sequence with large language models

    Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292, 2023. 2

    2023 arXiv

  2. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 4, 6, 1

    2023

  3. [10]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. 3, 6

    2019

  4. [11]

    Uatvr: Uncertainty-adaptive text-video retrieval,

    Bo Fang, Wenhao Wu, Chang Liu, Yu Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. Uatvr: Uncertainty-adaptive text-video retrieval,

  5. [12]

    Multi-modal transformer for video retrieval

    Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, pages 214–229, 2020. 3

    2020

  6. [13]

    Assistgpt: A gen- eral multi-modal assistant that can plan, execute, inspect, and learn

    Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A gen- eral multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640, 2023. 2

    2023 arXiv

  7. [14]

    X-pool: Cross-modal language-video attention for text- video retrieval, 2022

    Satya Krishna Gorti, Noel V ouitsis, Junwei Ma, Keyvan Golestan, Maksims V olkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text- video retrieval, 2022. 6, 8

    2022

  8. [15]

    Rgnet: A unified retrieval and grounding network for long videos

    Tanveer Hannan, Md Mohaiminul Islam, Thomas Seidl, and Gedas Bertasius. Rgnet: A unified retrieval and grounding network for long videos. arXiv preprint arXiv:2312.06729,

    arXiv

  9. [16]

    Cone: An efficient coarse-to-fine alignment frame- work for long video temporal grounding

    Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan. Cone: An efficient coarse-to-fine alignment frame- work for long video temporal grounding. arXiv preprint arXiv:2209.10918, 2022. 1, 3, 5, 6, 7

    2022 arXiv

  10. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 1

    2021 arXiv

  11. [18]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14271–14280, 2024. 1, 2, 4, 5, 7

    2024

  12. [19]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. arXiv preprint arXiv:2403.19046, 2024. 1, 2

    2024 arXiv

  13. [20]

    Audio- enhanced text-to-video retrieval using text-conditioned fea- ture alignment, 2023

    Sarah Ibrahimi, Xiaohang Sun, Pichao Wang, Amanmeet Garg, Ashutosh Sanan, and Mohamed Omar. Audio- enhanced text-to-video retrieval using text-conditioned fea- ture alignment, 2023. 8

    2023

  14. [21]

    Efficient long- text understanding with short-text models

    Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long- text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11:284–299,

  15. [22]

    Diffusionret: Generative text-video retrieval with diffusion model, 2023

    Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Diffusionret: Generative text-video retrieval with diffusion model, 2023. 8

    2023

  16. [23]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  17. [24]

    Large language models must be taught to know what they don’t know

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don’t know. arXiv preprint arXiv:2406.08391, 2024. 4

    2024 arXiv

  18. [25]

    Uncertainty-aware evaluation for vision- language models

    Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eu- gene Ilyushin. Uncertainty-aware evaluation for vision- language models. arXiv preprint arXiv:2402.14418 , 2024. 1, 4

    2024 arXiv

  19. [26]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34: 11846–11858, 2021. 1, 6, 7

    2021

  20. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  21. [28]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2

    2023 arXiv

  22. [29]

    Ground- inggpt: Language enhanced multi-modal grounding model

    Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Vu Tu, et al. Ground- inggpt: Language enhanced multi-modal grounding model. In Proceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: L...

    2024

  23. [30]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1

    2023 arXiv

  24. [31]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2794–...

    2023

  25. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    arXiv

  26. [33]

    Reler@ zju-alibaba submission to the ego4d natural language queries challenge 2022

    Naiyuan Liu, Xiaohan Wang, Xiaobo Li, Yi Yang, and Yuet- ing Zhuang. Reler@ zju-alibaba submission to the ego4d natural language queries challenge 2022. arXiv preprint arXiv:2207.00383, 2022. 3

    2022 arXiv

  27. [34]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. corr abs/2307.03172 (2023). arXiv preprint arXiv:2307.03172 , 10, 2023. 1

    2023 arXiv

  28. [35]

    Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection

    Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3042–3051, 2022. 1

    2022

  29. [36]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 6, 1

    2019

  30. [37]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2

    2023 arXiv

  31. [38]

    Query-dependent video representa- tion for moment retrieval and highlight detection

    WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representa- tion for moment retrieval and highlight detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23023–23033, 2023. 1

    2023

  32. [39]

    Snag: Scalable and accurate video grounding

    Fangzhou Mu, Sicheng Mo, and Yin Li. Snag: Scalable and accurate video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18930–18940, 2024. 1, 3, 6

    2024

  33. [40]

    To- wards calibrated robust fine-tuning of vision-language mod- els

    Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi-Qi Cheng, and Kyungwoo Song. To- wards calibrated robust fine-tuning of vision-language mod- els. arXiv preprint arXiv:2311.01723, 2023. 1, 4

    2023 arXiv

  34. [41]

    Obtaining well calibrated probabilities using bayesian binning

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial In- telligence, 2015:2901–2907, 2015. 2

    2015

  35. [42]

    Scanning only once: An end-to-end framework for fast temporal grounding in long videos

    Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao. Scanning only once: An end-to-end framework for fast temporal grounding in long videos. arXiv preprint arXiv:2303.08345, 2023. 1, 3, 5, 6

    2023 arXiv

  36. [43]

    Momen- tor: Advancing video large language model with fine-grained temporal reasoning, 2024

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat- Seng Chua, Yueting Zhuang, and Siliang Tang. Momen- tor: Advancing video large language model with fine-grained temporal reasoning, 2024. 1, 2

    2024

  37. [44]

    Chatvtg: Video temporal grounding via chat with video dialogue large language models

    Mengxue Qu, Xiaodong Chen, Wu Liu, Alicia Li, and Yao Zhao. Chatvtg: Video temporal grounding via chat with video dialogue large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1847–1856, 2024. 2

    2024

  38. [45]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. pages 8748–8763, 2021. 6

    2021

  39. [46]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14313–14323, 2024. 1, 2

    2024

  40. [47]

    Vlg-net: Video-language graph matching network for video grounding

    Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem. Vlg-net: Video-language graph matching network for video grounding. In ICCV, pages 3224–3234,

  41. [48]

    Mad: A scalable dataset for language grounding in videos from movie audio descriptions

    Mattia Soldan, Alejandro Pardo, Juan Le ´on Alc´azar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In CVPR, pages 5026–5035,

  42. [49]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  43. [50]

    Avicuna: Audio-visual llm with interleaver and context- boundary alignment for temporal referential dialogue

    Yunlong Tang, Daiki Shimada, Jing Bi, and Chenliang Xu. Avicuna: Audio-visual llm with interleaver and context- boundary alignment for temporal referential dialogue. arXiv preprint arXiv:2403.16276, 2024. 2

    2024

  44. [51]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  45. [52]

    Chatvideo: A tracklet-centric multimodal and versatile video understand- ing system

    Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Chatvideo: A tracklet-centric multimodal and versatile video understand- ing system. arXiv preprint arXiv:2304.14407, 2023. 2

    2023 arXiv

  46. [53]

    Omnivid: A generative framework for universal video understanding

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Omnivid: A generative framework for universal video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18209–18220, 2024. 2

    2024

  47. [54]

    Text is mass: Modeling as stochastic embedding for text-video retrieval, 2024

    Jiamian Wang, Guohao Sun, Pichao Wang, Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Text is mass: Modeling as stochastic embedding for text-video retrieval, 2024. 8

    2024

  48. [55]

    Hawkeye: Training video- text llms for grounding text in videos

    Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video- text llms for grounding text in videos. arXiv preprint arXiv:2403.10228, 2024. 2

    2024 arXiv

  49. [56]

    Five factors that guide attention in visual search

    Jeremy M Wolfe and Todd S Horowitz. Five factors that guide attention in visual search. Nature human behaviour, 1 (3):0058, 2017. 2

    2017

  50. [57]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 3, 8

    2016

  51. [58]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1

    2024 arXiv

  52. [59]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 2

    2024 arXiv

  53. [60]

    Clip-vip: Adapting pre- trained image-text model to video-language representation alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language representation alignment. ICLR, 2023. 3, 5, 6, 1

    2023

  54. [61]

    Clip-vip: Adapting pre- trained image-text model to video-language representation alignment, 2023

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language representation alignment, 2023. 8

    2023

  55. [62]

    Vidchapters-7m: Video chapters at scale,

    Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vidchapters-7m: Video chapters at scale,

  56. [63]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. InProceed- ings of the IEEE/CVF Conference on Computer Vi...

    2023

  57. [64]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  58. [65]

    A joint se- quence fusion model for video question answering and re- trieval

    Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint se- quence fusion model for video question answering and re- trieval. In ECCV, pages 471–487, 2018. 3

    2018

  59. [66]

    A sim- ple llm framework for long-range video question-answering,

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A sim- ple llm framework for long-range video question-answering,

  60. [67]

    Span-based localizing network for natural language video lo- calization

    Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video lo- calization. arXiv preprint arXiv:2004.13931, 2020. 3

    2004 arXiv

  61. [68]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 1, 2

    2023 arXiv

  62. [70]

    Learning video representations from large lan- guage models

    Yue Zhao, Ishan Misra, Philipp Kr ¨ahenb¨uhl, and Rohit Girdhar. Learning video representations from large lan- guage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6586– 6597, 2023. 2, 1 ReVisionLLM: Recursive Vision-Langu...

    2023

  63. [4096]

    <video> Does the <event> happen in the video? Answer yes or no

    [9] using a linear transformation. Large Language Model. We utilize a pre-trained Vicuna- 7B [9] model to ground queried events using the adapted visual features. Built upon LLaMA [51], this model con- sists of 32 transformer layers and has been fine-tuned on 70K user-shared c...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.