Pith. sign in

REVIEW 5 major objections 6 minor 70 references

IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that injecting instance-level features as prompts into an attention-based visual compressor lets a VideoLLM keep character identity across scene cuts, improving multi-shot question answering by 8.9 points over its…

desk verdict IPFormer is a fresh idea and the benchmark targets a real gap, but the evaluation is self-referential and the overlap concern needs to be resolved before trusting the gains. read the letter →

arxiv 2506.21116 v2 pith:B5EEEMMM submitted 2025-06-26 cs.CV cs.AI

Yujia Liang , Jile Jiao , Xuetao Feng , Zixuan Ye , Yuan Wang , Zhicheng Wang This is my paper
classification cs.CVcs.AI
keywords multi-shotvideounderstandinglargelanguagemodelsinstancepromptsvisualtokencompressionquestionansweringbenchmarkidentityattention-basedconnector
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VideoLargeLanguageModels (VideoLLMs) fail at multi-shot clips — videos with camera cuts and scene changes — because existing training data rarely contains multi-shot annotations and because current visual encoders compress instance features in a lossy way. The paper introduces MultiClip-Bench, a dataset of 6.7k multi-shot videos with dense captions, keyframe descriptions with person IDs, and 45.5k training QA pairs plus a 2.75k manually curated test set. It then proposes IPFormer-VideoLLM, which extracts instance-level features from detected bounding boxes, clusters them across frames, and injects them as prompts into an attention-based visual compressor. On MultiClip-Bench, the model with this training data reaches 57.0 accuracy versus 44.5 for the Video-LLaVA baseline, while cutting visual tokens from 2056 to 320 and reducing training time to about a quarter. The paper argues this shows instance-prompt injection plus multi-shot data yields substantial gains in identity consistency across scene cuts.

What carries the argument

The load-bearing component is the IP Token Generator inside the visual alignment module. It uses a Deformable-DETR-based detector (initialized from Groma's Region Proposer) to produce up to 10 bounding boxes per frame, pools RoI features, clusters instances across the eight frames of a slice by cosine similarity with a threshold of 0.9, and averages each cluster to form instance prompt tokens (maximum 80 per slice). These tokens, together with repeated frame-level class tokens and global-average-pooling tokens, are added to learnable queries that drive cross-attention, so the compressor is effectively told which instances to preserve before compression.

What would settle it

A multi-shot test set built from independent human annotations (fresh video clips labeled by multiple annotators with measured agreement) on which IPFormer-VideoLLM shows no significant accuracy advantage over Video-LLaVA, or an experiment adding the same MultiClip training data to a baseline model so the +8.9 point gap collapses when test answers are re-verified by humans.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that instance identity in multi-shot videos is lost because visual alignment modules encode instance features 'in a discrete or lossy manner' — full projection preserves tokens but spreads information thinly, while compression methods like average pooling or plain Q-Former discard identity cues. IPFormer-VideoLLM fixes this by generating instance tokens: a category-agnostic detector proposes region boxes per frame, global RoI pooling extracts instance features, cosine similarity clusters the same instance across frames within each 8-frame slice, and channel-wise averaging produces an instance prompt for each group. These instance prompts, concatenated with frame-level class-token and global-pool features, are added to learnable query tokens before cross-attention in the visual compressor, so the aggregation is guided toward the identities that matter. With this design, the paper reports that the model outperforms its Video-LLaVA baseline by 3.6 points without MultiClip training data and by 8.9 points with it, and also beats prior open models on MVBench, NExT-QA, and IntentQA.

Load-bearing premise

The validity of MultiClip-Bench as a measure of multi-shot understanding, since the training and test QA are generated by GPT-4 and the keyframe descriptions by LLaVA-1.5 and Gemini, then manually curated without external verification or inter-annotator agreement.

Editorial extensions

If this is right

  • With MultiClip training, IPFormer-VideoLLM improves MultiClip-Bench accuracy from 44.5 to 57.0 (+8.9), with the largest gains in consistency (+8.4), short-frame (+10.5), and unexpected-content (+12.0) question types.
  • The instance-prompt design makes aggressive token compression safe for identity: 320 visual tokens instead of 2056, with first-stage training time cut from about 3-4 days to 1-2 days on the same hardware.
  • Gains transfer beyond the new benchmark: the paper reports 70.6 on NExT-QA, 75.3 on IntentQA, and 48.3 on MVBench, exceeding the Video-LLaVA baseline and most open 7B models.
  • Ablations show that frame-level anchors add +1.7 points, instance prompts without clustering add little, and clustering is what unlocks the instance-token gains, rising from 42.3 to 46.2 on MultiClip.
  • Scaling to more frames continues to help: the 48-frame variant reaches 58.8 on MultiClip, suggesting the approach is not saturated at 16 frames.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark's ground truth inherits the same closed-source annotation pipeline (GPT-4, LLaVA-1.5, Gemini) that generated the training data, part of the reported multi-shot gain may reflect alignment with those models' annotation biases rather than a general improvement in identity understanding; a test on independently human-annotated multi-shot clips would separate the two.
  • The 0.9 cosine-similarity threshold for grouping instances is a fixed hyperparameter; appearance changes from lighting, pose, or occlusion could split one identity into several groups, and a learned or adaptive grouping rule is a natural next step.
  • Because instance tokens are grounded in detector bounding boxes, the method inherits detector failures in crowded or highly occluded scenes; counting or identity questions in such scenes could erode the advantage.
  • The general principle — compress video tokens around the instances that carry identity — could extend to object-centric tasks like video editing, retrieval, or embodied perception, where preserving specific objects across views matters more than global scene summaries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. IPFormer-VideoLLM addresses multi-shot video understanding in Video-LLMs. The paper contributes MultiClip-Bench, a dataset of filtered multi-shot videos with dense captions, keyframe descriptions, character IDs, and 45.5k training plus 2.75k manually curated test QA pairs; and IPFormer-VideoLLM, a Video-LLaVA-based model with an instance-prompt compressor that detects objects, clusters instance features across frames, and injects them as queries into an attention-based connector, reducing visual tokens from 2056 to 320. Experiments report gains on MultiClip-Bench (+8.9 over Video-LLaVA), on NExT-QA, IntentQA, and other video QA benchmarks, and a roughly 4x training-time reduction. The paper also reports that its training data improves ActivityNet and MVBench in ablations.

Significance. The contribution is potentially useful. The instance-prompt compressor is a clear, simple idea with a sensible motivation, and the efficiency numbers (320 tokens versus 2056, 10h versus 41h) are attractive if they hold. The dataset's focus on identity consistency, short-frame understanding, and unexpected content addresses an underexplored problem. The ablations in Table 8 (D1 to D3) are a genuinely informative sign that the MultiClip training data also helps ActivityNet and MVBench, which provides some external grounding beyond the self-built benchmark. However, the load-bearing evidence is weakened by the absence of deduplication analysis, the self-constructed nature of the main benchmark, a confounded architecture-only comparison, and single-run results. These issues are fixable and should be addressed before the claims can be accepted.

major comments (5)
  1. [§3.1, §5.4] The training and evaluation pools overlap by construction. Section 3.1 lists MultiClip training videos as sourced from Kinetics-710, VideoChatGPT, VideoChat, YouCook2, NExTQA, WebVid, and EgoQA, and Section 5.4 evaluates on NExT-QA, EgoSchema, IntentQA, MSVD-QA, MSRVTT-QA, and ActivityNet-QA; the instruction-tuning stage also reuses 100K Video-ChatGPT pairs. The paper reports no deduplication between MultiClip training videos (or the 68K added instruction pairs) and the test splits of these benchmarks. If a non-negligible fraction of test videos appears in training, the reported gains on NExT-QA and IntentQA (Table 5) and the open-ended QA benchmarks (Table 3) reflect memorization rather than multi-shot understanding. Please release the video IDs of both MultiClip splits and the 68K instruction subset, and re-run the evaluations after removing any overlapping videos.
  2. [§3.2, Table 2] The central multi-shot evaluation is on a self-constructed benchmark whose annotations are produced by the same automated pipeline used to generate the training data. Section 3.2 states that QA pairs are generated and refined by GPT-4, with keyframe descriptions produced by LLaVA-1.5 and Gemini, and then manually curated by the authors; no inter-annotator agreement or comparison with an externally built multi-shot benchmark is reported. Because the test set is therefore not independent of the training distribution, the +8.9-point gain in Table 2 may partly reflect alignment with the annotation pipeline's systematic biases. Please provide human agreement statistics on the 2,750 test pairs, add an external multi-shot or identity-tracking benchmark, or release the test set so that independent groups can evaluate.
  3. [§5.1, Table 2] The ablation that isolates the architecture is confounded with additional training data. In Table 2, 'Ours (w/o MultiClip)' differs from Video-LLaVA not only in the proposed visual alignment module but also in the instruction-tuning data: Section 5.1 adds 326K VideoChat2 pairs and 27K Perception plus 27K STAR pairs. The sentence 'our model design leads to a significant improvement (+3.6%)' therefore conflates data and architecture. Please include a Video-LLaVA baseline trained on the same added data without MultiClip, and an IPFormer variant trained on exactly Video-LLaVA's original data, to separate the two factors.
  4. [Tables 2–8] All quantitative results are single runs without error bars or significance tests. Several load-bearing comparisons are small, e.g., Table 7 ActivityNet 45.5 versus 45.3, Table 1 ActivityNet 45.5 versus 45.3, and Table 6 S1 versus S2 with a 1.7-point gain; without variance estimates or multiple seeds these differences cannot be distinguished from training noise. Please report means and standard deviations over at least three runs for the main tables, or otherwise justify that the differences are stable.
  5. [Table 1, §5.2] The efficiency claims are not consistently quantified. Table 1 reports a training time of 10h for Ours (8f) and '−' for Ours (16f), while Section 5.2 first says the first-stage training is shortened from 41h to 10h and then says the entire training process is reduced from 3–4 days to 1–2 days; similarly, Table 1 lists FPS 2.0 for Ours (16f) while the text reports 1.4 FPS when real-time box detection is included. Please state clearly which stage each training-time number refers to, and give the inference FPS with and without detection under identical conditions.
minor comments (6)
  1. [§3.1, §3.2] Section 3.1 reports '23k high-quality video-text pairs (6.7k videos)' while Section 3.2 reports that the training set contains 45.5k video-text pairs; please clarify whether the 45.5k count refers to QA pairs rather than video-text pairs and how the two counts relate.
  2. [§4.3] The hyperparameters X=5, V=80, similarity threshold 0.9, and M<10 are introduced without sensitivity analysis; please either add an ablation or state explicitly that these values are adopted from prior work and why the results are insensitive to them.
  3. [Tables 1, 2] The column headers 'Consistency uds' and 'Short uds' in Table 2 are not expanded, and the column label 'FPS Act-Net' in Table 1 conflates two different quantities; please define all abbreviations in the captions.
  4. [Figures 1, 2] Figure 1 has an overlapping and truncated caption ('Multi-shot T ransitionsVideo-LLaV A uses full projection...') and Figure 2 contains an incomplete sentence ending 'otherwise image.'; please fix the figure captions and these typographical issues.
  5. [§3.1] Reference [26] is used for both VideoChat and YouCook2 in Section 3.1; YouCook2 needs its own citation.
  6. [§5.3, §5.4, Table 8] The paper repeatedly defers implementation details and additional results to 'supplementary materials' (e.g., Section 5.3, Section 5.4, Table 8) without including them in the version under review; please include the supplementary material or summarize the missing details in the main text.

Circularity Check

1 steps flagged · score 4.0 of 10

MultiClip-Bench is generated by the same GPT-4/Gemini pipeline used to build MultiClip training data, so the +8.9-point gain is partly a self-consistency measure; external ActivityNet/MVBench improvements provide partial independent grounding.

  1. fitted input called prediction [Section 3.2 (QA Pairs of Multi-Shot Scenes); Section 5.3 (Result on MultiClip-Bench); Table 8 (Effect of training dataset)]
    "Then we pass the generated initial QA pairs to GPT-4 for refinement, where 4 to 6 high-quality QA pairs are retained. Finally, three incorrect options are added to each question, converting them into a multiple-choice format with GPT-4. ... The training set contains 45.5k video-text pairs, and the test set contains 2.75k video-text pairs, with no overlap between the two sets."

    Both the 68K MultiClip training pairs (Table 8, D3) and the 2.75K MultiClip-Bench test questions are outputs of the same GPT-4 QA generation/refinement pipeline over the same filtered video pool, with keyframe descriptions produced by LLaVA-1.5 and Gemini. The headline +8.9-point gain on MultiClip-Bench therefore measures how well the model fits a GPT-4/Gemini-generated annotation distribution rather than an independently established multi-shot ground truth: the training input and the evaluated target are produced by the same closed-source models. The claim that the test benchmark 'provides a reliable measure of the model capability in multi-shot scenarios' is not supported by an external benchmark of multi-shot ability.

full rationale

The central circularity concern is that MultiClip-Bench is an author-constructed dataset whose training and test splits share the same GPT-4/Gemini generation pipeline; the paper reports a large MultiClip-Bench gain from training on the MultiClip training set, so part of the improvement is alignment with that pipeline's annotation style. This is not a formal reduction of one equation to another, but it is a self-constructed evaluation that weakens the independence of the headline claim. The paper does provide some external evidence: Table 8 shows the MultiClip training data also improves ActivityNet-QA and MVBench, and Tables 3-5 report results on standard QA benchmarks. There are no load-bearing self-citations or imported uniqueness theorems; the architectural claims about instance-prompt aggregation are design statements rather than derived predictions. On balance, the circularity is moderate, not total, and the independent external benchmarks justify a score of 4 rather than 6.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or external entities are postulated; IP tokens are internal model features with no independent falsifiable handle outside the paper.

free parameters (4)
  • Instance clustering similarity threshold = 0.9
    Used in IP Token Generator to group instance features (Section 4.3); set by hand, no sensitivity analysis.
  • Max instance tokens per slice V = 80
    Instance tokens padded to 80 per slice (Section 4.3); arbitrary cap.
  • Frame token repeat factor X = 5
    Global features and cls tokens repeated 5 times as frame guidance (Section 4.2).
  • Max candidate boxes per frame M = <10
    Retain at most M boxes per frame (Section 4.3), truncating excess.
assumptions (5)
  • domain assumption Multi-shot deficiency is caused by lack of multi-shot annotations and lossy instance encoding.
    Paper's motivating diagnosis in Section 1 and 4.2; not proven independently.
  • domain assumption Character ID changes are a valid proxy for multi-shot content.
    Video selection uses OCSORT person tracking and ID transitions (Section 3.1).
  • domain assumption Gemini-1.5-Pro-flash can reliably filter and annotate multi-shot videos.
    Used as oracle for filtering and description refinement (Section 3.1); no quality assessment reported.
  • domain assumption GPT-4 generated QA pairs are correct and diverse enough for training and testing.
    QA generation and refinement via GPT-4 (Section 3.2); test set manually curated but training set not.
  • standard math Standard attention and Q-Former machinery is accepted as-is.
    Relies on BLIP-2 Q-Former and Conditional-DETR anchors (Sections 4.2, references [23,39]).

how reviews work

0 comments
Cite this review

Pith. "Pith review of IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes." pith.science (2026). https://pith.science/paper/B5EEEMMM

@misc{pith2026250621116,
  author       = {Pith},
  title        = {Pith review of: IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5EEEMMM}},
  note         = {Machine review of arXiv:2506.21116}
}
read the original abstract

Video Large Language Models (VideoLLMs) have demonstrated remarkable understanding capabilities, but are found struggling to tackle multi-shot scenarios,e.g., video clips with varying camera angles or scene changes. This challenge can render failures such as instance identity forgetting and key frame negligence. In this work, we first attribute the challenge to the lack of multi-shot annotations among existing datasets and therefore we introduce a new dataset termed MultiClip-Bench, featuring dense descriptions and instruction-based question-answering pairs tailored for multi-shot scenarios. We empirically find that the training set significantly boosts the multi-shot performance, while the testing benchmark provides a reliable measure of the model capability in multi-shot scenarios. By further analyzing and discovering that current models only encode instance features in a discrete or lossy manner, at the risk of missing identity information, we then contribute a new model IPFormer-VideoLLM. Its key idea is the injection of instance-level features as instance prompts through an efficient attention-based connector. This allows for the aggregation of instance-specific information across scenes. Experiments demonstrate that our proposed dataset and model not only enhance the multi-scene video understanding significantly, but also offer distinct advantages across various video benchmarks.

Figures

Figures reproduced from arXiv: 2506.21116 by the authors.

Figure 1
Figure 1. Multi-shot Transitions Video-LLaVA uses full projection, while VideoChat2 reduces video tokens via at￾tention mechanisms. In scenes with discontinuous charac￾ter appearances (multi-scene transitions), existing models often fail to distinguish characters, leading to counting er￾rors. Our method compresses tokens and uses detectors and clustering to generate instance prompts, to guide attention fusion across scenes an… view at source ↗
Figure 2
Figure 2. Challenges in Multi-shot Scenarios. In multi-shot scenarios, Dense description capability of PLLaVA falters. As shown on the left, the system misses some key frames, only recognizing the woman in green while failing to detect the man in yellow. In the example on the right, the system exhibits person confusion, incorrectly attributing the actions of two individuals to a single person. ID changes Clear identification … view at source ↗
Figure 3
Figure 3. Description Annotations of MultiClip-Bench First, we filter the target videos, and then use close-source large video models to generate the three key elements of video description: keyframe description, character information, and dense caption. tion process and examples are shown in the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: QA Generation Pipeline and MultiClip-Bench Examples. Questions fall into four types: consistency, short-frame, unexpected content, and others. The video shows children flying kites and running. Type 1 involves person re-identification and discontinuous actions. Type 2 …
Figure 5
Figure 5. Figure 5: The pipeline of Our IPFormer-VideoLLM. First, video slices are sampled using a sliding window, and a video encoder extracts visual features from each slice. These features are passed to a visual alignment module, where frame-level and instance-level information are ext…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 38 canonical work pages

  1. [1]

    Medec: A benchmark for medical error detection and correction in clinical notes

    Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.19260, 2024. 8

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 1

    2022

  3. [3]

    Qwen-vl: A versatile vision- language model for understanding, localization

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- language model for understanding, localization. Text Reading, and Beyond , 2, 2023. 3

    work page 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025. 1, 3, 7, 8

    arXiv 2025

  5. [5]

    Frozen in time: A joint video and image en- coder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G¨ ul Varol, and Andrew Zis- serman. Frozen in time: A joint video and image en- coder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 1728–1738, 2021. 3

    work page 2021

  6. [6]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In Workshop on Video-Language Models@ NeurIPS 2024 , 2024. 8

    work page 2024

  7. [7]

    Observation-centric sort: Rethinking sort for robust multi-object tracking

    Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9686–9696, 2023. 3

    work page 2023

  8. [8]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceed- ings of the 49th annual meeting of the association for computational linguistics: human language technolo- gies, pages 190–200, 2011. 7

    work page 2011

Show all 70 references
  1. [9]

    Timemarker: A versatile video- llm for long and short video understanding with su- perior temporal localization ability

    Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video- llm for long and short video understanding with su- perior temporal localization ability. arXiv preprint arXiv:2411.18211, 2024. 2

    2024 arXiv

  2. [10]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 , 2024. 1, 8

    2024 arXiv

  3. [11]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North Amer- ican chapter of the association for computational lin- guistics: human...

    2019

  4. [12]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wen- long Huang, et al. Palm-e: An embodied multimodal language model. 2023. 1

    2023

  5. [13]

    Video- mme: The first-ever comprehensive evaluation bench- mark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video- mme: The first-ever comprehensive evaluation bench- mark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 2

    2024 arXiv

  6. [14]

    Ego4d: Around the world in 3,000 hours of ego- centric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of ego- centric video. In Proceedings of the IEEE/CVF confer- ence on computer vis...

    2022

  7. [15]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 ,

    arXiv

  8. [16]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 14271–14280, 2024. 3

    2024

  9. [17]

    Lan- guage is not all you need: Aligning perception with language models

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Lan- guage is not all you need: Aligning perception with language models. Advances in Neural Information Pro- cessing Systems, 36:72096–72...

    2023

  10. [18]

    Chat-univi: Unified visual represen- tation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual represen- tation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 137...

    2024

  11. [19]

    An image grid can be worth a video: Zero-shot video question answering using a vlm

    Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth a video: Zero-shot video question answering using a vlm. IEEE Access, 2024. 8

    2024

  12. [20]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international con- ference on computer vision , pages 706–715, 2017. 1, 2

    2017

  13. [21]

    Seed-bench: Benchmark- ing multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmark- ing multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023. 2

    2023 arXiv

  14. [22]

    Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 1, 2, 3, 9

    2024 arXiv

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language mod- els

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language mod- els. In International conference on machine learning , pages 19730–19742. PMLR, 2023. 1, 7

    2023

  16. [24]

    Intentqa: Context-aware video intent reasoning

    Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF international confer- ence on computer vision , pages 11963–11974, 2023. 2, 8

    2023

  17. [25]

    Uniformerv2: Spa- tiotemporal learning by arming image vits with video uniformer

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spa- tiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211.09552 , 2022. 3

    2022 arXiv

  18. [26]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023. 1, 3, 8

    2023 arXiv

  19. [27]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 22195–...

    2024

  20. [28]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision , pages 323–340. Springer, 2024. 8

    2024

  21. [29]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 , 2023. 2, 3, 7, 8

    2023 arXiv

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36:34892– 34916, 2023. 7

    2023

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 26296– 26306, 2024. 3

    2024

  24. [32]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In European Conference on Computer Vision , pages 1–18. Springer, 2024. 1, 8

    2024

  25. [33]

    Mmbench: Is your multi-modal model an all-around player? In Eu- ropean conference on computer vision , pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Eu- ropean conference on computer vision , pages 216–233. Springer, 2024. 1, 2, 8

    2024

  26. [34]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. 7

    2023 arXiv

  27. [35]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pages 417–

  28. [36]

    Vista-llama: Reliable video narra- tor via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narra- tor via equal distance to visual tokens. arXiv preprint arXiv:2312.08870, 2023. 8

    2023 arXiv

  29. [37]

    Video-chatgpt: Towards detailed video understanding via large vision and lan- guage models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and lan- guage models. arXiv preprint arXiv:2306.05424 , 2023. 3, 8, 9

    2023 arXiv

  30. [38]

    Videogpt+: Integrating image and video encoders for enhanced video understanding

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.09418 , 2024. 1, 2

    2024 arXiv

  31. [39]

    Conditional detr for fast training convergence

    Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF international confer- ence on computer vision , pages 3651–3660, 2021. 6

    2021

  32. [40]

    Deep- stack: Deeply stacking visual tokens is surprisingly simple and effective for lmms

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deep- stack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural In- formation Processing Systems , 37:23464–23487, 2025. 8

    2025

  33. [41]

    Efficient non- maximum suppression

    Alexander Neubeck and Luc Van Gool. Efficient non- maximum suppression. In 18th international confer- ence on pattern recognition (ICPR’06), pages 850–855. IEEE, 2006. 6

    2006

  34. [42]

    Perception test: A diagnostic bench- mark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic bench- mark for multimodal video models. Advances in Neu- ral Information Processing ...

  35. [43]

    Ts-llava: Constructing visual to- kens through thumbnail-and-sampling for training- free video large language models

    Tingyu Qu, Mingxiao Li, Tinne Tuytelaars, and Marie- Francine Moens. Ts-llava: Constructing visual to- kens through thumbnail-and-sampling for training- free video large language models. arXiv preprint arXiv:2411.11066, 2024. 1, 3, 8

    2024 arXiv

  36. [44]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learni...

    2021

  37. [45]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural in- formation processing systems, 28, 2015. 6

    2015

  38. [46]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14313–14323,

  39. [47]

    Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  40. [48]

    Moviechat: From dense token to sparse memory for long video under- standing

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  41. [49]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 1, 3, 7, 8

    2024 arXiv

  42. [50]

    Llama: Open and effi- cient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 7

    2023 arXiv

  43. [51]

    Grounded-videollm: Sharpening fine- grained temporal grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine- grained temporal grounding in video large language models. arXiv preprint arXiv:2410.03290 , 2024. 2

    arXiv

  44. [52]

    Internvideo: General video founda- tion models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video founda- tion models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 , 2022. 1, 2

    2022 arXiv

  45. [53]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 , 2023. 1, 2

    2023 arXiv

  46. [54]

    Star: A benchmark for sit- uated reasoning in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenen- baum, and Chuang Gan. Star: A benchmark for sit- uated reasoning in real-world videos. arXiv preprint arXiv:2405.09711, 2024. 7

    2024 arXiv

  47. [55]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021. 2, 3, 8

    2021

  48. [56]

    Funqa: Towards surprising video comprehension

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuan- han Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. In European Conference on Computer Vision , pages 39–

  49. [57]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5288– 5296, 2016. 7

    2016

  50. [58]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994 , 2024. 3, 8

    2024 arXiv

  51. [59]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 3, 8

    2024 arXiv

  52. [60]

    Just ask: Learning to answer questions from millions of narrated videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international confer- ence on computer vision , pages 1686–1697, 2021. 3

    2021

  53. [61]

    Zero-shot video ques- tion answering via frozen bidirectional language mod- els

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video ques- tion answering via frozen bidirectional language mod- els. Advances in Neural Information Processing Sys- tems, 35:124–141, 2022. 8

    2022

  54. [62]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 8

    2023 arXiv

  55. [63]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Con- ference on Artificial Intelligence , pages 9127–9134,

  56. [64]

    Video- llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video- llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. 3, 8

    2023 arXiv

  57. [65]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 8

    2023 arXiv

  58. [66]

    Llava-mini: Efficient image and video large mul- timodal models with one vision token

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large mul- timodal models with one vision token. arXiv preprint arXiv:2501.03895, 2025. 8

    2025 arXiv

  59. [67]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems , 36: 46595–46623, 2023. 7

    2023

  60. [68]

    Languagebind: Ex- tending video-language pretraining to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Ex- tending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023. 7

    arXiv

  61. [69]

    Deformable detr: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 6, 7

    2010 arXiv

  62. [435]

    Springer, 2024. 6, 7

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.