Pith. sign in

REVIEW 4 major objections 6 minor 57 references

DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Representing a video as one sharp keyframe plus four small motion prompts beats 16-frame video question answering.

desk verdict Real efficiency win and a plausible representation, but the stated '~2% SOTA' gain and the attention-mechanism story are both unsupported by the paper's own tables. read the letter →

arxiv 2507.15569 v1 pith:5G2VZEBP submitted 2025-07-21 cs.CV

Xiaoyi Bao , Chenwei Xie , Hao Tang , Tingyu Weng , Xiaofeng Wang , Yun Zheng , Xingang Wang This is my paper
classification cs.CV
keywords videounderstandingquestionansweringvisualpromptstemporalpromptrotarypositionembeddingmulti-modallargelanguagemodelstokenefficiencykeyframes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Videos are usually fed to multimodal LLMs as many frames, with temporal and spatial information processed separately and merged later. This paper proposes Dynamic-Image (DynImg), a representation that composes one high-resolution keyframe with four heavily downscaled non-keyframes pasted below it in temporal order, so the visual encoder sees motion and space together from the start. The small frames act as temporal prompts: the encoder's self-attention can match moving objects in the prompts to their locations in the keyframe and focus fine-grained attention there. The paper claims this yields about 2 percent higher accuracy than state-of-the-art methods on MSVD, MSRVTT, ActivityNet, TGIF, and MVBench, while using only 576 visual tokens instead of 2304 for a 16-frame input. Because the same accuracy comes with only 576 visual tokens instead of 2304, the claim is a gain in both accuracy and token efficiency.

What carries the argument

The central mechanism is the DynImg composite itself: four non-keyframes are resized and concatenated in temporal order below a high-resolution keyframe, and the whole image is fed through the SigLIP vision transformer. Because the encoder's self-attention is long-range, keyframe patches can attend to visually similar patches in the prompt strip; the paper's claim is that this lets the small frames reveal which regions are moving and increases attention to those fine-grained keyframe regions during feature extraction. The second piece is a 4D rotary position embedding that assigns each visual token coordinates in height, width, temporal, and sequence dimensions, with the keyframe's temporal coordinate set to zero and the non-keyframes getting symmetric positive and negative offsets. The rotation angle is a weighted sum $x\cdot\theta = x_h\theta_h + x_w\theta_w + x_t\theta_t + x_s\theta_s$, with $\theta_h$, $\theta_w$, and $\theta_t$ trainable and initialized to zero so the pretrained 1D sequence behavior dominates at the start of training.

What would settle it

Keep the DynImg layout identical but replace each non-keyframe with a blurred or temporally shuffled copy of the keyframe, or with frames from an unrelated video, matching resolution and position; the claimed temporal-prompt mechanism predicts a clear accuracy drop on moving-object MVBench categories, whereas the simpler explanation that the gain comes only from feeding more pixels predicts little change.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a video can be collapsed into a single high-resolution keyframe augmented with a strip of four resized non-keyframes, and that this compact composite, with a 4D rotary position embedding, lets a pretrained image encoder perform spatio-temporal interaction during visual feature extraction instead of after it. The non-keyframes are not extra evidence for the language model; they are prompts that reweight attention inside the vision transformer toward regions that moved. The paper reports that this representation outperforms prior video-LLM methods by roughly 2 percent on four open-ended video QA datasets, raises MVBench average accuracy to 55.8 percent from 46.6 percent for the 16-frame PLLaVA baseline, and produces its largest gains on the multiple-choice categories that explicitly test movement, such as Moving Direction, Moving Count, and Moving Attribute.

Load-bearing premise

The load-bearing premise is that when four small, heavily downscaled non-keyframes are pasted below a high-resolution keyframe, the visual encoder's self-attention actually uses those small frames to locate fast-moving regions and shifts extra attention to those regions in the keyframe; the paper states this mechanism but does not show attention maps to verify it.

Editorial extensions

If this is right

  • Four DynImg inputs, totaling 576 visual tokens, match or beat a 16-frame input of 2304 tokens on MSVD, MSRVTT, ActivityNet, TGIF, and the Video-ChatGPT benchmark.
  • Moving-sensitive MVBench tasks improve sharply: Moving Direction +21.0, Moving Count +15.0, and Moving Attribute +26.5 relative to the compared prior method.
  • Applying temporal prompts before the encoder improves MSVD accuracy by about 2.4 points over the no-prompt baseline, whereas fusing the same frames after the encoder gives only a small gain, supporting the pre-encoder interaction design.
  • The 4D trainable rotary embedding outperforms 1D and 3D coordinate versions, indicating that the language model needs explicit spatio-temporal layout information for composite inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the attention-shifting mechanism is real, the same base-plus-prompt composition could apply to other vision encoders and to tasks that depend on locating change, such as temporal action localization or video captioning.
  • The paper does not ablate the content of the prompt frames; a natural extension is synthetic videos with controlled motion to test whether gains come specifically from correspondence between keyframe and non-keyframe motion.
  • The 4D rotary embedding idea is not video-specific: any structured composite input, such as multi-panel diagrams or image grids, could use dimension-specific trainable angles to tell the LLM which panel a token came from.
  • Because gains concentrate on motion-sensitive multiple-choice categories, a practical extension is adaptive selection of which I-frames and nearby frames form each DynImg rather than uniform sampling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DynImg, a video representation in which each keyframe is concatenated with four downscaled non-keyframes ('temporal prompts') into a single image before visual encoding, together with a 4D rotary position embedding that assigns height, width, temporal, and sequence coordinates to the resulting tokens. The method is evaluated on open-ended video QA benchmarks (MSVD, MSRVTT, ActivityNet, TGIF, and the Video-ChatGPT benchmark) and on MVBench, reporting higher accuracy and lower token count (576 tokens) than prior methods. The central claim is that the temporal prompts make the visual encoder attend to fast-moving regions during feature extraction, and that this is responsible for the reported improvements, which the paper summarizes as surpassing state-of-the-art methods by approximately 2%.

Significance. If the mechanism were verified, DynImg would be a practically interesting contribution: it couples spatial and temporal processing inside the visual encoder, reduces visual tokens by a factor of four relative to PLLaVA's 2304 tokens, and shows large gains in moving-sensitive MVBench categories (e.g., Moving Attribute +26.5 and Moving Count +15.0). The ablations on MSVD show a plausible ordering: temporal prompts before the encoder improve over after-encoder fusion, and adding 4D-RoPE improves further. However, the mechanism is not directly evidenced, the headline numerical claim is numerically overstated, and the lack of released code/checkpoints and multi-seed results limits reproducibility. The paper should be revised to supply the missing controls and corrected claims.

major comments (4)
  1. [Abstract; Section 4.2; Table 1] The abstract and Section 4.2 state that DynImg 'surpasses the state-of-the-art methods by approximately 2%' on MSVD, MSRVTT, TGIF, and ActivityNet, but the paper's own Table 1 does not support this claim. Taking the best prior accuracy per dataset, the gains are +2.0 on MSVD (76.6 to 78.6), +0.3 on MSRVTT (63.8 to 64.1), +0.9 on ActivityNet (57.0 to 57.9), and +0.0 on TGIF (77.5 to 77.5), for a mean gain of about +0.8. The claim should be revised, or DynImg should be compared against a reimplementation of the best prior method using the same encoder (SigLIP) and LLM (Qwen2.5-7B).
  2. [Section 3.1; Table 3] The ablation used to attribute the improvement to the temporal-prompt self-attention mechanism is confounded. In Table 3, the 'After Encoder' condition feeds non-keyframes and keyframes at the same resolution and then downsamples and concatenates at the feature level, whereas the 'Before Encoder' DynImg condition downscales the non-keyframes and concatenates them at the pixel level. Because fusion stage and prompt-frame resolution change simultaneously, the +2.4% gap cannot be uniquely assigned to the proposed in-encoder attention mechanism. A controlled comparison holding resolution fixed while varying fusion stage (or vice versa), together with attention maps or patch-correspondence statistics from the SigLIP encoder, is needed to support the central claim of Section 3.1.
  3. [Section 4.1; Tables 4 and 5] The number of non-keyframes per DynImg (4) and the number of DynImgs per video (4) are selected by ablations on MSVD only, and these same values are then used for all five open-ended benchmarks and MVBench. This is a selection-pressure concern: the cross-benchmark results are not independent confirmations of the design choices, and the reported margins on MSRVTT and TGIF are small. The authors should evaluate the chosen configuration on a held-out validation split, or report per-dataset ablations, to show that the hyperparameters are not overfit to a single benchmark.
  4. [Section 4.2; Tables 1 and 2] The paper does not report the number of training runs, random seeds, or variance for any of the reported accuracies. Given that the headline gains over the best prior methods are small on several datasets (+0.3 on MSRVTT, +0.0 on TGIF), a single run is insufficient to establish superiority. Please report multiple seeds with mean and standard deviation, or otherwise provide statistical support, and release code and checkpoints to make the comparisons reproducible.
minor comments (6)
  1. [Section 3.2; Figure 3] The text 'Height, Weight, Temporal, and Sequence' should read 'Height, Width, Temporal, and Sequence'.
  2. [Figure 2] The attention map is described as belonging to a patch in the keyframe, but it appears to be a schematic illustration; label it as such to avoid implying an empirical visualization.
  3. [Table 1] Several baselines use different visual encoders and LLMs (for example, IG-VLM uses GPT-4V, and PLLaVA uses a different 7B LLM), so the cross-method 'state-of-the-art' comparison is not controlled; a same-backbone comparison would strengthen the claim.
  4. [Section 4.1] The description of MPEG-4 keyframe selection is vague; please specify the exact I-frame extraction procedure and how 'randomly select two preceding I-frames' handles videos with fewer than two I-frames.
  5. [Table 2] The token count of 576 for DynImg should be derived explicitly from the keyframe resolution, prompt resolution, patch size, and projection pooling shape.
  6. [Equation (2)] Equation (2) is introduced as a weighted sum, but the learnable θ_h, θ_w, θ_t are treated as scalar rates in the ablation; clarify whether they are shared across all feature dimensions or are per-dimension vectors.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; DynImg's claims rest on empirical ablations and benchmarks rather than on inputs defined to equal outputs.

full rationale

The paper's derivation chain is empirical. DynImg is constructed by concatenating downsampled non-keyframes beneath a keyframe and applying a 4D RoPE; the claimed benefit (better fine-grained spatiotemporal attention) is tested by ablations and benchmark comparisons. No equation defines the output in terms of the input, no fitted parameter is renamed as a prediction, and no load-bearing premise is justified solely by self-citation. The closest concern is that the Table 3 before/after-encoder comparison changes fusion stage, prompt-frame resolution, and token geometry simultaneously, so the +2.4% gap cannot be uniquely attributed to the proposed temporal-prompt attention mechanism. That is an experimental attribution problem, not a circularity: the conclusion does not reduce to an input by construction. Hyperparameters such as the number of non-keyframes (4) and the number of DynImgs (4) are selected via MSVD ablations and then used in the final benchmark runs, which is a selection-pressure concern rather than a prediction forced by the method's definition. Self-citations in the reference list are not load-bearing for the central claim. The paper therefore exhibits no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by modeling and hyperparameter choices: N-frame count, DynImg count, pooling shape, and learned theta values are all chosen empirically, and the central mechanism relies on two unproven assumptions about how the encoder uses the prompt strips. No new physical entities are introduced.

free parameters (5)
  • Number of non-keyframes per DynImg = 4
    Selected by ablation on MSVD (Table 4); N=4 gives the highest accuracy, and smaller or larger N performs worse.
  • Number of DynImgs per video = 4
    Selected by ablation on MSVD (Table 5); 4 DynImgs balances temporal coverage before performance plateaus or declines.
  • 4D-RoPE learnable angle rates theta_h, theta_w, theta_t = Learned during training, initialized to 0
    Section 3.2 Eq. 2; ablations in Table 6 show trainable rates outperform fixed sinusoidal rates.
  • Projection pooling shape = (16, 12, 12)
    Taken from PLLaVA [49]; determines the visual token count fed to the LLM and is a free design choice.
  • Resizing and placement geometry of N-frames = Unspecified; constrained to patch multiples
    Section 3.1 says N-frames are resized and concatenated so patch extraction does not cross boundaries, but exact pixel sizes are not reported and layout changes with N (Table 4 note).
assumptions (5)
  • domain assumption Evenly sampled MPEG-4 I-frames are sufficiently informative to serve as keyframes for video QA.
    Section 3.1 selects four I-frames as keyframes without checking whether I-frame positions align with semantically important moments.
  • domain assumption Heavily downscaled non-keyframes retain enough motion information to act as effective temporal prompts.
    Section 3.1 claims resizing does not significantly impact motion-awareness; only indirect accuracy ablations support this.
  • domain assumption SigLIP self-attention will form correspondences between keyframe patches and matching prompt patches and use them to emphasize moving regions.
    Section 3.1 describes this mechanism, but no attention evidence is provided.
  • ad hoc to paper A 4D RoPE computed as a weighted sum of coordinate angles is a valid positional encoding and can be learned without destroying pretrained LLM behavior.
    Section 3.2 Eq. 2 proposes this design; only ablation results justify it, not theory.
  • domain assumption The PLLaVA training recipe and data mixture transfer to DynImg with SigLIP and Qwen2.5.
    Section 4.1 states 'The training recipe of DynImg follows PLLaVA' without providing hyperparameters or validating transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding." pith.science (2026). https://pith.science/paper/5G2VZEBP

@misc{pith2026250715569,
  author       = {Pith},
  title        = {Pith review of: DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5G2VZEBP}},
  note         = {Machine review of arXiv:2507.15569}
}
read the original abstract

In recent years, the introduction of Multi-modal Large Language Models (MLLMs) into video understanding tasks has become increasingly prevalent. However, how to effectively integrate temporal information remains a critical research focus. Traditional approaches treat spatial and temporal information separately. Due to issues like motion blur, it is challenging to accurately represent the spatial information of rapidly moving objects. This can lead to temporally important regions being underemphasized during spatial feature extraction, which in turn hinders accurate spatio-temporal interaction and video understanding. To address this limitation, we propose an innovative video representation method called Dynamic-Image (DynImg). Specifically, we introduce a set of non-key frames as temporal prompts to highlight the spatial areas containing fast-moving objects. During the process of visual feature extraction, these prompts guide the model to pay additional attention to the fine-grained spatial features corresponding to these regions. Moreover, to maintain the correct sequence for DynImg, we employ a corresponding 4D video Rotary Position Embedding. This retains both the temporal and spatial adjacency of DynImg, helping MLLM understand the spatio-temporal order within this combined format. Experimental evaluations reveal that DynImg surpasses the state-of-the-art methods by approximately 2% across multiple video understanding benchmarks, proving the effectiveness of our temporal prompts in enhancing video comprehension.

Figures

Figures reproduced from arXiv: 2507.15569 by the authors.

Figure 1
Figure 1. Comparison between the previous methods for video understanding and DynImg. As for the structure comparison in the left (1), [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of DynImg. Videos are decomposed into keyframes and non-keyframes. Several non-keyframes serve as [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Schematic diagram of coordinates in 4D position em [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 32 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 2022. 2

    2022

  2. [2]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 6836–6846,

  3. [3]

    Exploring visual prompts for adapting large- scale models

    Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large- scale models. arXiv preprint arXiv:2203.17274 , 2022. 2, 3

    arXiv 2022

  4. [4]

    Relevant intrinsic feature enhancement network for few-shot semantic segmentation

    Xiaoyi Bao, Jie Qin, Siyang Sun, Xingang Wang, and Yun Zheng. Relevant intrinsic feature enhancement network for few-shot semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 765–773, 2024. 3

    work page 2024

  5. [5]

    Cores: Orchestrating the dance of reasoning and seg- mentation

    Xiaoyi Bao, Siyang Sun, Shuailei Ma, Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, and Xingang Wang. Cores: Orchestrating the dance of reasoning and seg- mentation. In European Conference on Computer Vision , pages 187–204. Springer, 2024. 2

    work page 2024

  6. [6]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 3

    2021

  7. [7]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970, 2015. 6

    work page 2015

  8. [8]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 3

    work page 2017

Show all 57 references
  1. [9]

    Deep temporal linear encoding networks

    Ali Diba, Vivek Sharma, and Luc Van Gool. Deep temporal linear encoding networks. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, pages 2329–2338, 2017. 3

    2017

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  3. [11]

    Convolutional two-stream network fusion for video action recognition

    Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1933–1941,

    1933

  4. [12]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 3

    2019

  5. [13]

    A systematic survey of prompt engineer- ing on vision-language foundation models

    Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, V olker Tresp, and Philip Torr. A systematic survey of prompt engineer- ing on vision-language foundation models. arXiv preprint arXiv:2307.12980, 2023. 2, 3

    2023 arXiv

  6. [14]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In European Conference on Computer Vision, pages 202–218. Springer, 2024. 6

    2024

  7. [15]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 2, 3

    2022

  8. [16]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  9. [17]

    Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization

    Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03161, 2024. 3

    2024

  10. [18]

    Large-scale video classification with convolutional neural networks

    Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Pro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014. 3

    2014

  11. [19]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 3

    2023

  12. [20]

    An image grid can be worth a video: Zero- shot video question answering using a vlm

    Wonkyun Kim, Changin Choi, Wonseok Lee, and Won- jong Rhee. An image grid can be worth a video: Zero- shot video question answering using a vlm. arXiv preprint arXiv:2403.18406, 2024. 3, 6, 7

    2024 arXiv

  13. [21]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692,

    arXiv

  14. [22]

    Deep local video feature for action recog- nition

    Zhenzhong Lan, Yi Zhu, Alexander G Hauptmann, and Shawn Newsam. Deep local video feature for action recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1–7, 2017. 3

    2017

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023. 2

    2023 arXiv

  16. [24]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 3, 6, 7

    2023 arXiv

  17. [25]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  18. [26]

    Tgif: A new dataset and benchmark on animated gif description

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016. 6

    2016

  19. [27]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 2, 3, 6

    2025

  20. [28]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 2, 3, 6, 7

    2023 arXiv

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  22. [30]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in nat- ural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hi- roaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in nat- ural language processing. ACM Computing Surveys, 55(9): 1–35, 2023. 3

    2023

  23. [31]

    St-llm: Large language models are effective tem- poral learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective tem- poral learners. In European Conference on Computer Vision, pages 1–18. Springer, 2024. 6, 7

    2024

  24. [32]

    Ppllava: Varied video se- quence understanding with prompt guidance

    Ruyang Liu, Haoran Tang, Haibo Liu, Yixiao Ge, Ying Shan, Chen Li, and Jiankun Yang. Ppllava: Varied video se- quence understanding with prompt guidance. arXiv preprint arXiv:2411.02327, 2024. 3

    2024 arXiv

  25. [33]

    Hybrid-level instruction injection for video token com- pression in multi-modal large language models

    Zhihang Liu, Chen-Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, and Hongtao Xie. Hybrid-level instruction injection for video token com- pression in multi-modal large language models. In Proceed- ings of the Computer Vision and Pattern Recognition ...

    2025

  26. [34]

    Vista-llama: Reliable video narrator via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narrator via equal distance to visual tokens. arXiv preprint arXiv:2312.08870,

    arXiv

  27. [35]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 2, 3, 6, 7

    2024

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Mul- titask vision-language prompt tuning

    Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, and Trevor Darrell. Mul- titask vision-language prompt tuning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5656–5667, 2024. 2, 3

    2024

  30. [38]

    What does clip know about a red circle? vi- sual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? vi- sual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11987–11997, 2023. 2, 3

    2023

  31. [39]

    Two-stream con- volutional networks for action recognition in videos

    Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. Ad- vances in neural information processing systems , 27, 2014. 3

    2014

  32. [40]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  33. [41]

    Ufo: A unified approach to fine-grained visual perception via open- ended language interface

    Hao Tang, Chenwei Xie, Haiyang Wang, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Ufo: A unified approach to fine-grained visual perception via open- ended language interface. arXiv preprint arXiv:2503.01342,

  34. [42]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 6

    2024

  35. [43]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InProceedings of the IEEE inter- national conference on computer vision , pages 4489–4497,

  36. [44]

    A closer look at spatiotemporal convolutions for action recognition

    Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recogni- tion, pages 6450–6459, 2018. 3

    2018

  37. [45]

    Action recogni- tion with trajectory-pooled deep-convolutional descriptors

    Limin Wang, Yu Qiao, and Xiaoou Tang. Action recogni- tion with trajectory-pooled deep-convolutional descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4305–4314, 2015. 3

    2015

  38. [46]

    Temporal segment net- works: Towards good practices for deep action recognition

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment net- works: Towards good practices for deep action recognition. In European conference on computer vision , pages 20–36. Springer, 2016. 3

    2016

  39. [47]

    Deep learning for video classification and captioning

    Zuxuan Wu, Ting Yao, Yanwei Fu, and Yu-Gang Jiang. Deep learning for video classification and captioning. In Frontiers of multimedia research, pages 3–29. 2017. 6

    2017

  40. [48]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288–5296, 2016. 6

    2016

  41. [49]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 3, 6, 7

    2024 arXiv

  42. [50]

    Beyond short snippets: Deep networks for video classification

    Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vi- jayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4694–4702,

  43. [51]

    Unified vision and language prompt learning

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022. 3

    2022 arXiv

  44. [52]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. 2, 6

    2023

  45. [53]

    Real-time action recognition with enhanced motion vector cnns

    Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2718–2726,

  46. [54]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2, 6

    2023 arXiv

  47. [55]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16816–16825,

  48. [56]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  49. [57]

    Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els. arXiv:2304.10592, 2023. 2

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.