Pith. sign in

REVIEW 4 major objections 7 minor 45 references

Clapper: Compact Learning and Video Representation in VLMs

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Clapper compresses video tokens 13x and keeps QA accuracy across VideoMME, MLVU, and TempCompass.

desk verdict Useful, honest efficiency work on video VLMs, but the headline no-compromise 13x claim is only backed by 1/10-scale ablations; full-data results are competitive but not controlled. read the letter →

arxiv 2505.15529 v1 pith:VY5URRDW submitted 2025-05-21 cs.CV

Lingyu Kong , Hongzhi Zhang , Jingyuan Zhang , Jianzhao Huang , Kunze Li , Qi Wang , Fuzheng Zhang This is my paper
classification cs.CV
keywords videounderstandingvision-languagemodeltokencompressiontemporalmodelingTimePerceiverslow-fastrepresentationquestionansweringefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clapper claims that a video language model can compress the visual tokens produced per frame by about 13x—from 784 tokens per frame out of the image encoder to 61—without paying a quality penalty on multiple-choice video QA, and that this removes the usual trade-off where aggressive compression degrades long-video understanding. The paper argues that the right division of labor is a slow-fast representation: keep one high-resolution keyframe per four-frame segment for spatial detail, and let a small trainable cross-attention module call TimePerceiver summarise the temporal changes across those four frames into 49 tokens. With that design Clapper reports 62.0% on VideoMME, 69.8% on MLVU, and 67.4% on TempCompass, all under 6,000 visual tokens per video, and it documents that current VLMs degrade when forced below a quarter of their original token count. If true, this makes long-video QA practical at much lower compute and gives the field a way to compare models under a fixed visual token budget.

What carries the argument

The load-bearing mechanism is TimePerceiver, a trainable module that takes 2–4 frame features from the vision encoder and emits a fixed 49 tokens per segment. Its input queries come from aggressive spatial (stride 4) then temporal average pooling, so the queries encode an averaged summary of what the segment looks like; those queries then cross-attend to the full-resolution flattened frame features, with keys and values computed from the concatenation of the pooled summary and the full features. This lets the module spend its few output tokens on regions that actually change across frames, while the separately retained high-resolution keyframe carries static scene and object details. The combination moves compression from 4x (spatial pooling applied to every frame) to 13x while improving or holding accuracy on MVBench, TempCompass, and VideoMME.

What would settle it

Take a benchmark or a synthesized set of short clips where the correct answer depends on a small, brief, spatially localized change—a card flip, a counter changing, an object appearing in one frame—and compare Clapper at 61 tokens per frame against its 4x baseline at 196 tokens per frame on the same frames. If accuracy on those items falls well below the baseline while static-spatial items stay flat, the pooled prior has discarded the needed detail and the 13x claim does not hold for fine-grained temporal reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that a Perceiver-style module whose cross-attention queries are derived from a heavily pooled summary of four frame features preserves the information needed for video QA while cutting tokens by 13x. Concretely, each four-frame segment is represented by a 196-token keyframe (spatial pooling of the first frame with stride 2) plus a 49-token TimePerceiver output, so a segment costs 245 tokens rather than the 3136 tokens the encoder would emit for those four frames. The TimePerceiver builds its queries by spatial average pooling with stride 4 and then temporal average pooling over the four frames, then cross-attends to the full flattened features, with keys and values formed from the concatenation of that pooled prior and the full features. On six video QA benchmarks, the method is competitive with or better than models using many more tokens per frame, and it outperforms simpler 16x compressions (temporal pooling, spatial pooling, plain Perceiver) on the frame-sensitive VideoMME benchmark.

Load-bearing premise

The load-bearing assumption is that a summary of each four-frame segment produced by average-pooling in space and time is a good enough prior that cross-attention can recover the dynamic details the 49 tokens should carry; if that prior drops a detail a question needs, the 13x compression loses accuracy no matter how the cross-attention is trained.

Editorial extensions

If this is right

  • Under a fixed 2k or 6k visual-token budget on VideoMME, Clapper beats compared open models at the same budget, including on short and medium videos, so deployment can trade tokens for accuracy predictably.
  • Because average tokens per frame is independent of the number of frames, the same representation can extend to more frames and longer videos inside an LLM context; the paper's frame-sweep results show MLVU improving steadily as frames increase from 16 to 128.
  • The ablation results imply that compressing to 13x with TimePerceiver generalizes better than 16x pooling or a plain Perceiver, and that the two-stage training recipe (video-caption warm-up then instruction tuning) is needed for those gains, since direct fine-tuning hurts VideoMME.
  • Reporting both the number of frames and the average tokens per frame becomes a meaningful evaluation norm, because models with more tokens per frame are not directly comparable unless the total visual token budget is fixed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit: the 49-token temporal summary is only as good as the pooled prior that seeds its queries, so questions whose answer lives in a single brief high-frequency event at small spatial scale may be the first to break as compression is pushed further.
  • A natural extension is to make the keyframe-to-temporal-token ratio adaptive—allocating more keyframes in high-motion segments and more temporal tokens in static segments—which TimePerceiver's fixed 49-token budget currently cannot do.
  • The paper's fixed-token-budget evaluation could be adopted as a reporting protocol for other video VLMs; this editorial suggestion would separate genuine quality gains from gains that merely come from feeding more tokens to the model.
  • A testable extension is to push Clapper beyond 96 frames by training on longer sampled sequences or by adding length-extrapolation methods, which the paper identifies as a limitation and as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Clapper, a video VLM that reuses a SigLIP image encoder and adds a TimePerceiver module to compress each 4-frame segment into 49 temporal tokens, while retaining a spatially pooled 196-token keyframe. The resulting representation uses 245 tokens per segment, or about 61 tokens per frame, a 13x reduction from the encoder's 784 tokens per frame. Training is done in two stages on video-caption and instruction data, starting from LLaVA-OneVision-SI with a Qwen2 backbone. The paper reports results on TempCompass, MVBench, PerceptionTest, LongVideoBench, MLVU, and VideoMME, including comparisons under fixed visual-token budgets of 2k and 6k tokens. Ablations compare temporal pooling, spatial pooling, a standard Perceiver, and TimePerceiver, as well as the two-stage training design. The paper also includes a Limitations section noting the lack of length extrapolation beyond 96 frames and the absence of RLHF alignment.

Significance. If the full-scale results are representative, Clapper would demonstrate that strong video QA performance is achievable with roughly one thirteenth of the encoder's per-frame tokens, which is practically valuable for long-video VLM deployment. The paper's compression ratio is an architectural consequence rather than a fitted hyperparameter, and the explicit reporting of token counts and fixed-token-budget evaluations is a useful contribution to fairer benchmarking. The ablation framework is sensible and includes a comparison against a standard Perceiver. However, the headline 'without compromising QA accuracy' claim is currently supported only by 1/10-scale controlled experiments; at full scale, the comparisons are confounded by model, data, and frame-count differences. The claimed motivating degradation analysis is also absent. These issues make the contribution promising but not yet fully established.

major comments (4)
  1. [§4.3 (Tables 3–4)] The only controlled tests of the central '13x compression without compromising QA accuracy' claim are performed at one-tenth of the Stage 2 training data, as stated in the first sentence of §4.3. The full-data results in Tables 1 and 2 compare Clapper with externally reported models that differ in architecture, training data, and evaluation frames, so they cannot isolate the effect of token compression on accuracy. To make the headline claim stand, please add a full-data controlled comparison with at least a 4x spatial-pooling baseline and a standard Perceiver module trained on the same data and evaluated with the same frames and harness, or explicitly re-scope the claim to the reduced-scale ablations.
  2. [Abstract and §1] The paper asserts that 'most existing VLMs suffer severe performance degradation in long video understanding tasks when compressing visual tokens below a quarter of their original visual tokens,' but no empirical analysis supporting this statement appears anywhere in the manuscript. This claim motivates the entire approach and is presented as a finding, yet there is no figure, table, or experiment showing the degradation. Please add the supporting analysis, or if it is from prior work, cite it explicitly and describe it.
  3. [§4.2, Table 1] Comparisons in Table 1 mix numbers cited from original papers with numbers obtained by integrating models into LMMs-Eval, and the frame counts differ across models (e.g., Clapper is evaluated with 96 frames while several baselines use 32 or 64). Because Video QA accuracy depends on both frame count and evaluation harness, the reported differences cannot be attributed to token compression alone. Please mark each entry as cited or re-evaluated, and provide at least one matched comparison at the same frame count and token budget for the systems that are claimed to be outperformed.
  4. [§3.1, Figure 2] TimePerceiver constructs its queries by spatial average pooling with stride 4 followed by temporal averaging over the four frames. This design presumes that a single pooled summary of the segment is a sufficient prior for cross-attending to the full features. The paper does not ablate this query-construction choice (e.g., comparing it with learned queries or temporal-stride variants), so it is unclear whether the reported gains come from the cross-attention mechanism or from this specific pooling prior. Adding such an ablation would strengthen the causal interpretation of the module's contribution.
minor comments (7)
  1. [Abstract] The phrase 'we achieves' should be corrected to 'we achieve'.
  2. [§4.1] The word 'TimePerveiver' is a typo for 'TimePerceiver' and should be corrected.
  3. [Global] The text contains inconsistent spacing artifacts such as 'LLaV A' and 'LLaV A-Video'; these should be unified as 'LLaVA' and 'LLaVA-Video' throughout.
  4. [Figure 3] The axis labels and legend in Figure 3 are very small and difficult to read at print size; consider enlarging the text and adding value labels to the plotted curves.
  5. [§4.2, Table 2] The abbreviation 'wo' in the Table 2 caption is not defined; please spell out 'without subtitles' or define the abbreviation explicitly.
  6. [§3.1] The sentence about the last video segment says that a 2–3 frame segment 'still occupies 245 tokens,' which means the per-frame token average is not exactly 61 for all videos; please clarify how the reported 61 tokens/frame figure is computed.
  7. [§6] The Limitations section states that performance 'may degrade as the video length increases' for videos longer than 5 minutes; this should be reconciled with the abstract's claim about long-video understanding.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: PerceptionTest and NExT-QA appear in the Stage 2 training recipe and are then reported as evaluation benchmarks; the main 13x compression claim itself still rests on clean external benchmarks.

  1. fitted input called prediction [Section 3.2 (Training Recipe) vs Section 4.2/Table 1 and Figure 3 (Evaluation)]
    "The training data includes LLaVA-Video-178K (Zhang et al., 2024f), ActivityNet-QA (Yu et al., 2019), NExT-QA (Xiao et al., 2021), PerceptionTest (Patraucean et al., 2024), and LLaVA-Hound-255K (Zhang et al., 2024d), which together provide a total of 1.6 million video-language samples."

    PerceptionTest and NExT-QA are listed as training data in Section 3.2, yet the paper reports PerceptionTest accuracy in Table 1 and Figure 3 and NExT-QA accuracy in Figure 3. Those reported scores therefore measure performance on a distribution the model was explicitly fitted to, so they are not independent predictions of generalization. Unless a non-evaluation split is explicitly stated, the PerceptionTest and NExT-QA results cannot support the paper's 'without compromising QA accuracy' claim. The headline benchmarks VideoMME, MLVU, and TempCompass are not in the training list, so the central 13x compression claim retains independent content.

full rationale

The 13x per-frame compression ratio is an architectural constant, not a fitted outcome: Section 3.1 specifies 245 tokens per 4-frame segment (196 key-frame tokens plus 49 TimePerceiver tokens) versus 784 SigLIP tokens per frame, matching the compression-ratio definition in Eq. (1). TimePerceiver is trained on generic video-caption and instruction data, and the three benchmark numbers emphasized in the abstract (VideoMME, MLVU, TempCompass) are not in the Stage 2 training list, so those evaluations are external to the method. The one circular element is the use of PerceptionTest and NExT-QA in both training and evaluation, which makes those specific reported scores measures of benchmark fitting rather than transfer prediction. Section 4.3 explicitly states that the ablation models were trained on one-tenth of the Stage 2 data (~160k samples), and Section 6 notes the model is limited to 96 frames without length extrapolation and without RLHF; these are evidentiary caveats affecting generalization strength, but they are not circularity. The self-citations in the paper (Wei et al. 2025; Chen et al. 2024a) appear only as related-work mentions for high-resolution image processing and are not load-bearing for Clapper's design or results. Overall, the central compression claim remains independently testable, but the PerceptionTest and NExT-QA benchmark results are partially circular, giving a score of 4.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on typical domain assumptions about the sufficiency of pooled image features for video QA and on the reliability of the evaluation benchmarks. The chosen hyperparameters (49 output tokens, pooling strides) are architectural decisions, not fitted to benchmark results. No new physical entities are proposed.

free parameters (5)
  • TimePerceiver output tokens per segment = 49
    The module outputs 49 tokens per 4-frame segment; chosen by hand, not fitted to benchmark results.
  • Temporal average pooling stride in TimePerceiver = 4
    Applied to spatial features before computing queries; architectural hyperparameter.
  • Spatial pooling stride for key frames = 2
    Applied to key frame embeddings, reducing 784 to 196 tokens; design choice.
  • Segment length = 4 frames
    Each video segment consists of 4 consecutive frames; design choice.
  • Frame sampling rate = 1 fps (max 96 frames)
    Training sampling strategy; hyperparameter.
assumptions (4)
  • domain assumption SigLIP image encoder features retain sufficient spatial detail after 2x spatial pooling for key frame representation
    The key frame uses a 196-token representation; the paper assumes this preserves details for QA.
  • domain assumption Average pooling along spatial and temporal dimensions provides a good query prior for cross-attending to dynamic changes
    TimePerceiver queries are derived from pooled features; this assumes the pooled summary captures salient content.
  • domain assumption The LMMs-Eval framework and the selected benchmarks provide reliable and comparable measurements
    Evaluation standard, not verified in this paper.
  • domain assumption Training on publicly available video-caption and instruction datasets is sufficient to learn temporal reasoning without benchmark-specific tuning
    The model is trained on generic data, not on the evaluation benchmarks.
invented entities (1)
  • TimePerceiver module
    purpose: Learns temporal dynamics within each 4-frame segment and outputs 49 tokens, enabling 13x token compression
    A new neural module proposed in this paper. Its effectiveness is demonstrated only in the paper's own experiments; no external validation or theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clapper: Compact Learning and Video Representation in VLMs." pith.science (2026). https://pith.science/paper/VY5URRDW

@misc{pith2026250515529,
  author       = {Pith},
  title        = {Pith review of: Clapper: Compact Learning and Video Representation in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VY5URRDW}},
  note         = {Machine review of arXiv:2505.15529}
}
read the original abstract

Current vision-language models (VLMs) have demonstrated remarkable capabilities across diverse video understanding applications. Designing VLMs for video inputs requires effectively modeling the temporal dimension (i.e. capturing dependencies across frames) and balancing the processing of short and long videos. Specifically, short videos demand preservation of fine-grained details, whereas long videos require strategic compression of visual information to handle extensive temporal contexts efficiently. However, our empirical analysis reveals a critical limitation: most existing VLMs suffer severe performance degradation in long video understanding tasks when compressing visual tokens below a quarter of their original visual tokens. To enable more effective modeling of both short and long video inputs, we propose Clapper, a method that utilizes a slow-fast strategy for video representation and introduces a novel module named TimePerceiver for efficient temporal-spatial encoding within existing VLM backbones. By using our method, we achieves 13x compression of visual tokens per frame (averaging 61 tokens/frame) without compromising QA accuracy. In our experiments, Clapper achieves 62.0% on VideoMME, 69.8% on MLVU, and 67.4% on TempCompass, all with fewer than 6,000 visual tokens per video. The code will be publicly available on the homepage.

Figures

Figures reproduced from arXiv: 2505.15529 by the authors.

Figure 1
Figure 1. Architecture of Clapper. The model consists of a vision encoder, a TimePerceiver module, an MLP layer, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The TimePerceiver module processes 2-4 frames to generate a fixed number of temporal embed￾ding outputs (49 in this paper). In the second stage, we employ video instruction￾tuning data to train the model’s instruction￾following capabilities. The training data in￾cludes LLaVA-Video-178K (Zhang et al., 2024f), ActivityNet-QA (Yu et al., 2019), NExT-QA (Xiao et al., 2021), PerceptionTest (Patraucean et al., 2024), and … view at source ↗
Figure 3
Figure 3. Performance of Clapper under different frames on the eight video QA benchmarks. We also conducted a qualitative analysis of the model performance. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of video captioning results using Clapper and others. Key points are displayed in bold. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 10 canonical work pages

  1. [1]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar \' e n Simonyan

    Jean - Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Bin...

    work page 2022

  2. [2]

    Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2024 a . Onechart: Purify the chart structural extraction via one auxiliary token. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 147--155

    work page 2024

  3. [3]

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024 b . Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18407--18418

    work page 2024

  4. [4]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024 c . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    arXiv 2024

  5. [5]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu...

    arXiv 2024

  6. [6]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. CoRR, abs/2406.07476

    arXiv 2024

  7. [7]

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nvlm: Open frontier-class multimodal llms. arXiv preprint

    work page 2024

  8. [8]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075

    arXiv 2024

Show all 45 references
  1. [9]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  2. [10]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. https://proceedings.mlr.press/v202/li23q.html BLIP -2: Bootstrapping language-image pre-training with frozen image encoders and large language models . In Proceedings of the 40th International Conference on Machine L...

    2023

  3. [11]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. 2024 b . Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR , pages 22195--22206. IEEE

    2024

  4. [12]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024 c . Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323--340. Springer

    2024

  5. [13]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  6. [14]

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024 a . Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542

    2024 arXiv

  7. [15]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024 b . Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476

    2024 arXiv

  8. [16]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024

  9. [17]

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. 2024. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371

    2024 arXiv

  10. [18]

    OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774

    2023 arXiv

  11. [19]

    OpenAI. 2024. Gpt-4o. https://openai.com/index/hello-gpt-4o/

    2024

  12. [20]

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. 2024. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processin...

    2024

  13. [21]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  14. [22]

    Lillicrap, Jean - Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean - Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, M...

    2024 arXiv

  15. [23]

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. 2024. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434

    2024 arXiv

  16. [24]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  17. [25]

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. 2024 b . Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture. CoRR, abs/2409.02889

    2024

  18. [26]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. 2024 c . Internvideo2: Scaling video foundation models for multimodal video understanding. In ECCV

    2024

  19. [27]

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2025. Vary: Scaling up the vision vocabulary for large vision-language model. In Computer Vision -- ECCV 2024, pages 408--424, Cham. Springer Nature Switzerland

    2025

  20. [28]

    Haoning Wu, DONGXU LI, Bei Chen, and Junnan Li. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/329ad516cf7a6ac306f29882e9c77558-Paper-Datasets_and_Benchmarks_Track.pdf Longvideobench: A benchmark for long-context interleaved video-language understanding . In ...

    2024

  21. [29]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777--9786

    2021

  22. [30]

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024 a . https://arxiv.org/abs/2404.16994 Pllava : Parameter-free llava extension from images to videos for video dense captioning . Preprint, arXiv:2404.16994

    2024 arXiv

  23. [31]

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. 2024 b . Slowfast-llava: A strong training-free baseline for video large language models. arXiv:2407.15841

    2024 arXiv

  24. [32]

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. 2024. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188

    2024 arXiv

  25. [33]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  26. [34]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800

    2024 arXiv

  27. [35]

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. 2023. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499

    2023 arXiv

  28. [36]

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127--9134

    2019

  29. [37]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975--11986

    2023

  30. [38]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.49 Video- LL a MA : An instruction-tuned audio-visual language model for video understanding . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Syst...

    2023 doi

  31. [39]

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. 2024 a . Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772

    2024 arXiv

  32. [40]

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  33. [41]

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024 c . https://arxiv.org/abs/2406.16852 Long context transfer from language to vision . arXiv preprint arXiv:2406.16852

    2024 arXiv

  34. [42]

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. 2024 d . Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258

    2024 arXiv

  35. [43]

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024 e . https://llava-vl.github.io/blog/2024-04-30-llava-next-video/ Llava-next: A strong zero-shot video understanding model

    2024

  36. [44]

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024 f . https://arxiv.org/abs/2410.02713 Video instruction tuning with synthetic data . Preprint, arXiv:2410.02713

    2024 arXiv

  37. [45]

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.