Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read With a group resampler, a 3.6B video model beats several 7B models on four benchmarks while training in one day.

desk verdict Useful empirical paper, but the headline 16-frame results use the image-level equivalent of the proposed resampler, so the central attribution is unsettled. read the letter →

arxiv 2501.15513 v2 pith:MOXSPI7I submitted 2025-01-26 cs.CV

Xingjian Zhang , Xi Weng , Yihao Yue , Zhaoxin Fan , Wenjun Wu , Lei Huang This is my paper · ORCID
classification cs.CV
keywords videounderstandinglargemultimodalmodelsgroupresamplervisualtokencompressiontemporalreasoningefficienttraininglightweight
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a lightweight large multimodal model with about 3.6 billion parameters can outperform several open-source 7B video understanding models on Video-MME, MVBench, MLVU, and LongVideoBench, while training in roughly one day on eight A100-40G GPUs. The key mechanism is a video-level group resampler that splits the concatenated visual token sequence and the learnable queries into M groups before cross-attention, keeping the total number of visual tokens fixed and small regardless of frame count. Compared with per-frame resampling, this design lets each query attend only to a short temporal chunk of the video, which the paper argues avoids attention redundancy and preserves temporal order information. If the claim holds, strong video understanding does not require a 7B backbone or large-scale training data: the released 3.6B model with 888K training samples is competitive with larger models on standard benchmarks.

What carries the argument

The video-level group resampler is a cross-attention connector placed between a vision encoder and a small language model. It first concatenates the per-frame visual features into one long sequence and applies a linear projection. Then it splits both the projected visual sequence and the learnable query set into M equal groups, runs cross-attention group-wise with each query group attending only to its corresponding visual chunk, and concatenates the outputs. This keeps the total visual token count fixed and controllable (for instance 512) even when the number of sampled frames grows, prevents the attention redundancy seen in naive video-level resampling, and, because each group spans several consecutive frames, gives the model a way to register temporal dependencies that per-frame resampling misses.

What would settle it

Evaluate the chosen 512-query/16-group model against the image-level baseline on a video QA benchmark not used during development; if the 2–4-frame-per-group configuration no longer beats per-image resampling, the reported advantage is an artifact of benchmark selection.

Watch

Extended reading notes

Core claim

The central claim is that the video-level group resampler—which splits the concatenated visual token sequence and the learnable queries into M groups before cross-attention—significantly reduces the number of visual tokens while preserving the efficiency of query learning and enhances the understanding of temporal information (Section 3.2, Equations 6–9). The paper shows that with the chosen configuration (512 total queries, 16 groups, each group covering about 2–4 frames), the model achieves higher accuracy than both image-level resampling and a naive video-level resampler that lets every query attend to the whole sequence. It further reports that the resulting 3.6B model surpasses several open-source 7B-parameter video LMMs on Video-MME, MVBench, MLVU, and LongVideoBench, using a two-stage training pipeline with a pre-training stage on 397K video-caption pairs and a fine-tuning stage on 491K dialogue samples.

Load-bearing premise

The group configuration (512 total queries, 16 groups) was chosen by maximizing average accuracy on the same four benchmarks later used for the headline comparison, and the paper assumes this selection does not materially overfit those test sets.

Editorial extensions

If this is right

  • A video model can process arbitrarily long videos with a fixed, small token budget instead of scaling tokens with frame count.
  • The same architecture, trained on roughly 900K samples, matches or exceeds several 7B open-source models on video QA benchmarks, so parameter count and data scale are not the only route to video understanding.
  • The group size is a tunable hyperparameter: the paper's ablation shows a sweet spot around 2–4 frames per group, with both too-few queries per group and image-level grouping hurting accuracy.
  • Training cost drops to about one day on eight A100-40G GPUs (25 hours for the group resampler configuration), making video LMM training feasible on modest clusters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The group-splitting idea could generalize to compressing the visual tokens within a single frame, not just across frames, which may help high-resolution image inputs that currently rely on image-level resampling.
  • Because the group configuration was tuned on the four benchmark suites used for the headline comparison, the reported margins should be re-verified on a held-out benchmark before treating them as general.
  • The fixed token budget makes the design a natural fit for streaming inference, where the model could process video in temporal chunks and accumulate a constant number of tokens over time.
  • A cheaper alternative—replacing group-cross-attention with per-chunk pooling or local attention—might achieve similar temporal coverage; the paper does not test this, but the analysis suggests the grouping itself, not the attention mechanism, is the active ingredient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces TinyLLaVA-Video, a ~3.6B-parameter video LMM built from Qwen2.5-3B and SigLIP. Its proposed video-level group resampler (Eqs. 6-9) splits the concatenated visual token sequence and the learnable queries into M groups before cross-attention, with the aim of reducing visual token count, avoiding query redundancy, and improving temporal comprehension. The authors train on about 888K video-text samples in roughly one day on 8 A100-40G GPUs and report that the resulting model outperforms several 7B open-source video LMMs on Video-MME, MVBench, MLVU, and LongVideoBench (Table 2). The paper also presents controlled ablations (Figs. 5, 8, 10, 11, 12; Table 9) intended to isolate the effect of the resampler and to justify the choice of group count and number of queries.

Significance. If fully supported, the paper would offer a useful, resource-efficient baseline for small video LMMs, and the controlled comparison of three resampling schemes is a reasonable way to study token-reduction mechanisms. The open code and weights, the modest training budget, and the detailed per-benchmark tables are concrete strengths. However, the central attribution of the headline gains to the video-level group resampler is not currently established: the 16-frame, 512-query configuration used in the main comparison is equivalent to the image-level baseline, while the genuinely cross-frame configurations at that budget perform worse on several benchmarks. The significance of the work is therefore conditional on a re-analysis that separates image-level-equivalent settings from true cross-frame grouping and that addresses the test-set selection issue.

major comments (3)
  1. [Section 3.2, Table 9, Fig. 12] In the 16-frame, 512-total-query configuration used for the 'TinyLLaVA-Video 3B 16' row of Table 2, the setting 'Group=16, Query-per-group=32' in Table 9 is identical to the image-level resampler of Eqs. (1)-(3), because with 16 frames and 16 groups each group receives exactly one frame's visual tokens; the paper's own Fig. 12 marks this case as '512-16-Group=16(I)'. The genuinely cross-frame configuration at the same budget, Group=8 (two frames per group), scores lower on Video-MME (average 45.83 vs 46.97), LongVideoBench (36.4 vs 42.4), and MLVU (51.0 vs 52.5). The 1fps row does use cross-frame grouping, but the paper does not clearly separate which headline comparisons rest on the group resampler versus the image-level fallback. Consequently, the abstract's central claim that the video-level group resampler yields the reported gains is not supported by the 16-frame results in Table 2.
  2. [Section 3.2, Fig. 12, Table 9] The text states that 'the model achieves the best performance when each group of queries is responsible for learning information from approximately 2 to 4 frames, as shown in Fig. 12.' This generalization is not supported by Table 9: for 16 uniformly sampled frames at 512 total queries, the best average accuracy is obtained with Group=16 (one frame per group), while Group=8 (two frames per group) is worse by about two points on average; the 2-4 frame sweet spot holds only at the 128 and 256 total-query budgets. The paper should either restrict the claim to lower query budgets or explain why the effect reverses at the budget used in the final model.
  3. [Section 3.2, Table 9, Figs. 11-12] The group count M and total query count are selected by maximizing average accuracy on the same four benchmarks (Video-MME, MVBench, MLVU, LongVideoBench) that are later used for the headline comparison against 7B models in Table 2 and Tables 5-8. No held-out validation set, cross-validation, or multiple-seed evaluation is reported, and the Table 2 caption uses the word 'significantly' without any statistical test. This selection-on-test-set procedure makes it difficult to determine how much of the reported advantage over 7B models is a genuine property of the resampler rather than a result of tuning to these specific benchmarks. Please report a validation-based selection protocol or a separate held-out comparison.
minor comments (6)
  1. [Abstract] The first sentence contains a grammatical error: 'Through Large Multimodal Models have achieved' should be 'Though Large Multimodal Models have achieved'.
  2. [Abstract] The sentence 'The code and weights is available' should read 'The code and weights are available'.
  3. [Fig. 3 caption] The phrase 'PE and Concat represent Position Encoding and Concatenate, respectively' should be 'Positional Encoding and Concatenation' for grammatical consistency.
  4. [Table 9] The table is difficult to read because the #Group and #Query-per-group columns are not clearly aligned with the rows in the rendered text; please format the table so that each row explicitly shows the group count and the number of queries per group.
  5. [Section 4.2, Tables 10-12] The phrase 'under the same settings' should state explicitly that the total number of queries is 512 and that the naive video-level resampler is used in those comparisons, so that readers can relate the architecture ablations to the main results.
  6. [Section 3.2, Fig. 5] The 'zeroing out' experiment measures robustness to query removal; the authors should clarify why retaining 25% of queries at roughly 95% performance indicates redundancy rather than, for example, the LLM's ability to compensate for missing tokens.

Circularity Check

2 steps flagged · score 6.0 of 10

At the 16-frame/512-query setting used for the headline comparison, the proposed video-level group resampler reduces to the image-level resampler by construction, and the group configuration is selected on the same benchmarks later reported as the main results.

  1. renaming known result [Section 3.2, Eq. (6)-(9), Fig. 12, Table 9, Table 2]
    "We also explore the optimal number of groups in the video-level group method, with the results shown in Fig. 11... The best results are achieved when each group contains 16 to 32 queries... Under the same total number of queries, it demonstrates superior performance and stronger temporal comprehension compared to the setting where each group of queries is responsible for only one image, i.e. the image-level method. The detailed experimental results are presented in Table 9 of the Appendix."

    When N=16 uniformly sampled frames and M=16 groups, Eq. (7) splits the concatenated visual sequence into 16 blocks, one per frame; Eq. (8) gives each block 512/16=32 queries; Eq. (9) is then exactly the per-frame CrossAttention of Eq. (2), and the concatenation after Eq. (9) reproduces Eq. (3). The paper itself labels this setting 'equivalent to the Image-Level Method' in Fig. 12. Table 9's best 16-frame/512-query row is '16 512 16 32' with VM 55.9/46.6/38.4, LVB 42.4, MVBench 45.5, MLVU 52.5, and Table 2's 'TinyLLaVA-Video 3B 16' row reports exactly these numbers. Thus the headline 16-frame result is produced by the image-level baseline, not by the proposed cross-frame grouping; the claimed temporal-comprehension advantage is the baseline under a new label.

  2. fitted input called prediction [Section 3.2, Fig. 11/Fig. 12/Table 9; Section 4.2, Table 2]
    "We also explore the optimal number of groups in the video-level group method, with the results shown in Fig. 11. In order to explore the upper bound of query-learning efficiency, we experiment with a large number of groups... However, the results show that having too few queries per group does not improve the model performance. The best results are achieved when each group contains 16 to 32 queries."

    The same four benchmarks (Video-MME, LongVideoBench, MVBench, MLVU) are used both to select the number of groups (Fig. 11, Fig. 12, Table 9) and to report the final comparison (Table 2, Tables 5-8). The configuration that maximizes average accuracy on these benchmarks is then presented as the model's headline performance on those same benchmarks. The reported gains are therefore partly a consequence of searching over the evaluation benchmarks, not an independent prediction of them; this is selection-on-the-target rather than a statistically independent evaluation.

full rationale

The paper is not wholly circular: the video-level group resampler of Eq. (6)-(9) is a real mechanism when M<N (e.g., 1fps/512/group=16), and some 128/256-query comparisons do show a benefit from grouping. However, the central 16-frame/512-query configuration used in the main comparison reduces to the image-level resampler by construction, as the paper's own Fig. 12 label '(I)' admits. Additionally, the group count and query count are chosen by maximizing the same four benchmarks later used for the headline comparison, making the headline numbers partly fitted to the evaluation targets. These two issues together constitute partial circularity: one headline result is the baseline under a new name, and the other is tuned on the evaluation benchmarks. Score 6 reflects this partial reduction; no self-citation chain or formal derivation circularity is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on a few hyperparameters chosen on the evaluation benchmarks and on the assumption that contiguous grouping of temporally ordered visual tokens is sufficient for temporal understanding. No new physical entities are introduced, and no mathematical derivation is attempted.

free parameters (4)
  • Total number of learnable queries = 512
    Chosen based on Table 9; configurations with 256 and 512 queries were tested and 512 with 16 groups gave the best average over the four benchmarks.
  • Number of groups M = 16
    Selected from Fig. 11 and Fig. 12; 16 groups, each attending to 2 to 4 frames, gave the best average performance.
  • Frame sampling strategy = 16 uniform frames or 1 fps (max 64 at inference)
    Training data is limited to videos under 5 minutes; the sampling choice affects token count and temporal coverage.
  • Group splitting of concatenated sequence = Contiguous split into M equal groups
    The paper does not compare with alternative grouping strategies such as interleaved frames or learned assignment.
assumptions (3)
  • domain assumption Visual tokens are concatenated in temporal order, so contiguous grouping preserves local temporal structure.
    Section 3.2 Eq. 6-9; the temporal-comprehension advantage is attributed to this ordering without a dedicated temporal position encoding for groups.
  • domain assumption The four benchmarks (Video-MME, MVBench, MLVU, LongVideoBench) are representative proxies for video understanding.
    Section 4.1 and A.7; these benchmarks are used both to select hyperparameters and to report final results.
  • ad hoc to paper Attention redundancy, measured by zeroing out queries, explains the performance gap between naive and group video-level resamplers.
    Section 3.2 Fig. 5; the zeroing experiment supports this hypothesis but is only shown on Video-MME-Short.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler." pith.science (2026). https://pith.science/paper/MOXSPI7I

@misc{pith2026250115513,
  author       = {Pith},
  title        = {Pith review of: TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOXSPI7I}},
  note         = {Machine review of arXiv:2501.15513}
}
read the original abstract

Video behavior recognition and scene understanding are fundamental tasks in multimodal intelligence, serving as critical building blocks for numerous real-world applications. Through large multimodal models (LMMs) have achieved remarkable progress in video understanding, most existing open-source models rely on over 7B parameters and require large-scale datasets for training, making them resource-intensive and inaccessible to many researchers. Furthermore, lightweight models face persistent challenges in effectively processing long visual sequences and temporal understanding. In this work, we introduce TinyLLaVA-Video, a lightweight yet powerful video understanding model with approximately 3.6B parameters. The cornerstone of our design is the video-level group resampler, a novel mechanism that significantly reduces and controls the number of visual tokens at the video level. Unlike traditional image-level resampler, our approach effectively mitigates redundancy while enhancing temporal comprehension, leading to improved performance on video-based tasks. In addition, TinyLLaVA-Video demonstrates exceptional efficiency, requiring only one day of training on 8 A100-40G GPUs. It surpasses several existing 7B-parameter models on multiple benchmarks. We believe this work provides a valuable foundation for future research on lightweight video understanding models. The code and weights is available at https://github.com/ZhangXJ199/TinyLLaVA-Video.

Figures

Figures reproduced from arXiv: 2501.15513 by the authors.

Figure 1
Figure 1. Compared to other pipelines, our method is simple [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The model performance on MLVU (Dev) versus the total number of visual tokens used per video, with bub￾ble size indicating the training resources required for each model. In light of these challenges, we aim to develop a simple yet effective model that significantly reduces the number of vi￾sual tokens at the video sequence level while enhances tem￾poral comprehension across frames. Additionally, our goal is to minim… view at source ↗
Figure 3
Figure 3. Comparison of different resampler. The total number of queries obtained by the image-level [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The structure and pipeline of TinyLLaVA-Video. This section will present the design choices of TinyLLaVA-Video, introduce the model architecture, and analyze three different resampling methods, fol￾lowed by an overview of the training pipeline and training data, as sho…
Figure 6
Figure 6. Figure 6: Attention visualization between the learnable queries and the visual token sequence in [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 5
Figure 5. Figure 5: Performance degradation on Video-MME-Short of different resampler when zeroing out partial queries. The re￾sults show that the query learning efficiency in naive video-level method is relatively low, resulting in noticeable redundancy. Connector (Resampler) Pϕ. In the …
Figure 7
Figure 7. Figure 7: Attention visualization and temporal comprehension capability comparison of three re [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: The redundancy phenomenon in image-level resampler shows that as the number of queries increases, the model’s performance does not increase indefinitely. ’16&8(128)’ represents sampling 16 frames, with each image resampled to 8 queries, and a total of 128 queries. This…
Figure 10
Figure 10. Figure 10: Performance comparison of three resampler with the same total number of queries. Video [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 9
Figure 9. Figure 9: Video-level group resampler demonstrates significantly better per￾formance on temporal comprehension tasks. Based on above observations, we believe that in the naive video-level resampler, the learnable queries must attend to the entire visual token sequence. This appr…
Figure 11
Figure 11. Figure 11: The impact of different group set￾tings on model performance. ’16&128’ rep￾resents uniformly sampling 16 frames, with a total of 128 queries, and the "Im" label indi￾cates that this setting is equivalent to image￾level method. By using the video-level group resampler,…
Figure 12
Figure 12. Figure 12: The average performance of the model across four benchmarks under differ￾ent grouping settings. "128-16-Group=16(I)" indicates a total of 128 queries with 16 uni￾formly sampled frames and 16 groups, which is equivalent to the Image-Level Method. We also explore the op…
Figure 13
Figure 13. Figure 13: The case demonstrates the understanding capability of TinyLLaVA-Video and compares it [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 14
Figure 14. Figure 14: The case demonstrates the understanding capability of TinyLLaVA-Video on different [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs

    cs.CV 2026-07 conditional novelty 5.0 of 10

    GMoT's gated motion tokens improve multimodal LLM micro-gesture recognition on iMiGUE and SMG, with limited support for reasoning-grounding claims.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    arXiv 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    On attention redundancy: A comprehensive study

    Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. On attention redundancy: A comprehensive study. InProceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 930–945, 2021

    work page 2021

  5. [5]

    Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    arXiv 2024

  6. [6]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    arXiv 2024

  7. [7]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

  8. [8]

    Efficient multimodal learning from data-centric perspective.arXiv preprint arXiv:2402.11530, 2024

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective.arXiv preprint arXiv:2402.11530, 2024

    arXiv 2024

Show all 40 references
  1. [9]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  2. [10]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  3. [11]

    Phi-2: The surprising power of small language models.Microsoft Research Blog, 1(3):3, 2023

    Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models.Microsoft Research Blog, 1(3):3, 2023

    2023

  4. [12]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700–13...

    2024

  5. [13]

    Otter: A multi-modal model with in-context instruction tuning.arXiv preprint arXiv:2305.03726, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning.arXiv preprint arXiv:2305.03726, 2023

    2023 arXiv

  6. [14]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  7. [15]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  8. [16]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2025. 10

    2025

  9. [17]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  10. [18]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  11. [19]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  12. [20]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  13. [21]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. InEuropean Conference on Computer Vision, pages 1–18. Springer, 2025

    2025

  14. [22]

    Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207, 2023

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207, 2023

    2023 arXiv

  15. [23]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  16. [24]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  18. [26]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  19. [27]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  20. [28]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    2024 arXiv

  21. [29]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024

    2024 arXiv

  22. [30]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  23. [31]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  24. [32]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  25. [33]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  26. [34]

    Tinyllama: An open-source small language model.arXiv preprint arXiv:2401.02385, 2024

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model.arXiv preprint arXiv:2401.02385, 2024. 11

    2024 arXiv

  27. [35]

    Llava-mini: Efficient image and video large multimodal models with one vision token.arXiv preprint arXiv:2501.03895, 2025

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token.arXiv preprint arXiv:2501.03895, 2025

    2025 arXiv

  28. [36]

    Beyond llava-hd: Diving into high-resolution large multimodal models.arXiv preprint arXiv:2406.08487, 2024

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models.arXiv preprint arXiv:2406.08487, 2024

    2024 arXiv

  29. [37]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  30. [38]

    Tinyllava: A framework of small-scale large multimodal models.arXiv preprint arXiv:2402.14289, 2024

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models.arXiv preprint arXiv:2402.14289, 2024

    2024 arXiv

  31. [39]

    Attri",

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024. 12 A Technical Appendices and Supplementary Mater...

    2024 arXiv

  32. [512]

    The best results are indicated byboldface

    LVB represents LongVideoBench. The best results are indicated byboldface. Pre-training Data Video-MME LVB MVBench MLVU 397k Mixed Data 44.737.6 42.5 48.1 405k Unfil Data44.836.1 40.7 47.2 As mentioned in section 3.3.3, we perform pre-training using 397k mixed data and 405k unf...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.