Pith. sign in

REVIEW 5 major objections 5 minor 6 cited by

LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that grouping video tokens into semantic connected components, first inside frames and then across frames, represents a whole video with fewer non-overlapping tokens and outperforms attention- or segment-based…

desk verdict The two-stage semantic connected-component compression is a real idea, but the low-retention wins are confounded by a merge-versus-discard asymmetry and unreported tau selection, so the empirical headline needs fairer baselines before it holds. read the letter →

arxiv 2506.21862 v1 pith:EBY5VQD7 submitted 2025-06-27 cs.CV cs.AIcs.HCcs.MM

Boyuan Sun , Jiaxing Zhao , Xihan Wei , Qibin Hou This is my paper
classification cs.CVcs.AIcs.HCcs.MM
keywords videolargelanguagemodelstokencompressionsemanticconnectedcomponentstraining-freeinferenceretentionratiospatio-temporalquestionansweringvisualredundancy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video large language models turn every frame into hundreds of visual tokens, and inference cost grows with token count. The paper proposes a training-free compressor, LLaVA-Scissor, that groups tokens by pairwise similarity instead of by attention importance. Its claim is that one averaged token per semantic region, with regions found first inside each frame and then across frames, covers all of a video's content without overlap or temporal duplication. The authors argue this is why their method outperforms attention-based and segment-based compression on video question answering, long-video understanding, and multi-choice benchmarks—especially when only a small fraction of tokens is kept. If correct, it gives a simple, model-agnostic way to make video LLMs much cheaper at inference time.

What carries the argument

The central object is the Semantic Connected Components (SCC) operation: a graph whose vertices are visual tokens and whose edges come from a binary similarity map $A = (K \cdot K^{\top} / (\|K\| \cdot \|K\|) > \tau)$. Connected components are found with a union-find data structure over $N' = \min(N, \lceil \log(N)/\epsilon^2 \rceil)$ sampled vertices, and any uncovered vertex is treated as its own component, so the token set is partitioned into disjoint semantic regions. Averaging the tokens inside each component yields one representative token per region. LLaVA-Scissor then applies SCC to each frame's tokens, applies it again to the concatenated frame representatives to remove temporal duplication, and finally assigns every original token to its nearest representative by cosine similarity and averages, producing the final non-overlapping token set for the whole video.

What would settle it

A concrete test: on a video with two distinct objects whose visual encodings are similar, run SCC at the paper's threshold and check whether the two objects fall into one component; if they do and question accuracy on those objects drops below the no-compression baseline while a lower threshold keeps them separate, the single-threshold assumption is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the set of all video tokens can be partitioned into a small number of non-overlapping semantic regions by thresholding pairwise cosine similarity and taking connected components, and that one averaged token per region is enough to represent the entire video. The partition is done twice: first spatially inside each frame, then temporally over the frame-level representatives, so the same object appearing in many frames becomes a single token. The paper argues this coverage-based grouping avoids the two failure modes it attributes to prior work: attention-based selection over-picks salient objects while missing other semantics, and segment-based compression merges temporally adjacent tokens that are not semantically related. Experimental comparisons on video QA, long-video, and MVBench benchmarks with a 7B base model show higher accuracy than FastV, DyCoke, PLLaVA, and VisionZip at equal retention ratios, with the advantage growing as retention falls to 10 percent and below.

Load-bearing premise

The method assumes that one fixed similarity cutoff, applied to every video, groups visual tokens into the regions that carry the content a question would ask about, and that averaging each group keeps the details needed to answer such questions.

Editorial extensions

If this is right

  • At a 50 percent retention ratio, compressed videos perform nearly as well as the uncompressed model on the evaluated benchmarks, so the method removes roughly half the tokens without a visible accuracy cost.
  • The performance gap over attention-based and segment-based baselines widens as the token budget tightens, which makes the method a candidate for long-video understanding and edge deployment.
  • Because compression happens before the LLM and is training-free, the FLOP reduction tracks the retained token count, with only extra similarity computation added.
  • The same pattern appears with a smaller 0.5B base model, indicating the semantic-coverage benefit is not tied to large-model capacity.
  • The paper's reducing-law analysis—performance stable until roughly 35 percent retention, then a sharp drop—supports the view that most visual tokens in video LLMs are redundant and that content coverage is what degrades gracefully.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, SCC could serve as a general pre-LLM token organizer for any long multimodal sequence, since it never consults the language model's attention or training objective.
  • Beyond the paper, an adaptive per-video similarity threshold is a natural extension: the paper's own token-count figures show the same threshold compresses different benchmarks by different amounts, so choosing a threshold from each video's similarity distribution could reduce sensitivity.
  • Beyond the paper, the approximate component step has no stated formal bound linking the error tolerance to component accuracy, so a direct test would check how often an unsampled vertex would bridge two sampled components and whether merging those changes answers.
  • Beyond the paper, the spatial-then-temporal recipe could be made hierarchical—components within shots and then across shots—to scale to hour-long videos without one global temporal pass.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces LLaVA-Scissor, a training-free token compression method for video large language models. It constructs a thresholded pairwise cosine-similarity graph over visual tokens, finds connected components (via a sampling-based approximation), and averages the tokens inside each component to produce one representative token per semantic region. This operation is applied first spatially within each frame and then temporally across frames, and the resulting representative tokens are used as targets to which every original token is assigned and averaged, yielding the final compressed token set. The method is evaluated on video QA benchmarks (ActivityNet-QA, VideoChatGPT, Next-QA), long-video benchmarks (EgoSchema, MLVU, VideoMME, VideoMMMU), and MVBench, using an enhanced LLaVA-OneVision model (7B and 0.5B). The paper reports consistent gains over FastV, DyCoke, PLLaVA, and VisionZip, especially at 10% token retention, together with FLOPs analysis and an ablation study.

Significance. If the empirical claims hold, LLaVA-Scissor offers a simple and practical alternative to attention-based token selection for video LLMs, with the appealing property of being training-free and of explicitly targeting semantic diversity rather than salience. The evaluation is broad, covering question answering, long-video understanding, and multiple-choice video benchmarks, and the method is additionally tested on a 0.5B model, which strengthens the practical utility argument. The ablation comparing SCC with random, uniform, and L2-norm selection within the same merge framework is a useful control that partially isolates the contribution of the connected-components idea. However, the central empirical claim is currently undersupported by the experimental protocol: the similarity threshold is not reported, the comparison with discard-based baselines is not matched on the crucial final merge step, and there are no variance estimates. The significance is therefore conditional on these experimental issues being resolved.

major comments (5)
  1. [Section 4.1, Fig. 3, Tables 1–3] The paper states that the retention ratio is controlled through the similarity threshold τ, but no τ value is reported for any row of Tables 1–3. Figure 3 shows that the token count produced by a fixed τ varies strongly across benchmarks, so reaching exact ratios such as 10%, 35%, or 50% requires per-benchmark or per-video calibration. If this calibration was performed on the test benchmarks while baselines used fixed published settings (FastV and DyCoke at layer 3, VisionZip with a 5:4 dominant-to-contextual ratio), the comparison is not controlled. Please report all τ values, describe the calibration procedure (including whether it uses a validation split), and include at least one comparison in which a single fixed τ is applied across all benchmarks.
  2. [Eq. (8), Section 3.2, Table 4a] The final merge in Eq. (8) averages every original token into the M final tokens, so at the same reported retention ratio the method uses information from all source tokens, whereas FastV and VisionZip discard tokens entirely. This information asymmetry, rather than the semantic connected-component structure, may explain part of the performance gap at low retention ratios. The ablation in Table 4a shows that the merge step improves performance, but no matched experiment is reported in which the same merge-to-centroid operation is applied to tokens selected by FastV or VisionZip under the same budget. Please add such controlled comparisons, or report LLaVA-Scissor without the Eq. (8) merge against the baselines.
  3. [Eq. (2), Algorithm 2, Appendix C.2] The approximate connected-components algorithm samples N' = min(N, log(N)/ε^2) vertices and treats every uncovered vertex as a singleton component. As a result, the extracted sets are not the true connected components of the threshold graph, and Eq. (3) holds only by construction after the singletons are appended. No formal guarantee links ε to component accuracy, and the approximation is active in the reported regimes: with ε=0.05 and 5644 tokens, N' is about 3455, well below N. The manuscript's claim of "comprehensive semantic coverage" is therefore not supported for these settings. Please provide either an exact-mode evaluation, a formal approximation bound, or an empirical comparison against exact connected components on a subset of videos.
  4. [Tables 1–3, Section 4.2] No seeds, error bars, or significance tests are reported, and many of the differences at 35% retention are within rounding distance of zero: for example, Table 1 shows ActivityNet 47.88 vs. 47.83 for FastV and VideoChatGPT 3.47 vs. 3.46, and Table 3 shows MVBench 61.98 vs. 61.78 for DyCoke. The headline claim of superiority "particularly at low token retention ratios" rests mostly on the 10% rows. Please report multiple runs with variance, and where appropriate tests of significance or effect sizes, so the reader can assess whether the moderate-ratio differences are meaningful.
  5. [Section 4.1, Appendix B.2] All experiments use an "enhanced LLaVA-OneVision" model with SIGLIP and Qwen2.5 retrained on Oryx data, but the manuscript gives no training hyperparameters, data recipe, or checkpoint release. Since every comparison is conducted on this private base model, the results are difficult to reproduce or verify externally. Please release the base model and training details, or rerun the main comparisons on a publicly available base model; the current setup is a reproducibility risk for the central empirical claim.
minor comments (5)
  1. [Appendix C.3] The statement that the O(log(N)·N·α(N)) complexity holds "once the number of tokens exceeds 1200" for ε=0.05 is numerically incorrect: solving log(N)/ε^2 < N with natural log gives a threshold close to 3000, not 1200, and for N=1200 the sampled count is still N. Please correct the threshold or state the inequality used.
  2. [Appendix (Table of Contents)] The appendix contents list promises a section on "limitations and broader impacts," but no such section appears in the manuscript. Please add the section or remove the entry from the contents list.
  3. [Appendix B.1, Table 6] Table 6 compares methods that use different base models by reporting relative performance with respect to each method's own baseline. This is not a controlled comparison and should be explicitly labeled as indicative rather than as evidence of superiority; otherwise readers may interpret the relative numbers as directly comparable.
  4. [Throughout] There are numerous naming inconsistencies, including "LLaV A" vs. "LLaVA," "Dycoke" vs. "DyCoke," "PLLaV A" vs. "PLLaVA," and "mluti-choice" in Appendix A.3. Please standardize the notation and fix the typo.
  5. [Fig. 3] Figure 3 plots token counts against τ, but it does not show the retention ratio corresponding to each τ. Adding this would let the reader connect the calibration behavior of τ to the reported retention ratios in Tables 1–3.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the main results are external benchmark comparisons; only the 'comprehensive semantic coverage' claim is a mild definitional tautology.

  1. self definitional [Section 3.1, Eq. (3) and the following paragraph]
    "This property suggests that the token set K can be partitioned into M distinct semantic regions throughout connected components C, which can encompass all non-overlapping semantics indicated by similarity map A."

    The 'semantic regions' are defined by the thresholded similarity map A (Eq. 1) and the connected components C of that graph. The partition property in Eq. (3), namely disjoint components whose sizes sum to N, holds for any connected-components decomposition by construction. Thus, the statement that SCC 'encompass[es] all non-overlapping semantics' is a restatement of how the components were built, not an independent derivation. This tautology is conceptual only: it does not produce the reported accuracy numbers, which are measured against external baselines on public benchmarks, so it is not load-bearing for the empirical claims.

full rationale

The paper's central contribution is algorithmic and empirical rather than a derivation chain, so most circularity patterns do not apply. No fitted parameter is renamed as a prediction: the retention ratio is controlled through tau, and the reported numbers are measured on standard benchmarks against external methods; even if per-benchmark tau calibration occurred, that would be a comparison-fairness issue, not a by-construction equivalence. The approximate connected-components algorithm (Eq. 2, Algorithm 2) is an approximation with stated assumptions, not a hidden reuse of the conclusion. Self-citations (e.g., LLaVA-OneVision as the base model and prior video-LLM papers by the same group) provide context or base architectures but are not invoked as a uniqueness theorem or as justification for the compression design. The only circular element is the phrase 'comprehensive semantic coverage': semantic regions are operationally defined as connected components of the thresholded similarity graph, so covering all semantic regions is true by definition. This does not force the benchmark outcomes, which are external and hence independent of that tautology. Accordingly, the analysis scores 2, reflecting one mild definitional tautology with no load-bearing reduction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. The load-bearing assumptions are the semantic interpretation of thresholded similarity components, the adequacy of the approximate component algorithm, and the validity of the retrained base model as a comparison testbed. Free parameters are tau and epsilon, both influencing token counts.

free parameters (2)
  • similarity threshold tau = not reported per benchmark
    Binarizes pairwise cosine similarities (Eq. 1) and is the primary control of token retention ratio; exact values used for each table are not given.
  • error tolerance epsilon = 0.05
    Controls the number of sampled vertices N' in approximate connected components (Eq. 2); set because smaller values did not change token counts (Fig. 3c).
assumptions (5)
  • domain assumption Tokens with cosine similarity above tau belong to the same semantic region, and averaging them preserves task-relevant information.
    This is the core premise of SCC in Eq. (1)-(4); if thresholding merges distinct objects or splits one object, compressed tokens lose semantics.
  • standard math Connected components of the thresholded similarity graph partition the token set into non-overlapping semantic regions.
    Graph-theoretic property (Eq. 3) that holds for exact components, but its semantic interpretation is assumed.
  • ad hoc to paper The approximate component algorithm with N' sampled vertices and singleton treatment for uncovered nodes is a sufficient approximation of exact semantic components.
    Eq. (2) and Algorithm 2 introduce sampling with an error tolerance epsilon but no proof that components are preserved; unsampled nodes are made singletons.
  • domain assumption A single global threshold tau works across frames and semantic categories.
    Spatial and temporal SCC both use one tau; different videos or object types may need different thresholds.
  • ad hoc to paper The enhanced LLaVA-OneVision base model retrained with SIGLIP/Qwen2.5 on Oryx data is a fair and valid testbed for comparing compression methods.
    Section 4.1 introduces a custom base model not released; comparisons depend on its properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs." pith.science (2026). https://pith.science/paper/EBY5VQD7

@misc{pith2026250621862,
  author       = {Pith},
  title        = {Pith review of: LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBY5VQD7}},
  note         = {Machine review of arXiv:2506.21862}
}
read the original abstract

In this paper, we present LLaVA-Scissor, a training-free token compression strategy designed for video multimodal large language models. Previous methods mostly attempt to compress tokens based on attention scores, but fail to effectively capture all semantic regions and often lead to token redundancy. Differently, we propose to leverage the Semantic Connected Components (SCC) approach that assigns tokens to distinct semantic regions within the token set, ensuring comprehensive semantic coverage. The outcome is a two-step spatio-temporal token compression strategy that utilizes SCC in both spatial and temporal domains. This strategy can effectively compress tokens by representing the entire video with a set of non-overlapping semantic tokens. We conduct extensive evaluations of the token compression capabilities of LLaVA-Scissor across diverse video understanding benchmarks, including video question answering, long video understanding, and comprehensive multi-choices benchmarks. Experimental results show that the proposed LLaVA-Scissor outperforms other token compression methods, achieving superior performance in various video understanding benchmarks, particularly at low token retention ratios. Project page: https://github.com/HumanMLLM/LLaVA-Scissor.

Figures

Figures reproduced from arXiv: 2506.21862 by the authors.

Figure 1
Figure 1. Illustration of different token compression paradigms. □ denotes video tokens, with color representing different semantics. (a) Attention-based methods fail to cover all semantic regions. (b) Segment-based methods introduce temporal redundancy by stacking tokens from each segment. (c) Our two-step spatio-temporal compression strategy is able to identify unique semantic information within each frame and eliminate tem… view at source ↗
Figure 2
Figure 2. Pipeline of LLaVA-Scissor. (a) The Semantic Connected Components (SCC) compress tokens by extracting connected components from the token set. (b) The two-step spatio-temporal compression strategy that extracts unique semantics by leveraging SCC both spatially and temporally. In the binary similarity map A, A(i, j) indicates the similarity between any two tokens ki and kj . Therefore, if we consider each token as a v… view at source ↗
Figure 3
Figure 3. Token number statistics of similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance degradation of methods on different benchmarks as the retained token number [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

  2. C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    C-PTQ weights quantization error by per-channel Fisher information of the task loss, improving low-bit accuracy of multimodal LLMs by small margins over existing channel-wise scaling methods.

  3. EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

    cs.CV 2025-12 conditional novelty 6.0 of 10

    EchoingPixels prunes audio-visual LLM input tokens jointly across modalities and re-tunes RoPE frequencies so that 5–20% of tokens retain roughly full-model performance.

  4. MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs

    cs.CV 2025-08 reject novelty 6.0 of 10

    MMG-Vid prunes up to 75% of visual tokens in video LLMs with near-lossless accuracy by allocating budgets to segments and selecting temporally novel, spatially diverse tokens.

  5. Depth Anything at Any Condition

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A fine-tuned Depth Anything V2 model using perturbation consistency and spatial distance constraints improves monocular depth estimation under adverse conditions without any labeled data.

  6. A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    The paper claims 92.6% dynamic visual-token pruning with retained (or 110% of) baseline VQA performance, but the full text supplied is a different paper, so the claim is unverifiable here.

Reference graph

Works this paper leans on

113 extracted references · 22 canonical work pages · cited by 6 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1

    arXiv 2023

  2. [2]

    When Less is Enough: Adaptive Token Reduction for Efficient Image Representation

    Eduard Allakhverdov, Elizaveta Goncharova, and Andrey Kuznetsov. When less is enough: Adaptive token reduction for efficient image representation.arXiv preprint arXiv:2503.16660, 2025. 1

    work page Pith review arXiv 2025

  3. [3]

    Union-find with constant time deletions

    Stephen Alstrup, Inge Li Gørtz, Theis Rauhe, Mikkel Thorup, and Uri Zwick. Union-find with constant time deletions. InAutomata, Languages and Programming, pages 78–89, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. 21

    2005

  4. [4]

    Divprune: Diversity-based visual token pruning for large multimodal models, 2025

    Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models, 2025. 18

    2025

  5. [5]

    Claude-3.5, 2024

    Anthropic. Claude-3.5, 2024. 3

    2024

  6. [6]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  7. [7]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. InInternational Conference on Learning Representations,

  8. [8]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Tom B Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020. 3

    arXiv 2005

Show all 113 references
  1. [9]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InProceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 17

    2015

  2. [10]

    Efficient large multi-modal models via visual context compression

    Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi-modal models via visual context compression. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 3

    2024

  3. [11]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024. 1, 3, 5, 6, 7, 8, 9, 18, 19

    2024

  4. [12]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023. 1

    2023 arXiv

  5. [13]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024. 1

    2024 arXiv

  6. [14]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024. 3

    2024 arXiv

  7. [15]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 3 10

    2023

  8. [16]

    Pact: Pruning and clustering-based token reduction for faster visual language models, 2025

    Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, and Aymen Shabou. Pact: Pruning and clustering-based token reduction for faster visual language models, 2025. 18

    2025

  9. [17]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.arXiv preprint arXiv:2401.16420, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.arXiv preprint arXiv:2401.1642...

    2024 arXiv

  10. [18]

    Adaptive token sampling for efficient vision transformers

    Mohsen Fayyaz, Soroush Abbasi Kouhpayegani, Farnoush Rezaei Jafari, Eric Sommerlade, Hamid Reza Vaezi Joze, Hamed Pirsiavash, and Juergen Gall. Adaptive token sampling for efficient vision transformers. European Conference on Computer Vision (ECCV), 2022. 1

    2022

  11. [19]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024. 6, 9, 17

    2024 arXiv

  12. [20]

    Framefusion: Combining similarity and importance for video token reduction on large visual language models, 2024

    Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models, 2024. 18

    2024

  13. [21]

    Gabow and Robert Endre Tarjan

    Harold N. Gabow and Robert Endre Tarjan. A linear-time algorithm for a special case of disjoint set union.Journal of Computer and System Sciences, 30(2):209–221, 1985. Funding Information: supported by the National Science Foundation, Grant MCS78-18909. 21

    1985

  14. [22]

    Llama-adapter v2: Parameter-efficient visual instruction model.arXiv preprint arXiv:2304.15010, 2023

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model.arXiv preprint arXiv:2304.15010, 2023. 3

    2023 arXiv

  15. [23]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 3

    2025 arXiv

  16. [24]

    Fila-video: Spatio-temporal compression for fine-grained long video understanding.arXiv preprint arXiv:2504.20384, 2025

    Yanan Guo, Wenhui Dong, Jun Song, Shiding Zhu, Xuan Zhang, Hanqing Yang, Yingbo Wang, Yang Du, Xianing Chen, and Bo Zheng. Fila-video: Spatio-temporal compression for fine-grained long video understanding.arXiv preprint arXiv:2504.20384, 2025. 2

    2025 arXiv

  17. [25]

    Filter, correlate, compress: Training-free token reduction for mllm acceleration, 2025

    Yuhang Han, Xuyang Liu, Zihan Zhang, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. Filter, correlate, compress: Training-free token reduction for mllm acceleration, 2025. 3

    2025

  18. [26]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. 2025. 6, 17

    2025

  19. [27]

    Prunevid: Visual token pruning for efficient video large language models.arXiv preprint arXiv:2412.16117, 2024

    Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models.arXiv preprint arXiv:2412.16117, 2024. 2, 3, 18

    2024 arXiv

  20. [28]

    Mixtral of experts.arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv:2401.04088, 2024. 3

    2024 arXiv

  21. [29]

    Lookupvit: Compressing visual information to a limited number of tokens

    Rajat Koner, Gagan Jain, Prateek Jain, V olker Tresp, and Sujoy Paul. Lookupvit: Compressing visual information to a limited number of tokens. InEuropean Conference on Computer Vision, pages 322–337. Springer, 2024. 3

    2024

  22. [30]

    Llava-next: What else influences visual instruction tuning beyond data?, 2024

    Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruction tuning beyond data?, 2024. 3

    2024

  23. [31]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    arXiv

  24. [32]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 1

    2024 arXiv

  25. [33]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023. 1 11

    2023 arXiv

  26. [34]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–222...

    2024

  27. [35]

    Tokenpacker: Efficient visual projector for multimodal llm, 2024

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm, 2024. 3

    2024

  28. [36]

    Llama-vid: An image is worth 2 tokens in large language models.European Conference on Computer Vision, 2024

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models.European Conference on Computer Vision, 2024. 1, 3

    2024

  29. [37]

    Not all patches are what you need: Expediting vision transformers via token reorganizations.arXiv preprint arXiv:2202.07800, 2022

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations.arXiv preprint arXiv:2202.07800, 2022. 3

    2022 arXiv

  30. [38]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 1, 3, 18

    2023 arXiv

  31. [39]

    Boosting multimodal large language models with visual tokens withdrawal for rapid inference.arXiv preprint arXiv:2405.05803, 2024

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference.arXiv preprint arXiv:2405.05803, 2024. 3

    2024 arXiv

  32. [40]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  33. [41]

    Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023. 3

    2023 arXiv

  34. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. 3

    2023

  35. [43]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 3

    2024

  36. [44]

    World model on million-length video and language with ringattention.arXiv preprint arXiv:2402.08268, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention.arXiv preprint arXiv:2402.08268, 2024. 1

    2024 arXiv

  37. [45]

    Multi-stage vision token dropping: Towards efficient multimodal large language model.arXiv preprint arXiv:2411.10803,

    Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model.arXiv preprint arXiv:2411.10803,

    arXiv

  38. [46]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024. 6

    2024 arXiv

  39. [47]

    Hybrid-level instruction injection for video token compression in multi-modal large language models.arXiv preprint arXiv:2503.16036, 2025

    Zhihang Liu, Chen-Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, and Hongtao Xie. Hybrid-level instruction injection for video token compression in multi-modal large language models.arXiv preprint arXiv:2503.16036, 2025. 2

    2025 arXiv

  40. [48]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424,

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424,

    arXiv

  41. [49]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 3, 6, 17

    2024

  42. [50]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 6

    2023

  43. [51]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36, 2024

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36, 2024. 17

    2024

  44. [52]

    ChatGPT.https://openai.com/blog/chatgpt/, 2023

    OpenAI. ChatGPT.https://openai.com/blog/chatgpt/, 2023. 3

    2023

  45. [53]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. 1, 3

    2023

  46. [54]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 3 12

    2023

  47. [55]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 3

    2024

  48. [56]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  49. [57]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

  50. [58]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InAdvances in Neural Information Processing Systems (NeurIPS), 2021. 3

    2021

  51. [59]

    Timechat: A time-sensitive multimodal large language model for long video understanding.ArXiv, abs/2312.02051, 2023

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding.ArXiv, abs/2312.02051, 2023. 3

    2023 arXiv

  52. [60]

    Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles

    Michael S. Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms, 2024. 1

    2024

  53. [61]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388, 2024

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388, 2024. 1, 3, 18

    2024

  54. [62]

    Tempme: Video temporal token merging for efficient text-video retrieval.arXiv preprint arXiv:2409.01156, 2024

    Leqi Shen, Tianxiang Hao, Tao He, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval.arXiv preprint arXiv:2409.01156, 2024. 1

    2024 arXiv

  55. [63]

    Fastvid: Dynamic density pruning for fast video large language models.arXiv preprint arXiv:2503.11187,

    Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. Fastvid: Dynamic density pruning for fast video large language models.arXiv preprint arXiv:2503.11187,

  56. [64]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spatiote...

    arXiv

  57. [65]

    Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers.arXiv preprint arXiv:2305.17455, 2023

    Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers.arXiv preprint arXiv:2305.17455, 2023. 3

    2023 arXiv

  58. [66]

    Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023. 3

    2023 arXiv

  59. [67]

    Corrmatch: Label propagation via correlation matching for semi-supervised semantic segmentation.IEEE Computer Vision and Pattern Recognition (CVPR), 2024

    Boyuan Sun, Yuqi Yang, Le Zhang, Ming-Ming Cheng, and Qibin Hou. Corrmatch: Label propagation via correlation matching for semi-supervised semantic segmentation.IEEE Computer Vision and Pattern Recognition (CVPR), 2024. 8

    2024

  60. [68]

    Dycoke: Dynamic compression of tokens for fast video large language models.arXiv preprint arXiv:2411.15024, 2024

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models.arXiv preprint arXiv:2411.15024, 2024. 2, 3, 5, 6, 7, 8, 18

    2024 arXiv

  61. [69]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team. Gemini: A family of highly capable multimodal models, 2024. 3

    2024

  62. [70]

    Qwen2-vl

    Qwen team. Qwen2-vl. 2024. 1

    2024

  63. [71]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 3, 6, 18

    2024

  64. [72]

    Llama 2: Open foundation and fine-tuned chat models.arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv:2307.09288, 2023. 3

    2023 arXiv

  65. [73]

    [cls] token tells everything needed for training-free efficient mllms.arXiv preprint arXiv:2412.05819, 2024

    Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [cls] token tells everything needed for training-free efficient mllms.arXiv preprint arXiv:2412.05819, 2024. 3

    2024 arXiv

  66. [74]

    Videoagent: Long-form video understanding with large language model as agent, 2024

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent, 2024. 3 13

    2024

  67. [75]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 1

    2024 arXiv

  68. [76]

    Videollamb: Long video understanding with recurrent memory bridges.arxiv, 2024

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long video understanding with recurrent memory bridges.arxiv, 2024. 2, 3

    2024

  69. [77]

    Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos, 2024

    Ying Wang, Yanlai Yang, and Mengye Ren. Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos, 2024. 3

    2024

  70. [78]

    Dymu: Dynamic merging and virtual unmerging for efficient vlms, 2025

    Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, and Ran Xu. Dymu: Dynamic merging and virtual unmerging for efficient vlms, 2025. 3

    2025

  71. [79]

    Longvlm: Efficient long video understanding via large language models.arXiv preprint arXiv:2404.03384, 2024

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models.arXiv preprint arXiv:2404.03384, 2024. 3

    2024 arXiv

  72. [80]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777–9786, 2021. 6, 17

    2021

  73. [81]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024. 1

    arXiv

  74. [82]

    Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024. 3, 5, 6, 7, 8, 9, 18

    2024

  75. [83]

    xgen-mm (blip-3): A family of open large multimodal models, 2024

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin C...

    2024

  76. [84]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    arXiv

  77. [85]

    Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model.arXiv preprint arXiv:2503.18278, 2025

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model.arXiv preprint arXiv:2503.182...

    2025 arXiv

  78. [86]

    Visionzip: Longer is better but not necessary in vision language models.arXiv preprint arXiv:2412.04467, 2024

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models.arXiv preprint arXiv:2412.04467, 2024. 1, 3, 5, 6, 7, 8, 9, 18

    2024

  79. [87]

    Deco: Decoupling token compression from semantic abstraction in multimodal large language models.arXiv preprint arXiv:2405.20985, 2024

    Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compression from semantic abstraction in multimodal large language models.arXiv preprint arXiv:2405.20985, 2024. 3

    2024 arXiv

  80. [88]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,

  81. [90]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023. 3 14

    2023 arXiv

  82. [91]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,

  83. [92]

    Atp-llava: Adaptive token pruning for large vision language models.ArXiv, abs/2412.00447, 2024

    Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models.ArXiv, abs/2412.00447, 2024. 3

    2024 arXiv

  84. [93]

    V oCo-LLaMA: Towards Vision Compression with Large Language Models.arXiv preprint arXiv:2406.12275, 2024

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. V oCo-LLaMA: Towards Vision Compression with Large Language Models.arXiv preprint arXiv:2406.12275, 2024. 1

    2024 arXiv

  85. [94]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InAAAI, pages 9127–9134, 2019. 6

    2019

  86. [95]

    Timesuite: Improving mllms for long video understanding via grounded tuning, 2024

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, Yali Wang, Yu Qiao, and Limin Wang. Timesuite: Improving mllms for long video understanding via grounded tuning, 2024. 3

    2024

  87. [96]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 6, 18

    2023

  88. [97]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding.arX...

    2025 arXiv

  89. [98]

    A simple llm framework for long-range video question-answering, 2023

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering, 2023. 3

    2023

  90. [99]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023. 3

    2023 arXiv

  91. [100]

    Long context transfer from language to vision, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision, 2024. 3

    2024

  92. [101]

    Vision transformer with quadrangle attention

    Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle attention. arXiv preprint arXiv:2303.15105, 2023. 1

    2023 arXiv

  93. [102]

    [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster.arXiv preprint arXiv:2412.01818, 2024

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, MinQi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster.arXiv preprint arXiv:2412.01818, 2024. 3

    2024 arXiv

  94. [103]

    Token-level correlation-guided compression for efficient multimodal document understanding, 2024

    Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, and Liqiang Nie. Token-level correlation-guided compression for efficient multimodal document understanding, 2024. 3

    2024

  95. [104]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference.arXiv preprint arXiv:2410.04417, 2024

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference.arXiv preprint arXiv:2410.04417, 2024. 1

    arXiv

  96. [106]

    Llava-next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, 2024. 3

    2024

  97. [107]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 3

    2024

  98. [108]

    Facial dynamics in video: Instruction tuning for improved facial expression perception and contextual awareness.arXiv preprint arXiv:2501.07978, 2025

    Jiaxing Zhao, Boyuan Sun, Xiang Chen, and Xihan Wei. Facial dynamics in video: Instruction tuning for improved facial expression perception and contextual awareness.arXiv preprint arXiv:2501.07978, 2025. 1

    2025 arXiv

  99. [109]

    Llava-octopus: Unlocking instruction- driven adaptive projector fusion for video understanding.arXiv preprint arXiv:2501.05067, 2025

    Jiaxing Zhao, Boyuan Sun, Xiang Chen, Xihan Wei, and Qibin Hou. Llava-octopus: Unlocking instruction- driven adaptive projector fusion for video understanding.arXiv preprint arXiv:2501.05067, 2025. 3

    2025 arXiv

  100. [110]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning.arXiv preprint arXiv:2503.05379, 2025

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning.arXiv preprint arXiv:2503.05379, 2025. 3 15

    2025 arXiv

  101. [111]

    Humanomni: A large vision-speech language model for human-centric video understanding.arXiv preprint arXiv:2501.15111, 2025

    Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. Humanomni: A large vision-speech language model for human-centric video understanding.arXiv preprint arXiv:2501.15111, 2025. 1

    2025 arXiv

  102. [113]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 17

    2024 arXiv

  103. [114]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    arXiv

  104. [115]

    Apollo: An exploration of video understanding in large multimodal models.arXiv preprint arXiv:2412.10360, 2024

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, , and Xide Xia. Apollo: An exploration of video understanding in large multimodal models.arXiv preprint arXiv:2412...

    2024 arXiv

  105. [2024]

    3, 5, 6, 7, 8, 18, 19

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.