Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Token pruning that adapts per step speeds video LLMs 1.5x

desk verdict Credible training-free speedup with a real dynamic-pruning idea, but the refresh threshold is unspecified and the 'improving performance' claim overreaches. read the letter →

arxiv 2411.15024 v3 pith:E3QNJ6T6 submitted 2024-11-22 cs.CV cs.LG

Keda Tao , Can Qin , Haoxuan You , Yang Sui , Huan Wang This is my paper · ORCID
classification cs.CVcs.LG
keywords videolargelanguagemodelstokencompressiondynamicKVcachepruningattentionsparsityinferenceaccelerationtemporalredundancytraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DyCoke is a training-free, plug-and-play method for making video large language models faster and lighter. Its central claim is that, in video inputs, the set of visual tokens the model needs changes from one decoding step to the next, so token pruning must be dynamic rather than one-shot. DyCoke combines a first stage that merges visually similar tokens across adjacent frames with a second stage that prunes and reactivates KV-cache entries based on current attention scores. Across video QA and description benchmarks on three model sizes, it reports up to 1.5x inference speedup and 1.4x memory reduction while matching or improving accuracy.

What carries the argument

Two components carry the argument. The temporal token merging (TTM) module, applied during prefilling, groups frames into windows, merges similar tokens across even/odd frame groups using cosine similarity, and keeps full tokens in the first frame of each window, cutting visual tokens by 50-60%. The dynamic KV cache pruning module keeps a small active KV cache selected by attention scores at layer L, stores pruned tokens in a DP cache, and refreshes the active set only when the cosine similarity between successive attention distributions falls below a threshold, so tokens that become important can re-enter during later decoding steps.

What would settle it

Compute, on a sample of videos, the overlap between the top-p% visual token indices selected by layer 3 and those selected by each other layer at the same decoding step; if the average overlap is substantially below the p% retention rate (or below 50% relative overlap), the single-layer selector cannot be reliably representing all layers.

Watch

Extended reading notes

Core claim

The paper's key empirical observation is that attention in a video LLM shifts across decoding iterations: early frames may dominate at one step and later frames at another. On this basis it argues that one-shot pruning methods, which select important tokens once during prefilling, will drop tokens that later decoding steps need. DyCoke therefore maintains a small active KV cache and a separate DP cache; at each step it uses attention scores at one layer to keep the top-p% visual tokens, and it refreshes the active set only when the attention distribution changes enough. This dynamic selection is what lets the method cut tokens to roughly 15 per frame while preserving or improving output quality.

Load-bearing premise

The pruning choices are driven by attention scores from a single transformer layer (layer 3) at one decoding step, and the method assumes that layer's view of token importance matches every other layer's needs.

Editorial extensions

If this is right

  • Video LLMs can run at 1.5x speed with 1.4x less memory at equal or better accuracy, without any fine-tuning or model modification.
  • The same dynamic pruning mechanism could be applied to any long-context multimodal LLM where attention targets shift over time, not just video.
  • Because compression removes redundant tokens, models can accept more video frames (e.g., 32 instead of 16) under a fixed compute budget, which improves performance on short and medium-length videos.
  • The DP-cache design means pruned tokens are not destroyed, so mistakes in one pruning step are recoverable at the next refresh.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-layer attention proxy (layer 3) could be replaced by a lightweight aggregator across layers, which might make the refresh trigger more reliable.
  • The cosine-similarity refresh threshold is a free parameter; an adaptive threshold based on attention entropy could make the method more robust to videos with abrupt scene changes.
  • Combining DyCoke with quantization or distillation could push video LLMs toward mobile deployment, since the method already cuts memory and compute independently of those techniques.
  • The temporal merging stage operates uniformly across frames; making the merge ratio depend on local motion (e.g., merging more in static scenes) could prevent the minor information loss noted in the paper's discussion of rapid scene changes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes DyCoke, a training-free two-stage token compression method for video large language models. In the first stage (TTM), visually redundant tokens across neighboring frames are merged during prefilling on the basis of cosine similarity inside a sliding window. In the second stage, the KV cache is dynamically pruned during decoding: at a chosen transformer layer L, attention scores of the current token over visual tokens are computed, the top-p% indices are kept in the active KV cache, and the remaining tokens are stored in a DP cache; the active set is refreshed when the attention distribution changes substantially. The method is evaluated on LLaVA-OneVision 0.5B, 7B, and 72B across ActivityNet-QA, NextQA, PerceptionTest, VideoDetailCaption, VideoMME, and MVBench, reporting up to about 1.5x latency speedup and 1.4x memory reduction relative to full tokens. The authors position the method as improving accuracy while compressing tokens, and compare against FastV and LLaVA-PruMerge.

Significance. If the central claims hold, DyCoke is a practically useful plug-in: it requires no training, uses the frozen model's own attention signals, and the speedup and memory numbers are measured end-to-end on multiple model scales. The paper includes useful ablations (w/o DP, random pruning) that isolate the contribution of the dynamic mechanism, and the Fig. 2 observation that attention targets shift during decoding is well motivated. The main reservations are that the headline 'still improving the performance' is not consistent across the reported benchmarks, the DP refresh trigger is not specified, and the single-layer attention proxy is not validated against other layers; these issues matter because the accuracy-parity claim rests on them.

major comments (4)
  1. [Sec. 3.2, Eqs. (5)-(7)] The condition that triggers a KV/DP cache swap is described only as 'a low similarity' between successive attention distributions, with no numerical threshold or update schedule given anywhere in the paper. Since the DP mechanism is the central novelty and the paper claims that it preserves accuracy, this missing value prevents reproduction of the results in Tables 1-3 and 5 and leaves the regime where updates are too rare or too frequent unspecified. Please report the exact cosine-similarity threshold and how often the swap occurs in practice, and add a sensitivity analysis around the threshold.
  2. [Sec. 3.2, Eq. (5)] The importance index set I_p^(L) computed at layer L (default L=3) is used to prune the KV cache of every layer via Eq. (6), but the manuscript never checks whether attention at layer 3 is representative of other layers. Table 5 only compares aggregate scores for L=3 and L=10, not the overlap of the selected token sets. Please measure per-layer agreement between I_p^(3) and I_p^(l) (for example, Jaccard similarity of the selected index sets across decoding steps), or justify the choice of L with such data.
  3. [Abstract, Tables 1-3] The claim that DyCoke 'still improves the performance' is only true for selected configurations and benchmarks. For example, in Table 1 on LLaVA-OV-7B the K=0.3 setting loses on ActNet-QA (51.80 vs 51.93) and VideoDC (3.19 vs 3.30) relative to full tokens; in Table 2 the 7B MVBench average at K=0.7 is 57.5 vs 58.0 for full tokens; and in Table 5 increasing P to 0.9 drops VideoDC to 2.86 from the full-token 3.30. The abstract and introduction should be qualified to 'competitive or slightly better on several benchmarks, with large efficiency gains,' and the exact configurations used for the headline should be identified.
  4. [Sec. 4.1 and Table 5] The hyperparameters K, L, and P appear to be selected on the evaluation benchmarks themselves; no validation split or selection procedure is described. Given the sensitivity shown in Table 5 (for example, K=0.9, L=0, P=0.9 collapses ActivityNet accuracy to 40.21), the reported gains may reflect favorable tuning. Please clarify how these values were chosen and provide results for a fixed setting across all benchmarks, or perform selection on a small validation set.
minor comments (5)
  1. [Sec. 4.2, Video Description] The sentence 'As shown in Tab. 2' should refer to Table 1, since the VideoDC results appear in Table 1, not Table 2.
  2. [References [2] and [3]] References [2] and [3] are the same paper (Chen et al., arXiv:2403.06764); please merge them or cite distinct works.
  3. [Sec. 3.2, Eq. (7)] The formal update equations for the DP cache swap after the initial construction in Eq. (7) are missing; the prose describes re-adding tokens, but an exact update rule or pseudocode would improve reproducibility.
  4. [Notation throughout] The model names alternate between 'LLaVA-OV-0.5B' and 'LLaVA-OneVision-0.5B'; please standardize the notation.
  5. [Fig. 2 vs Sec. 3.2] Figure 2 reports attention averaged over all layers, while the method uses layer L only; the discrepancy should be explained because it bears on the single-layer proxy concern.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DyCoke's dynamic pruning uses the model's own attention scores as an input signal; no claim reduces to its fit or to a self-citation.

full rationale

The derivation chain is self-contained. Stage I (TTM) computes cosine similarity between tokens of adjacent frame groups and prunes/merges the most similar tokens; Stage II (DP) takes the model's own cross-attention scores at layer L, selects the top p% visual-token indices, retains those in the KV cache, and stores the rest in a DP cache for possible re-entry when the attention distribution changes. The accuracy claim is empirical, not derivational: there is no equation in which the predicted benchmark score appears as an input, and no fitted parameter is renamed as a prediction. The hyperparameters K, L, and P are swept and reported in ablation tables, which is post-hoc selection rather than circular reasoning. The only author-overlapping citation is xGen-MM-Vid [37], which includes co-author Can Qin; it appears in the related-work survey as an example of mapping video tokens to a compact set and is not load-bearing for any of DyCoke's claims. No uniqueness theorem is imported, and no ansatz is smuggled in through a self-citation. The unquantified refresh threshold and the single-layer attention proxy are reproducibility/robustness concerns, not circularity: they do not make the prediction equivalent to the input. Since no claim reduces by construction to its own inputs, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central efficiency result rests on three empirical assumptions about token redundancy and attention: cosine similarity across adjacent frames identifies redundant tokens; attention at one layer is a sufficient importance proxy; token importance changes slowly between decoding iterations. These are reasonable heuristics for transformer-based VLLMs, but none is proven, and the hyperparameters that control them (K, L, P, window size, DP update threshold) are chosen by hand rather than derived.

free parameters (5)
  • K (TTM pruning rate) = 0.3 / 0.5 / 0.7, headline uses 0.7
    Controls how many tokens are merged in the temporal stage; treated as the primary experimental variable, and different values are reported as best for different benchmarks.
  • P (second-stage retention threshold) = 0.7 (0.9 in one ablation)
    Top-p% of attention scores retained in the KV cache at each refresh; chosen by hand, not derived.
  • L (attention evaluation layer) = 3 (10 tested)
    Layer from which attention scores are read for dynamic pruning; default 3, with only one alternative tested.
  • TTM sliding window length = 4 frames
    Window size used to group frames into odd/even sets for temporal merging; fixed without sensitivity analysis.
  • DP cache update similarity threshold = not specified
    The KV cache is updated only when the attention distribution has low cosine similarity, but no numerical threshold is given in the paper.
assumptions (4)
  • domain assumption Tokens at corresponding spatial positions in adjacent video frames with high cosine similarity carry redundant information and can be merged without loss.
    Invoked in Sec 3.2 Stage I (TTM): even-group tokens with high similarity to odd-group tokens, and similar tokens within odd groups, are pruned. This assumes positional correspondence across frames is semantically meaningful, which can fail with camera motion or scene cuts.
  • domain assumption Attention scores computed at a single transformer layer L and at one decoding step are a sufficient proxy for the importance of every visual KV entry across all layers and future steps.
    Sec 3.2 Stage II uses Eq. 5 at layer L (default L=3) to set the top-p% threshold and decide pruning and restoration for all layers. Table 5 compares L=3 and L=10 but does not validate per-layer consistency.
  • domain assumption The set of important visual tokens changes slowly across consecutive decoding iterations, so the KV cache only needs to be refreshed when the attention distribution changes substantially.
    Sec 3.2 states 'tokens of interest tend to remain consistent across successive iterations' and updates the KV cache only at iteration N when cosine similarity of attention distributions is low; the time scale and threshold are assumed, not measured.
  • domain assumption Low average attention score implies a token can be pruned without changing the next-token distribution.
    Figure 2 and Sec 3.2 motivate pruning by sparsity of averaged attention; averaged sparsity does not guarantee that individual tokens are safe to drop, especially for later tokens that depend on pruned context.
invented entities (1)
  • Dynamic pruning cache (DP cache)
    purpose: Stores KV entries pruned from the active cache during decoding so tokens whose importance rises again can be restored without recomputation.
    Introduced in Sec 3.2 Eq. 7 as an internal buffer. It is not an observable physical entity; its only evidence is end-task accuracy and speed measurements inside the proposed system.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models." pith.science (2026). https://pith.science/paper/E3QNJ6T6

@misc{pith2026241115024,
  author       = {Pith},
  title        = {Pith review of: DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3QNJ6T6}},
  note         = {Machine review of arXiv:2411.15024}
}
read the original abstract

Video large language models (VLLMs) have significantly advanced recently in processing complex video content, yet their inference efficiency remains constrained because of the high computational cost stemming from the thousands of visual tokens generated from the video inputs. We empirically observe that, unlike single image inputs, VLLMs typically attend visual tokens from different frames at different decoding iterations, making a one-shot pruning strategy prone to removing important tokens by mistake. Motivated by this, we present DyCoke, a training-free token compression method to optimize token representation and accelerate VLLMs. DyCoke incorporates a plug-and-play temporal compression module to minimize temporal redundancy by merging redundant tokens across frames, and applies dynamic KV cache reduction to prune spatially redundant tokens selectively. It ensures high-quality inference by dynamically retaining the critical tokens at each decoding step. Extensive experimental results demonstrate that DyCoke can outperform the prior SoTA counterparts, achieving 1.5X inference speedup, 1.4X memory reduction against the baseline VLLM, while still improving the performance, with no training.

Figures

Figures reproduced from arXiv: 2411.15024 by the authors.

Figure 1
Figure 1. Left: We introduce DyCoke (dynamic compression of tokens), a training-free token compression method for fast video large language models. The key innovation of DyCoke over its predecessors is to dynamically remove redundant tokens during the decoding stage, squeezing both the temporal (video frames) and spatial redundancy in visual tokens. Right: Efficiency and performance comparison of various training-free token p… view at source ↗
Figure 2
Figure 2. Attention score between the predicted token at different [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Detailed overview of our DyCoke method. DyCoke compresses visual tokens in VLLMs through a two-stage pruning process: visual token temporal merging (TTM) and KV cache dynamic pruning. Token temporal merging (illustrated in the red dashed box on the left) merges similar tokens in video frames at the prefilling stage, tapping into the temporal redundancy of the video input; KV cache dynamic pruning (illustrated in the… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Showcases of our DyCoke compared to FastV with LLaVA-OV 7B on MVBench. The first row shows that after token compression by FastV, the model generates the wrong answer while our method still retains the correct answer. The second row demonstrates a case that our token c…
Figure 5
Figure 5. Figure 5: Performance vs. K values in different input frames. ing multi-head attention (MHA) and feed-forward network (FFN) modules. Let n, d, and m denote the number of tokens, the hidden state size, and the intermediate size of the FFN, respectively. In the prefilling phase, t…
Figure 6
Figure 6. Figure 6: Showcases of our DyCoke with LLaVA-OV 7B on long video input. The red mark indicates that the reasoning results after token compression remain consistent with the original results, highlighting content comprehension. V1: Video Discription User Describe what’s happening…
Figure 7
Figure 7. Figure 7: Showcases of our DyCoke with LLaVA-OV 7B on short video input. The red mark indicates that the reasoning results after token compression remain consistent with the original results, highlighting content comprehension [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

  2. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  3. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  4. TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A streaming video assistant drops temporally redundant tokens (over 80%) with minimal accuracy loss and uses the drop-ratio curve to trigger proactive responses at scene changes.

  5. Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.

  6. AdaTP: Attention-Debiased Token Pruning for Video Large Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    AdaTP prunes visual tokens in video LLMs by debiasing attention scores, reducing FLOPs to about a quarter of the vanilla model at matching benchmark accuracy.

  7. DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    DynTok dynamically merges similar adjacent visual tokens into groups, reducing video token counts to 44.4% with comparable or better video understanding accuracy.

Reference graph

Works this paper leans on

56 extracted references · 24 canonical work pages · cited by 7 Pith papers

  1. [1]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 3

    arXiv 2022

  2. [3]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 1, 2, 3, 5

    arXiv 2024

  3. [4]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1, 2

    arXiv 2024

  4. [5]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 1, 3

    arXiv 2023

  5. [6]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766,

    arXiv

  6. [7]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettle- moyer. Gpt3. int8 (): 8-bit matrix multiplication for trans- formers at scale. Advances in Neural Information Processing Systems, 35:30318–30332, 2022. 3

    work page 2022

  7. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 5, 6

    arXiv 2024

  8. [9]

    Lazyllm: Dynamic token pruning for efficient long context llm inference

    Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mo- hammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv preprint arXiv:2407.14057, 2024. 3

    arXiv 2024

Show all 56 references
  1. [10]

    Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% perfor- mance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% perfor- mance. Visual Intelligence, 2(1):1–17, 2024. 3

    2024

  2. [11]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 2

    2022

  3. [12]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In ECCV, 2024. 3

    2024

  4. [13]

    An empirical study of llama3 quan- tization: From llms to mllms

    Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of llama3 quan- tization: From llms to mllms. Visual Intelligence, 2(1):36,

  5. [14]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jy- oti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1:3, 2023. 3

    2023

  6. [15]

    Effec- tiveness assessment of recent large vision-language models

    Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, and Fahad Shahbaz Khan. Effec- tiveness assessment of recent large vision-language models. Visual Intelligence, 2(1):17, 2024. 3

    2024

  7. [16]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In CVPR, 2024. 2, 3

    2024

  8. [17]

    Lmms-eval: Accelerating the development of large multimodal models, 2024

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, 2024. 6

    2024

  9. [18]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chun- yuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2, 3, 11

    2024 arXiv

  10. [19]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 2, 3

    2024 arXiv

  11. [20]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 3

    2023

  12. [21]

    Tp2o: Creative text pair-to-object generation using balance swap-sampling

    Jun Li, Zedong Zhang, and Jian Yang. Tp2o: Creative text pair-to-object generation using balance swap-sampling. In ECCV, 2024. 2

    2024

  13. [22]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 1, 3

    2023 arXiv

  14. [23]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In CVPR, 2024. 1, 5, 6, 7, 11

    2024

  15. [24]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV. Springer, 2024. 3

    2024

  16. [25]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 2

    2023 arXiv

  17. [26]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 3

    2024 arXiv

  18. [27]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024. 3

    2024

  19. [28]

    Video- teller: Enhancing cross-modal generation with fusion and decoupling

    Haogeng Liu, Qihang Fan, Tingkai Liu, Linjie Yang, Yunzhe Tao, Huaibo Huang, Ran He, and Hongxia Yang. Video- teller: Enhancing cross-modal generation with fusion and decoupling. arXiv preprint arXiv:2310.04991, 2023. 2

    2023 arXiv

  20. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  21. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024. 3

    2024

  22. [31]

    Video detail caption, 2024

    LMMs-Lab. Video detail caption, 2024. Accessed: 2024-11. 5, 6, 7

    2024

  23. [32]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207,

    arXiv

  24. [33]

    Gpt-4 model, 2023

    OpenAI. Gpt-4 model, 2023. Accessed: 2023-11-08. 5

    2023

  25. [34]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2024

  26. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 5

    2021

  27. [36]

    Testa: Temporal-spatial token aggregation for long-form video-language understanding

    Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. Testa: Temporal-spatial token aggregation for long-form video-language understanding. arXiv preprint arXiv:2310.19060, 2023. 2, 3

    2023 arXiv

  28. [37]

    xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms

    Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms. arXiv preprint arXiv:2410.16267, 2024. 3

    arXiv

  29. [38]

    Pb-llm: Partially binarized large language models

    Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034, 2023. 3

    2023 arXiv

  30. [39]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  31. [40]

    Tempme: Video temporal token merging for efficient text-video retrieval

    Leqi Shen, Tianxiang Hao, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval. arXiv preprint arXiv:2409.01156, 2024. 3

    2024 arXiv

  32. [41]

    Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural infor- mation processing systems, 35:10078–10093, 2022. 2

    2022

  33. [42]

    Look-m: Look- once optimization in kv cache for efficient multimodal long- context inference

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look- once optimization in kv cache for efficient multimodal long- context inference. arXiv preprint arXiv:2406.18139, 2024. 3

    2024 arXiv

  34. [43]

    Tarsier: Recipes for training and evaluating large video description models

    Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024. 1, 2, 3

    2024 arXiv

  35. [44]

    Small language model meets with reinforced vision vocabu- lary

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Small language model meets with reinforced vision vocabu- lary. arXiv preprint arXiv:2401.12503, 2024. 3

    2024 arXiv

  36. [45]

    Mamballie: Implicit retinex-aware low light enhancement with global-then-local state space

    Jiangwei Weng, Zhiqiang Yan, Ying Tai, Jianjun Qian, Jian Yang, and Jun Li. Mamballie: Implicit retinex-aware low light enhancement with global-then-local state space. In NeurIPS,

  37. [46]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, 2021. 5

    2021

  38. [47]

    Robustmq: benchmarking robustness of quantized models

    Yisong Xiao, Aishan Liu, Tianyuan Zhang, Haotong Qin, Jinyang Guo, and Xianglong Liu. Robustmq: benchmarking robustness of quantized models. Visual Intelligence, 1(1):30,

  39. [48]

    Novel object synthesis via adaptive text-image harmony

    Zeren Xiong, Zedong Zhang, Zikun Chen, Shuo Chen, Xiang Li, Gan Sun, Jian Yang, and Jun Li. Novel object synthesis via adaptive text-image harmony. In NeurIPS, 2024. 2

    2024

  40. [49]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1

    2024 arXiv

  41. [50]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019. 5, 6

    2019

  42. [51]

    Tinygpt-v: Efficient multimodal large language model via small backbones

    Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862,

    arXiv

  43. [52]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 1

    2023 arXiv

  44. [53]

    Lmms- eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms- eval: Reality check on the evaluation of large multimodal models, 2024. 6

    2024

  45. [54]

    Tinyllava: A framework of small-scale large multimodal models

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289, 2024. 1, 3

    2024 arXiv

  46. [55]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3

    2023 arXiv

  47. [56]

    Llava-phi: Efficient multi-modal assistant with small language model

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. In Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited, pages 18–22, 2024. 3

    2024

  48. [57]

    counterfactual reasoning

    Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, and Haoji Hu. St 3: Accelerating multimodal large lan- guage model by spatial-temporal visual token trimming. In AAAI, 2025. 3 DyCoke : Dynamic Compression of Tokens for Fast Video Large Language Models Supplementa...

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.