Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models can run 2-3x faster while keeping their accuracy

desk verdict Useful acceleration idea and strong retention numbers, but the token-merging algorithm is internally inconsistent and the key results are not reproducible from the text. read the letter →

arxiv 2509.00419 v1 pith:OI4UPZPS submitted 2025-08-30 cs.CV

Lianyu Hu , Fanhua Shang , Wei Feng , Liang Wan This is my paper
classification cs.CV
keywords vision-languagemodelsinferenceaccelerationtokenmergingKVcachecompressiontraining-freeattentionsparsitylong-sequencegenerationmultimodalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LightVLM claims that most of the image tokens a vision-language model processes are redundant, and that the LLM's own attention scores reveal which tokens matter. It therefore proposes a training-free plug-in that progressively merges low-attention image tokens during encoding and prunes image tokens from the KV cache during decoding. The paper reports that keeping only 35% of image tokens preserves 100% of benchmark performance, keeping 3% preserves about 98%, and inference time drops by 2.02x overall and 3.21x when generating long outputs. If right, this makes large multimodal models practical on constrained hardware and lets bigger models beat smaller ones on both speed and accuracy.

What carries the argument

Pyramid token merging: a layer-wise schedule that keeps all image tokens in early LLM layers, then gradually merges low-attention tokens into a single weighted-sum token at chosen layers, using only the cumulative attention scores returned by Flash-Attention (no full attention matrix, O(1) merge). KV cache compression: after early layers, per-head pruning of image tokens whose accumulated attention falls below a threshold, shrinking the cache used in generation. Both rely on the observed 'attention sink' phenomenon—a few image tokens dominate attention in deep layers.

What would settle it

Build a benchmark of fine-grained peripheral details (small text, small objects, corner regions) and run LightVLM at 3% token retention. If accuracy drops well below 98% of vanilla while attention remains concentrated elsewhere, the attention-as-importance proxy fails. A second check: compare attention-based selection against random token selection at the same budget; if random selection matches attention-based accuracy, attention scores are not doing the work.

Watch

Extended reading notes

Core claim

The central claim is that VLM inference can be accelerated without any retraining by exploiting a measured asymmetry: across LLM layers, a small number of image tokens accumulate most of the attention, while the rest become progressively less informative. LightVLM encodes this as two operations. Pyramid token merging reduces image tokens layer by layer, keeping only the highest-attention tokens and collapsing the remainder into a single merged token via a weighted sum; the whole merge is O(1) and works with Flash-Attention's cumsum scores. KV cache compression then uses the same attention signal to discard low-scoring image tokens from cached keys and values before generation, with per-head

Load-bearing premise

The claim rests on the assumption that attention scores from the LLM are a stable, reliable measure of which image tokens carry task-relevant information—so tokens with low attention can be merged or discarded without losing what the model needs.

Editorial extensions

If this is right

  • A deployment can cut prefill time by about 3.65x and more than double throughput with no accuracy loss at 35% token retention.
  • Large VLMs (e.g., 26B) can serve faster than smaller counterparts (e.g., 8B) while scoring higher, inverting the usual accuracy-latency tradeoff.
  • Long-form generation (4096 tokens) becomes 3.21x faster, and memory use drops 1.85x, allowing longer outputs under a fixed memory cap.
  • Because the method is training-free and plug-in, it applies to off-the-shelf VLMs without dataset-specific tuning.
  • The approach transfers to video understanding: at 3% video tokens it retains about 98% of vanilla accuracy on four video benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Attention-score importance may extend to other modalities (audio, depth) and to encoder-side token selection before the LLM, where similar concentration may hold.
  • The 98%-at-3% result suggests an information bottleneck in VLM vision encoders: only a few tokens carry the answer-relevant content, which could inform future training losses or tokenization.
  • Combining this cache pruning with other KV-compression strategies (e.g., eviction of old text tokens) might yield further gains, since the two mechanisms target different cache entries.
  • A testable extension is to make the merge budget per-image adaptive, e.g., allocating more tokens to high-detail images by measuring attention entropy, rather than using a fixed global ratio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. LightVLM proposes a training-free, plug-in acceleration method for vision-language models. During encoding, "pyramid token merging" reduces image tokens at selected LLM layers by merging low-attention tokens according to attention scores, eventually retaining a small fraction (e.g., 3%) of image tokens. During decoding, "KV Cache compression" discards low-attention image tokens from the KV cache using a cumulative attention threshold. Experiments on QWen2.5-VL 7B, InternVL2.5 8B/26B/38B, LLaVA-OneVision 7B, and MiniCPM-V2.6 8B over image and video benchmarks report roughly 100% retained performance at 35% image tokens, around 98% at 3%, a 2.02x throughput increase, a 3.65x prefilling-time reduction, and a 3.21x latency reduction for 4096-token generation.

Significance. If the empirical claims hold, the practical significance is high: the method is training-free, compatible with FlashAttention, and appears to dominate prior token-pruning methods (SparseVLM, FasterVLM, VisionZip, PyramidDrop) at extreme retention ratios. The paper provides a broad benchmark comparison and an efficiency analysis, and it makes falsifiable quantitative predictions. However, the core algorithm is not uniquely defined in the text, and no code or complete hyperparameter settings are provided, so the headline numbers are currently unverifiable. The token-merging idea is incremental relative to PyramidDrop/VisionZip, but the KV-cache component adds a useful angle for long-sequence decoding.

major comments (4)
  1. [§3.2.1] The token-merging algorithm is internally inconsistent. The text says to repeatedly merge the last two least-important tokens with equal weights, but then states that the result is obtained as Pnew = Wv × Pmerged with Wv = {r+1, r, ..., 1}. Repeated equal-weight pairwise merging yields coefficients that are powers of 1/2 (e.g., for r=3, [1/2, 1/4, 1/8, 1/8] on the ordered tokens), not {4, 3, 2, 1}; an unnormalized multiplication by {4, 3, 2, 1} would also change the feature scale by O(r). These operations produce different merged tokens and therefore different attention scores and downstream pruning decisions. Since Tables 1–5 are the central evidence, the manuscript must specify exactly which operation was run, provide corrected pseudo-code, and ideally release code. Without this, the 97.8%-at-3%-tokens claim is not tied to a uniquely defined, reproducible method.
  2. [§3.2.1] The stated O(1) complexity for the merging step is incorrect. The multiplication Wv × Pmerged with Wv ∈ R^{r+1} and Pmerged ∈ R^{(r+1)×C} has cost O(rC). Even if r and C are treated as small constants, the claim "only O(1) computational complexity" is not accurate and should be corrected, e.g., to O(rC) per merging layer.
  3. [Appendix A.2] Reproducibility of the empirical results requires missing configuration details. Appendix A.2 reports only that token merging is applied in layers 5, 9, and 13 with "a constant token reduction ratio" but gives no ratio values for the 35%/15%/3% retained-token budgets; no information is given on how the layers and ratios were selected, how β=0.995 is applied per head, or how per-head KV pruning interacts with the merged token sequence. No code is released. These choices are essential to the reported accuracy/latency trade-offs.
  4. [§3.2.2] The KV Cache compression section is ambiguous about head handling. The method text says attention scores are averaged across heads for clarity, but later says pruning is "conducted independently per head." These lead to different retained-token sets per head and different KV memory/latency behavior. Please clarify the exact per-head procedure and state whether the cumulative attention threshold β is applied to per-head or averaged scores.
minor comments (6)
  1. [Title and throughout] Typos and formatting issues: "Acceleraing" in the title and first line, "imge" in §2.2, "diferent" in §3.2.1, "LLaV A" in Table 3 and related work. Please proofread carefully.
  2. [§4.5, Table 5] The claim that LightVLM has a "clear advantage by driving VLMs faster" is not supported by Table 5: VisionZip achieves higher throughput (3.83 vs 3.75 images/s) and lower prefilling time (96 vs 117 ms). LightVLM's advantage is accuracy, not raw speed; please rephrase.
  3. [§3.2.1, Fig. 2] The paper relies on attention scores as the importance metric but does not include a control such as random token retention or low-attention retention. The evidence is indirect through comparisons with other attention-based methods. Adding such a control would strengthen the causal claim.
  4. [Tables 1–2] No error bars or repeated-run variance are reported. Most benchmarks are deterministic under greedy decoding, but if sampling is used, this should be stated. Also, reporting average values to one decimal place for relative scores that are computed from per-benchmark ratios should be clarified.
  5. [Limitations] The Limitations section mentions untested attention variants and model coverage but does not mention the absence of released code or the sensitivity of results to the merging schedule and KV threshold. These are important for a training-free plug-in method and should be acknowledged.
  6. [Figures] Figures 2 and 7 are visually dense; axis labels, legends, and the distinction between retained and dropped tokens are hard to read in the PDF. Please enlarge fonts and use clearer annotations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LightVLM's claims are empirical evaluations of a training-free pruning scheme, not derivations that reduce to their inputs.

full rationale

LightVLM is an empirical acceleration method, not a derivation whose outputs are predetermined by its inputs. The central claims (97.8% accuracy at 3% image tokens, 2.02x/3.21x speedups) are benchmark measurements obtained by running a token-merging/KV-cache-pruning scheme, and no reported quantity is a fitted parameter renamed as a prediction. The use of attention scores as an importance signal for pruning is a standard heuristic; selecting top-attention tokens and then reporting downstream accuracy on held-out benchmarks is not circular because the benchmark scores are not constructed from the selection rule (Sec. 3.2.1, Sec. 3.2.2). The hyperparameters (pruning layers 5/9/13, beta=0.995, reduction ratios) are stated implementation choices, not quantities inferred from the target result. The paper's self-citations are contextual and not load-bearing, and no uniqueness theorem or prior-work ansatz is invoked to force the method. The internal inconsistency in Sec. 3.2.1 between repeated pairwise averaging and the closed-form weight list Wv={r+1,...,1} is a reproducibility/correctness concern rather than circularity: even if the algorithm is underspecified and the headline numbers are currently unreproducible from the text, the reported accuracy values are not equivalent to the method's inputs by construction. Therefore no circular step is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

There are no invented physical or mathematical entities. The free parameters are hand-chosen hyperparameters (layer schedule, threshold, reduction ratios). The central assumptions are empirical: attention scores indicate importance and merged or pruned tokens are dispensable. The O(1) concern is not a ledger item but a correctness flag.

free parameters (3)
  • per-layer token reduction ratio = not reported (constant across merging layers)
    Determined by the computing budget rather than derived; no sensitivity analysis is provided (Sec. A.2).
  • merging layers = 5, 9, 13
    Chosen by hand for the default QWen2.5-VL 7B setting; no ablation of adjacent layer choices (Sec. A.2).
  • KV cache attention threshold beta = 0.995
    Set to a single value for all pruned layers; no ablation across beta values is reported (Sec. A.2).
assumptions (4)
  • domain assumption LLM attention scores are a reliable proxy for image-token importance across layers and tasks.
    Motivated only by Fig. 2 observations; not proven or stress-tested. Both the merging and pruning steps rely on this (Sec. 3.2.1, 3.2.2).
  • domain assumption Merging low-attention tokens into a weighted average preserves the information needed for downstream generation.
    Asserted as the basis of pyramid token merging; no theoretical argument or ablation isolates the merge operation from the selection (Sec. 3.2.1).
  • domain assumption Tokens that receive high attention during encoding remain the only tokens needed during decoding.
    KV cache compression prunes image tokens before generation based on encoding-stage attention; the stability of importance during decoding is assumed (Sec. 3.2.2).
  • domain assumption The cumsum attention weights returned by flash_attn_varlen_func can be used as per-token importance.
    An implementation detail; the paper does not validate that cumsum weights behave equivalently to full attention scores (Sec. 3.2.1 footnote).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression." pith.science (2026). https://pith.science/paper/OI4UPZPS

@misc{pith2026250900419,
  author       = {Pith},
  title        = {Pith review of: LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OI4UPZPS}},
  note         = {Machine review of arXiv:2509.00419}
}
abstract

In this paper, we introduce LightVLM, a simple but effective method that can be seamlessly deployed upon existing Vision-Language Models (VLMs) to greatly accelerate the inference process in a training-free manner. We divide the inference procedure of VLMs into two stages, i.e., encoding and decoding, and propose to simultaneously accelerate VLMs in both stages to largely improve model efficiency. During encoding, we propose pyramid token merging to reduce tokens of different LLM layers in a hierarchical manner by finally only keeping a few dominant tokens to achieve high efficiency. During decoding, aimed at reducing the high latency of outputting long sequences, we propose KV Cache compression to remove unnecessary caches to increase the network throughput. Experimental results show that LightVLM successfully retains 100% performance when only preserving 35% image tokens, and maintains around 98% performance when keeping only 3% image tokens. LightVLM could 2.02$\times$ the network throughput and reduce the prefilling time by 3.65$\times$. LightVLM also makes large VLMs faster again by enabling a heavy model (e.g., InternVL2.5 26B) to infer faster than significantly smaller models (e.g., InternVL2.5 8B), hopefully facilitating the real-world deployment. When generating long text sequences (e.g., 4096 tokens), LightVLM could reduce the inference time by 3.21$\times$, largely outperforming existing methods.

Figures

Figures reproduced from arXiv: 2509.00419 by the authors.

Figure 1
Figure 1. (a) When retaining only 3% image tokens, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The attention score distribution for input tokens including system prompts, image tokens and user [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The input image is first transformed into image token sequences by an image encoder and a projector, which [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: We measure the latency of generating long [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: We measure the memory usage of generating [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Visualizations for the preserved image tokens after pyramid token merging compared to original images [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    ALTR keeps model accuracy nearly intact while pruning up to ~89% of visual tokens, using entropy ranking plus stride sampling on an MLP consistency signal, without attention maps or pairwise comparisons.

Reference graph

Works this paper leans on

67 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716--23736

    work page 2022

  4. [4]

    Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. 2024. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource-constrained environments. arXiv preprint arXiv:2408.10945

    arXiv 2024

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  6. [6]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024 a . An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19--35. Springer

    2024

  7. [7]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024 b . Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330

    arXiv 2024

  8. [8]

    Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. 2023. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2061--2070

    work page 2023

Show all 67 references
  1. [9]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024 c . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    2024 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075

    2024 arXiv

  4. [12]

    Zijian Gao, Shanhao Han, Xingxing Zhang, Kele Xu, Dulan Zhou, Xinjun Mao, Yong Dou, and Huaimin Wang. 2025 a . https://doi.org/10.1609/aaai.v39i16.33842 Maintaining fairness in logit-based knowledge distillation for class-incremental learning . Proceedings of the AAAI Conferen...

    2025 doi

  5. [13]

    Zijian Gao, Wangwang Jia, Xingxing Zhang, Dulan Zhou, Kele Xu, Feng Dawei, Yong Dou, Xinjun Mao, and Huaimin Wang. 2025 b . Knowledge memorization and rumination for pre-trained model-based class-incremental learning. In Proceedings of the Computer Vision and Pattern Recogniti...

    2025

  6. [14]

    Zijian Gao, Kele Xu, Huiping Zhuang, Li Liu, Xinjun Mao, Bo Ding, Dawei Feng, and Huaimin Wang. 2024 a . https://doi.org/10.1016/j.neunet.2024.106513 Less confidence, less forgetting: Learning with a humbler teacher in exemplar-free class-incremental learning . Neural Networks...

    2024

  7. [15]

    Zijian Gao, Xingxing Zhang, Kele Xu, Xinjun Mao, and Huaimin Wang. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/e7feb9dbd9a94b6c552fc403fcebf2ef-Paper-Conference.pdf Stabilizing zero-shot prediction: A novel antidote to forgetting in continual vision-lan...

    2024

  8. [16]

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pages 3690--3699. PMLR

    2020

  9. [17]

    grok 1.5v. 2024. https://x.ai/blog/grok-1.5v Realworldqa benchmark

    2024

  10. [18]

    Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2023. Continuous sign language recognition with correlation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2529--2539

    2023

  11. [19]

    Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2024 a . Scalable frame resolution for efficient continuous sign language recognition. Pattern Recognition, 145:109903

    2024

  12. [20]

    Lianyu Hu, Tongkai Shi, Wei Feng, Fanhua Shang, Liang Wan, and 1 others. 2024 b . Deep correlated prompting for visual recognition with missing modalities. Advances in Neural Information Processing Systems, 37:67446--67466

    2024

  13. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  14. [22]

    Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, and 1 others. 2024. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405.10739

    2024

  15. [23]

    Gyuwan Kim and Kyunghyun Cho. 2020. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003

    2020 arXiv

  16. [24]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579--9589

    2024

  17. [25]

    Carlos Lassance, Maroua Maachou, Joohee Park, and St \'e phane Clinchant. 2021. A study on token pruning for colbert. arXiv preprint arXiv:2112.06540

    2021 arXiv

  18. [26]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  19. [27]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024 b . Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299--13308

    2024

  20. [28]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024 c . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  21. [29]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597

    2023 arXiv

  22. [30]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024 d . Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  23. [31]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 e . Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947--22970

    2024

  24. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  25. [33]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  26. [34]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024 c . Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216--233. Springer

    2024

  27. [35]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212--46244

    2023

  28. [36]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  29. [37]

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. 2022. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309--12318

    2022

  30. [38]

    Xavier Ochoa, AWDG Charles Lang, and George Siemens. 2017. Multimodal learning analytics. The handbook of learning analytics, 1:129--141

    2017

  31. [39]

    Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. 2022. Less is more: Pay less attention in vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2035--2043

    2022

  32. [40]

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. Cake: Cascading and adaptive kv cache eviction with layer preferences. In International Conference on Learning Representation

    2025

  33. [41]

    Dhanesh Ramachandram and Graham W Taylor. 2017. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine, 34(6):96--108

    2017

  34. [42]

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937--13949

    2021

  35. [43]

    Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. Tokenlearner: Adaptive space-time tokenization for videos. Advances in neural information processing systems, 34:12786--12797

    2021

  36. [44]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388

    2024

  37. [45]

    Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, and Mi Zhang. 2025 a . Meda: Dynamic kv cache allocation for efficient multimodal long-context inference. In 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

    2025

  38. [46]

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, and 1 others. 2024. Efficient large language models: A survey. Journal of Machine Learning Research

    2024

  39. [47]

    Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and 1 others. 2025 b . D2o: Dynamic discriminative operations for efficient long-context inference of large language models. In International Conference on Learn...

    2025

  40. [48]

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, and 1 others. 2024 a . Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411

    2024 arXiv

  41. [49]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024 b . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  42. [50]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, and 1 others. 2024 c . Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pages 396--416. Springer

    2024

  43. [51]

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2025. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. Proceedings of the IEEE/CVF Conference...

    2025

  44. [52]

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023 a . Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265

    2023 arXiv

  45. [53]

    Peng Xu, Xiatian Zhu, and David A Clifton. 2023 b . Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113--12132

    2023

  46. [54]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  47. [55]

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025. Visionzip: Longer is better but not necessary in vision language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2025

  48. [56]

    Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. 2023. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.14074

    2023 arXiv

  49. [57]

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and 1 others. 2024 a . Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220

    2024

  50. [58]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  51. [59]

    Zhongzhi Yu, Zheng Wang, Yuhan Li, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reddy Bommu, Yang Zhao, and Yingyan Lin. 2024 b . Edge-llm: Enabling efficient large language model adaptation on edge devices via unified compression and adaptive layer voting. In Proceedings of the 61st AC...

    2024

  52. [60]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024 a . Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF...

    2024

  53. [61]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, and 1 others. 2024 b . Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813

    2024 arXiv

  54. [62]

    Cai Zefan, Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv e-prints, pages arXiv--2406

    2024

  55. [63]

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. 2024 a . [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818

    2024 arXiv

  56. [64]

    Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, and Liqiang Nie. 2024 b . Token-level correlation-guided compression for efficient multimodal document understanding. arXiv preprint arXiv:2407.14439

    2024 arXiv

  57. [65]

    Xiaodan Zhang, Aozhe Jia, Junzhong Ji, Liangqiong Qu, and Qixiang Ye. 2025 a . Intra-and inter-head orthogonal attention for image captioning. IEEE Transactions on Image Processing

    2025

  58. [66]

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and 1 others. 2025 b . Sparsevlm: Visual token sparsification for efficient vision-language model inference. International Conference on Mac...

    2025

  59. [67]

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.