Pith. sign in

REVIEW 5 major objections 6 minor 77 references

EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EffiVLM-Bench shows training-free LVLM acceleration is best chosen by task: vision-guided token pruning wins at extreme budgets, KV-cache compression wins on loyalty, and quantization preserves performance best.

desk verdict Useful benchmark of LVLM acceleration methods, but the central ranking needs the loyalty metric and reimplementation details pinned down before the numbers become citable. read the letter →

arxiv 2506.00479 v1 pith:CKRBALEE submitted 2025-05-31 cs.CL cs.CVcs.LG

Zekun Wang , Minghua Ma , Zexin Wang , Rongchuan Mu , Liping Shan , Ming Liu , Bing Qin This is my paper
classification cs.CLcs.CVcs.LG
keywords training-freeaccelerationlargevision-languagemodelstokencompressionKVcacheparametervisualsinktokensloyaltymetricEffiVLM-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish, with a unified benchmark called EffiVLM-Bench, that training-free acceleration methods for large vision-language models can be compared on four axes at once—performance, generalization, loyalty to the original model's predictions, and actual inference speed—and that no single method wins on all of them. Running seventeen benchmarks across three frontier LVLMs at retention budgets from 1% to 40%, it finds that under extreme budgets vision-encoder-guided token pruning (VisionZip, PruMerge+) consistently beats text-guided pruning (FastV), while KV-cache compression methods generalize better and stay more loyal to the original model's outputs. It also finds parameter compression—especially quantization—preserves performance better than token compression, and that the two families are orthogonal and can be combined. The practical payoff would be a principled recipe for choosing an acceleration strategy from task statistics (input length, output length, resolution) rather than from default method loyalty.

What carries the argument

The carrying object is the benchmark suite itself together with its four standardized metrics: overall performance $OP_{m,c}$ (RMS of per-benchmark relative scores), generalization $OG_c$ (coefficient of variation of relative performance across benchmarks), loyalty $OL_c$ (agreement between compressed and original predictions), and efficiency $OE_c$ (measured latency speedup). EffiVLM-Bench fixes uniform token-retention and KV-cache budgets across layers (1%, 5%, 10%, 20%, 40%) and uses three LVLMs—LLaVA-OneVision-7B, Qwen2-VL-7B, and InternVL2.5-38B—across 17 single-image, multi-image, and video benchmarks. That machinery lets the paper attribute differences to the compression mechanism rather than to model or task, and it supports the ablations (sink-token retention, layer-adaptive vs uniform budgets, head-adaptive selection, within-modality merging) that explain the observed rankings.

What would settle it

Run FastV at a 1% budget on LLaVA-OneVision-7B using its original per-layer adaptive budget and decoding-time eviction rules instead of the benchmark's uniform allocation, and compare on DocVQA and OCRBench; if FastV then matches or beats VisionZip's roughly 75% relative performance (versus 48%), the claim that vision-guided pruning is intrinsically better at extreme budgets would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that token-compression behavior in LVLMs is budget- and task-dependent, with a sharp divide at low budgets: pruning visual tokens inside the vision encoder (VisionZip, PruMerge+) retains far more relative performance than pruning inside the LLM using text-guided attention (FastV)—for instance 75% vs 48% on LLaVA-OneVision-7B at a 1% budget—because text-guided selection tends to discard visual "sink" tokens that carry disproportionate weight. KV cache compression methods (H2O, SnapKV, PyramidKV, VL-Cache) show better generalization across benchmarks and better loyalty to the original model's predictions than token pruning, at the cost of limited time-to-first-token speedup. Parameter compression (pruning and quantization) preserves overall performance much better than token compression even at 50% or 2:4 sparsity, with AWQ/GPTQ quantization ahead of pruning, and it combines orthogonally with token compression. The paper further argues that layer-adaptive budget allocation is not automatically beneficial in LVLMs—aggressive front-loading of early layers starves later ones—and that head-adaptive token selection and modality-specific merging of evicted tokens both help.

Load-bearing premise

The rankings rest on the assumption that the benchmark's standardized reimplementations of each published method preserve what made each method work; if the uniform budgets across layers and the decision not to evict during decoding misrepresent a method's original design, the relative ordering—and the main observations—could change.

Editorial extensions

If this is right

  • For short-answer VQA at low budgets, vision-encoder-guided token pruning should be preferred because it cuts time-to-first-token by up to about 3.2× while keeping far more accuracy than text-guided pruning.
  • For long-output tasks like LLaVA-Wilder or ImageDC, KV-cache compression is the safer choice: better loyalty and generalization, with decoding speedups comparable to token pruning.
  • Parameter compression via quantization (AWQ, GPTQ) preserves performance well enough to be applied before token pruning, and the two can be stacked—combining AWQ with SnapKV gives a 1.65× speedup over AWQ alone at roughly stable performance.
  • Layer-adaptive KV budgets should be rebalanced: an 80% uniform / 20% adaptive split outperforms the fully adaptive VL-Cache allocation at 5% budget on document and OCR tasks.
  • Head-adaptive token selection improves KV-cache compression at 1% budgets across H2O, SnapKV, and VL-Cache, so shared-head selection is a needless handicap under extreme compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the sink-token mechanism predicts that any text-guided pruning method—not just FastV—can be repaired by explicitly reserving a small fraction of the budget for the highest-attention visual tokens; that is directly testable on the benchmark's own 1% setting.
  • If the Pareto-front observations transfer, task statistics (input token count, expected output length, image resolution) could be fed into a simple decision rule that picks between token pruning and KV-cache compression without per-method tuning, which the paper does not itself propose.
  • The loyalty metric, measured here on generative tasks, could be used as a cheap proxy for detecting compression-induced hallucination or task shift on new benchmarks, since low $OL_c$ flags when a method changes what the model would have said.
  • The finding that LLaVA-OneVision (thousands of visual tokens) tolerates 1% budgets better than Qwen2-VL (fewer tokens) suggests a token-count-normalized budget, rather than a fixed percentage, is the fairer comparison across architectures; the paper reports the raw numbers but does not draw this design recommendation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces EffiVLM-Bench, a unified evaluation framework for training-free acceleration of large vision-language models (LVLMs). It evaluates token pruning (FastV, VisionZip, PruMerge+, plus iLLaVA and SparseVLM in the appendix), KV-cache compression (StreamingLLM, H2O, SnapKV, PyramidKV, LOOK-M, VL-Cache), and parameter compression (EcoFLAP, Wanda, SparseGPT, AWQ, GPTQ) across three LVLMs and 17 benchmarks. The paper defines four metrics—performance (OP), generalization (OG), loyalty (OL), and efficiency (OE)—and reports extensive tables and Pareto-style trade-off analyses. Its main observations are that vision-encoder-guided token pruning outperforms text-guided pruning at very low budgets, that KV-cache compression methods generalize better and are more loyal than token pruning methods, that the choice between pruning and cache compression depends on the TTFT versus decoding bottleneck, and that these trends extend to multi-image and video tasks. The paper also proposes a hybrid layer-budget allocation for VL-Cache and a modality-specific merge variant of LOOK-M.

Significance. If the empirical claims are made fully verifiable, this would be a useful benchmark contribution. The paper broadens the evaluation of training-free LVLM acceleration beyond a single architecture and accuracy metric, and the analyses of layer/head adaptivity, visual attention sinks, and cross-modal merging generate concrete, testable hypotheses for method design. The experimental scope is large—three model families, 17 benchmarks, and both token- and parameter-level methods—and the appendix contains abundant raw results that can support future comparisons. However, the central ranking currently rests on an underspecified loyalty metric, a generalization metric that does not measure cross-model variance as claimed, and a reimplementation harness whose exact modifications to published methods are not described. The paper also promises open-source code but provides only a project page URL. These issues must be resolved before the headline observations can be relied upon.

major comments (5)
  1. [Section 3.1, Eq. (3)] The loyalty metric OLc is defined as the expectation of I(P^{m,c}_b, P^m_b), but the agreement function I(P1,P2) is never specified anywhere in the paper. The Table 2 caption further states that loyalty is computed on a subset (MathVista, LLaVA-Wilder, MMVet), which is inconsistent with the all-benchmark expectation in Eq. (3). Because Observation 2's claim that KV-cache compression is more loyal than token pruning depends directly on this metric, the paper must define I concretely (e.g., exact-match accuracy, token-level F1, or a GPT-judged similarity score), state how predictions were collected and compared, and report the per-benchmark loyalty values.
  2. [Section 3.1, Eq. (2)] The generalization metric OGc is introduced as the coefficient of variation 'across benchmarks and models,' but the formula applies σb only after averaging over models inside the numerator, while the denominator is a pooled average over b and m. Model-to-model variation therefore never enters the variance term; a method that is stable across benchmarks but highly unstable across models would receive the same OG as one that is stable across both. Please either redefine OG to include a total or two-way variance over (b,m), or revise the prose to say that generalization is measured across benchmarks after model-averaging. The current definition weakens the cross-model component of Observation 2.
  3. [Appendix A.4 and Section 4.1] The paper states that 'we standardize average token retention rates across layers' and that methods were 'adapted consistently across all evaluated LVLMs,' but it never specifies what this standardization does to FastV. In the original FastV, visual tokens are pruned once after the second LLM layer and the same subset is used for all subsequent layers; if the harness instead re-selects tokens or enforces a per-layer budget in every layer, the 1% rankings in Table 1 could be an artifact of the harness rather than a property of the algorithms. Please provide the exact adaptation for each method (layer index, re-selection schedule, threshold computation), release the code and configuration files, and include a sensitivity check showing that the qualitative rankings are unchanged when each method is run in its original unmodified form where feasible.
  4. [Section 5.1, Table 5] The hybrid allocation strategy is introduced and the U-80% variant is selected as best on the same set of benchmarks and models (LLaVA-OneVision-7B and Qwen2-VL-7B) that are then used to demonstrate its benefit. Because the choice among A-Only, U-40%, and U-80% is made after seeing these results, the comparison is subject to selection-on-the-test-set bias. Table 5 also omits InternVL2.5-38B, so the cross-model generality of the recommendation is not established. Please validate the strategy on held-out benchmarks/models or report it explicitly as a post-hoc observation rather than a tested recommendation.
  5. [Section 4.1, Tables 1 and 15] Headline point estimates such as 'FastV retains 48% while VisionZip retains 75%' are reported without error bars, repeated runs, or seed information, and several benchmarks (MathVista, MMVet, LLaVA-Wilder, ImageDC) use GPT-based scoring that is stochastic. For a benchmark whose purpose is comparative ranking, at least the headline numbers in Tables 1 and 2 need variance estimates, such as bootstrap over evaluation samples or multiple evaluation runs, to establish that the observed gaps are not within noise.
minor comments (6)
  1. [Section 3.1, Eq. (4)] The efficiency metric uses T^{m,c}_b in the displayed equation but the prose refers to T^{*,c} and T^{*}; please harmonize the notation.
  2. [Section 4.2, Observation 5] The sentence 'the two types of compression are orthogonal that can be effectively combined, Crucially, these two compression types are orthogonal...' is duplicated and grammatically broken; please fix.
  3. [Section 5.3, Table 7] The ablation requires a definition of the 'top 10% most critical visual tokens'; without specifying how this set is computed, the evidence for the visual-sink hypothesis cannot be evaluated.
  4. [Appendix A.4] Please report the lmms-eval version and the exact prompt templates used for GPT-scored benchmarks, since those scores are prompt-sensitive and important for reproducibility.
  5. [References] The H2O reference appears twice as Zhang et al. 2023a and Zhang et al. 2023b; please consolidate and ensure all citations have complete bibliographic information.
  6. [Title and abstract] The benchmark name is rendered inconsistently as 'EffiVLM-Bench' and 'EFFIVLM-B ENCH'; please choose one consistent spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper benchmarks external methods on external benchmarks; the U-80% hybrid and LOOK-M ablations are empirical comparisons, not fitted predictions, and self-citations are non-load-bearing.

full rationale

EffiVLM-BENCH is a comparative benchmark study. The central observations (Observations 1-5) are empirical rankings of externally published token-pruning, KV-cache, pruning, and quantization methods measured on standard benchmarks through the transparent normalized metrics of Eqs. (1)-(4). No target result is derived from a fitted parameter, and no evaluation quantity is defined in terms of the conclusion it supports. The Section 5.1 hybrid allocation experiment compares three allocation variants (A-Only, U-40%, U-80%) and reports which performed best on the same table; this is an exploratory ablation, not a prediction validated on withheld data, so it does not constitute a fit-then-predict cycle. The Section 5.3 attention-sink ablations and Section 5.4 LOOK-M modality-specific merging are controlled interventions that test stated hypotheses rather than renaming or re-deriving the hypotheses. The paper's Limitations section openly confines scope to a subset of models, tasks, and training-free methods, which is a scope limitation, not circularity. The only self-citations (e.g., Wang et al., 2024b for avoiding structured pruning without recovery training, and Qiu et al., 2025 as future work) are ancillary and not used to force the paper's conclusions. The undefined agreement function I in Eq. (3) is an incompleteness or correctness concern, not a circular reduction. Because no load-bearing step reduces by definition or by self-citation to its own output, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or model entities are introduced. The empirical claims rest on data, benchmark choices, and standardized implementations rather than on fitted model parameters; the two hand-set evaluation choices above are the main adjustable knobs.

free parameters (2)
  • Recent-window fraction for KV compression = 10%
    Appendix A.4 sets a 10% recent window for all KV methods; a standardization choice that changes results and is not the original setting of every method.
  • Hybrid allocation uniform fraction U = 40% and 80% tested, 80% best
    Section 5.1 and Table 5: U=80% is chosen on the same benchmarks used to present the improvement; it is hand-tuned rather than derived.
assumptions (3)
  • domain assumption Benchmark accuracy and GPT-based scores measure capability relevant to deployment
    Used throughout; no validation that the 17 tasks predict real-world performance, and judge details for GPT-scored tasks are omitted.
  • domain assumption The 128-sample COCO-Caption set is adequate for pruning and quantization calibration
    Appendix A.4; calibration data affects all parameter compression results.
  • domain assumption Excluding decoding-time KV compression isolates the prefill effect fairly
    Appendix A.4; this deviates from full method designs and could disadvantage methods whose gains come from decoding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/CKRBALEE

@misc{pith2026250600479,
  author       = {Pith},
  title        = {Pith review of: EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKRBALEE}},
  note         = {Machine review of arXiv:2506.00479}
}
read the original abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success, yet their significant computational demands hinder practical deployment. While efforts to improve LVLM efficiency are growing, existing methods lack comprehensive evaluation across diverse backbones, benchmarks, and metrics. In this work, we systematically evaluate mainstream acceleration techniques for LVLMs, categorized into token and parameter compression. We introduce EffiVLM-Bench, a unified framework for assessing not only absolute performance but also generalization and loyalty, while exploring Pareto-optimal trade-offs. Our extensive experiments and in-depth analyses offer insights into optimal strategies for accelerating LVLMs. We open-source code and recipes for EffiVLM-Bench to foster future research.

Figures

Figures reproduced from arXiv: 2506.00479 by the authors.

Figure 1
Figure 1. Performance comparison of KV cache compression methods across multiple benchmarks and models. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Layer-wise Budget Allocation of VL-Cache [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Visualizations of attention weight heatmaps from layers 0, 14, and 27 of Qwen2-VL-7B. Different heads [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Budgets distribution of LLaVA-OV-7B and Qwen2-VL-7B across 8 benchmarks under 5% budget. Each [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Budgets distribution of InternVL2.5-38B across 8 benchmarks under 5% budget. Each subplot shows a [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Visualization of token selection strategies on ChartQA with Qwen2-VL-7B under 10% budget. Left to [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Visualization of VL-Cache selected visual tokens in layers 0–2 on LLaVA-OneVision-7B. Highlighted [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Visualization of token selection strategies on DocVQA and MMMU with Qwen2-VL-7B under 10% [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Heatmap visualizations of 15 randomly selected examples from the ChartQA benchmark on LLaVA [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 20 canonical work pages

  1. [1]

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. https://doi.org/10.1609/aaai.v38i10.28960 Fluctuation-based adaptive structured pruning for large language models . In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympo...

    doi:10.1609/aaai.v38i10.28960 2024

  2. [2]

    Anthropic. 2024. https://www.anthropic.com/news/developing-computer-use Developing a computer use model

    work page 2024

  3. [3]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations

    work page

  4. [4]

    Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, and Xiang Bai. 2024 a . https://doi.org/10.48550/ARXIV.2410.16236 Llava-kd: A framework of distilling multimodal large language models . CoRR, abs/2410.16236

    arXiv doi:10.48550/arxiv.2410.16236 2024

  5. [5]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and 1 others. 2024 b . Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069

    arXiv 2024

  6. [6]

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. Honeybee: Locality-enhanced projector for multimodal LLM . In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 13817--13827. IEEE

    work page 2024

  7. [7]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024 a . Allava: Harnessing gpt4v-synthesized data for A lite vision-language model. CoRR, abs/2402.11684

    arXiv 2024

  8. [8]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024 b . An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19--35. Springer

    work page 2024

Show all 77 references
  1. [9]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024 c . Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330

    2024 arXiv

  2. [10]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C. Lawrence Zitnick. 2015. https://api.semanticscholar.org/CorpusID:2210455 Microsoft coco captions: Data collection and evaluation server . ArXiv, abs/1504.00325

    2015 arXiv

  3. [11]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024 d . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    2024 arXiv

  4. [12]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and 1 others. 2023. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886

    2023 arXiv

  5. [13]

    Tri Dao. 2024. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR)

    2024

  6. [14]

    Timoth \' e e Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  7. [15]

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, and 1 others. 2023. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Adva...

    2023

  8. [16]

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen - Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison - B...

    2024 doi arXiv

  9. [17]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337. PMLR

    2023

  10. [18]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    2022 arXiv

  11. [19]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large language models . Preprint, ...

    2024 arXiv

  12. [20]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive KV cache compression for llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  13. [21]

    Lianyu Hu, Fanhua Shang, Liang Wan, and Wei Feng. 2024. illava: An image is worth fewer than 1/3 input tokens in large multimodal models. CoRR, abs/2412.06263

    2024

  14. [22]

    Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaoshen Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. 2024. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876

    2024 arXiv

  15. [23]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  16. [24]

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. https://arxiv.org/abs/1603.07396 A diagram is worth a dozen images . Preprint, arXiv:1603.07396

    2016 arXiv

  17. [25]

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. 2024 a . https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/ Llava-next: Stronger llms supercharge multimodal capabilities in the wild

    2024

  18. [26]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024 b . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  19. [27]

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2024 c . Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392

    2024 arXiv

  20. [28]

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. 2024 d . Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814

    2024 arXiv

  21. [29]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 e . Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  22. [30]

    Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, and 3 others. 2025. Percep...

    2025 arXiv

  23. [31]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

    2024

  24. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 26286--26296

    2024

  25. [33]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023

    2023

  26. [34]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 b . https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    2024 arXiv

  27. [35]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024 c . https://doi.org/10.1007/s11432-024-4235-6 Ocrbench: on the hidden mystery of ocr in large multimodal models . Science China Information Sc...

    2024 doi

  28. [36]

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng - Yu Hsieh, De - An Huang, An - Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, and 8 others. 2024 d . https:...

    2024 doi arXiv

  29. [37]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024 a . https://doi.org/10.48550/ARXIV.2403.05525 Deepseek-vl: Towards real-world vision-languag...

    2024 doi arXiv

  30. [38]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024 b . Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Re...

    2024

  31. [39]

    Yaxin Luo, Gen Luo, Jiayi Ji, Yiyi Zhou, Xiaoshuai Sun, Zhiqiang Shen, and Rongrong Ji. 2024. https://doi.org/10.48550/arXiv.2410.13859 \( \) -mod: Exploring mixture-of-depth adaptation for multimodal large language models . CoRR, abs/2410.13859

    2024 doi arXiv

  32. [40]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  33. [41]

    Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. 2020. Docvqa: A dataset for vqa on document images. corr abs/2007.00398 (2020). arXiv preprint arXiv:2007.00398

    2020 arXiv

  34. [42]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o Hello gpt-4o

    2024

  35. [43]

    OpenAI. 2025. https://openai.com/index/introducing-operator Introducing operator

    2025

  36. [44]

    Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, and 1 others. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. arXiv preprint arXiv:2505.06708

    2025 arXiv

  37. [45]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388

    2024

  38. [46]

    Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Bolin Li, Zhelun Yu, Si Liu, Hongsheng Li, and Hao Jiang. 2024. https://doi.org/10.48550/ARXIV.2408.15881 Llava-mod: Making llava tiny via moe know...

    2024 doi arXiv

  39. [47]

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317--8326

    2019

  40. [48]

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, and 1 others. 2023. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449

    2023 arXiv

  41. [49]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    2023 arXiv

  42. [50]

    Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. 2023. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models. arXiv preprint arXiv:2310.02998

    2023 arXiv

  43. [51]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context ....

    2024 arXiv

  44. [52]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwen2.5-vl/ Qwen2.5-vl

    2025

  45. [53]

    Rub \`e n Tito, Dimosthenis Karatzas, and Ernest Valveny. 2022. Hierarchical multimodal transformers for multi-page docvqa. arXiv preprint arXiv:2212.05935

    2022 arXiv

  46. [54]

    Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. 2024. Vl-cache: Sparsity and modality-aware kv cache compression for vision-language model inference acceleration. arXiv preprint arXiv:2410.23317

    2024 arXiv

  47. [55]

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. 2024. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4065--4078

    2024

  48. [56]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  49. [57]

    Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. 2023 a . Efficientvlm: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-1...

    2023

  50. [58]

    Yuxin Wang, Minghua Ma, Zekun Wang, Jingchang Chen, Huiming Fan, Liping Shan, Qing Yang, Dongliang Xu, Ming Liu, and Bing Qin. 2024 b . https://doi.org/10.48550/ARXIV.2409.13199 CFSP: an efficient structured pruning framework for llms with coarse-to-fine activation information...

    2024 doi arXiv

  51. [59]

    Zekun Wang, Jingchang Chen, Wangchunshu Zhou, Ming Liu, and Bing Qin. 2023 b . https://doi.org/10.48550/ARXIV.2305.15033 Smarttrim: Adaptive tokens and parameters pruning for efficient vision-language models . CoRR, abs/2305.15033

    2023 doi arXiv

  52. [60]

    Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, and Furu Wei. 2021. https://arxiv.org/abs/2112.08723 Distilled dual-encoder model for vision-language understanding . CoRR, abs/2112.08723

    2021 arXiv

  53. [61]

    https://x.ai/blog/grok-1.5v Grok-1.5 vision preview

    x.ai. https://x.ai/blog/grok-1.5v Grok-1.5 vision preview

  54. [62]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv

    2023

  55. [63]

    McAuley, and Furu Wei

    Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, and Furu Wei. 2021. Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual E...

    2021

  56. [64]

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024 a . https://doi.org/10.48550/ARXIV.2412.04454 Aguvis: Unified pure vision agents for autonomous GUI interaction . CoRR, abs/2412.04454

    2024 doi arXiv

  57. [65]

    Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2024 b . https://doi.org/10.48550/ARXIV.2407.21018 Think: Thinner key cache by query-driven pruning . CoRR, abs/2407.21018

    2024 doi arXiv

  58. [66]

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2024. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467

    2024

  59. [67]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800

    2024 arXiv

  60. [68]

    JiangYong Yu, Sifan Zhou, Dawei Yang, Shuo Wang, Shuoyu Li, Xing Hu, Chen Xu, Zukang Xu, Changyong Shu, and Zhihang Yuan. 2025. Mquant: Unleashing the inference potential of multimodal large language models via full static quantization. arXiv preprint arXiv:2502.00425

    2025 arXiv

  61. [69]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. https://arxiv.org/abs/2308.02490 Mm-vet: Evaluating large multimodal models for integrated capabilities . Preprint, arXiv:2308.02490

    2023 arXiv

  62. [70]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-...

    2024

  63. [71]

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2024 a . https://arxiv.org/abs/2407.12772 Lmms-eval: Reality check on the evaluation of large multimodal models . Preprint, arX...

    2024 arXiv

  64. [72]

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and 1 others. 2024 b . Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417

    2024 arXiv

  65. [73]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\' e , Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023 a . https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb28...

    2023

  66. [74]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, and 1 others. 2023 b . H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Informat...

    2023

  67. [75]

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. 2024. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289

    2024 arXiv

  68. [76]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  69. [77]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.