Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read STAR claims that two-stage attention-guided token pruning removes up to 95% of visual tokens in large vision–language models while retaining at least 97.95% of baseline performance, and that this beats one-stage methods at high pruning…

desk verdict A promising two-stage pruning idea, but the method as written cannot be computed during inference: Eq. (5) uses masked image-to-text attention and response tokens that don't exist at prefill, and the FLOPs math assumes per-layer pruning rather than the one-time pruning described. read the letter →

arxiv 2505.12359 v1 pith:KMFTUXUW submitted 2025-05-18 cs.LG cs.CV

Yichen Guo , Hanze Li , Zonghao Zhang , Jinhao You , Kai Tang , Xiande Huang This is my paper · ORCID
classification cs.LGcs.CV
keywords visualtokenpruninglargevision-languagemodelstraining-freeaccelerationcross-modalattentionself-attentionimportancescoresreductionLLaVAFLOPs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STAR claims that visual token pruning in large vision–language models fails when it happens in one place, and that a two-stage schedule can cut most visual tokens without losing task performance. The first stage prunes conservatively using visual self-attention right after the vision encoder; the second stage prunes aggressively in an intermediate decoder layer using cross-modal attention between the remaining visual tokens and the question-and-response text. If this is right, training-free acceleration becomes practical at the high pruning ratios where existing single-stage methods degrade sharply. The paper reports up to 95% token reduction, more than 28.7% FLOPs savings, and at least 97.95% of baseline accuracy across seven vision–language benchmarks on LLaVA-1.5 and LLaVA-NeXT.

What carries the argument

The load-bearing object is the two-stage importance score. Stage 1 uses the visual encoder's self-attention matrix $A\in\mathbb{R}^{L_v\times L_v}$ and scores token $i$ by $r_i=\frac{1}{L_v}\sum_{j}A_{ij}$; a dynamic threshold $\tau$ keeps the top $(1-R)L_v$ tokens. Stage 2 uses the cross-modal attention map at decoder layer $K$, $C_K=\mathrm{Softmax}(H_v\tilde{H}_q^{\top}/\sqrt{d})$, and scores each visual token by its mean attention to all text positions; the $P\cdot L_v$ lowest-scoring tokens are dropped. The two stages are complementary: the first reduces input length before the projector, the second removes task-irrelevant tokens after vision–language fusion has matured. The theoretical FLOPs account treats each pruned token as saving $6D^2$ plus $2N_iD$ per layer, giving the paper's claimed $\Delta_{\mathrm{total}}$ sum over stages.

What would settle it

Run the exact STAR pipeline on LLaVA-1.5-7B with $H_{\mathrm{resp}}$ replaced by a first-pass draft response (no ground-truth answers), measure VQAv2, GQA, and MME accuracy and wall-clock time, and compare against the paper's tables. If accuracy drops materially or per-image latency rises by the extra generation pass, the central claim as stated fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that pruning visual tokens twice, with different attention signals at different depths, preserves more task-critical visual information than either signal alone. In stage one, tokens are scored by mean self-attention weight inside the vision encoder and the lowest-scoring fraction is dropped before the LLM. In stage two, at an intermediate decoder layer $K$, surviving tokens are scored by their average cross-modal attention to the concatenated query and response stream $\tilde{H}_q=[H_q;H_{\mathrm{resp}}]$, and the least-attended fraction is discarded. The authors report that this schedule keeps near-baseline accuracy even with 29 of 576 tokens remaining, that it consistently beats single-stage baselines at the same token budgets, and that on MME it can exceed the unpruned baseline under heavy pruning. The proposed explanation is that conservative self-attention pruning removes low-level noise while the later cross-modal stage keeps only tokens aligned with the actual question-and-answer context.

Load-bearing premise

The load-bearing premise is that the cross-modal stage can know the generated response tokens before they are generated; if those tokens come from the answer key rather than from the model itself, the reported accuracy and FLOPs savings do not transfer to normal single-pass inference.

Editorial extensions

If this is right

  • At equal remaining-token budgets, STAR reports near-baseline accuracy on VQAv2, GQA, and MME where FastV, SparseVLM, and FasterVLM degrade, and at 29 tokens it can surpass the unpruned baseline on MME.
  • Because STAR is training-free and plug-and-play, the same procedure transfers across model scales (7B, 13B) and input resolutions (LLaVA-NeXT) without retuning.
  • The enforced schedule $R<P$—conservative early pruning, aggressive late pruning—is claimed to be the reason high reduction ratios remain safe; reversing the schedule should lose the benefit.
  • The cumulative FLOPs reduction grows superlinearly in the number of pruned tokens because each pruned token removes both linear projection and quadratic attention work per layer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The second-stage score needs $H_{\mathrm{resp}}$, the generated response, which does not exist when the prompt is first processed; the paper never says whether it uses a draft pass or the benchmark's answer. A testable repair is to generate a short draft with the unchanged model, then prune using that draft, and compare accuracy and latency end-to-end.
  • If response-aware scores are replaced by the model's own draft, STAR becomes a two-pass method, and the paper's per-image latency and throughput numbers likely undercount the real cost.
  • The depth $K$ and the stage-wise split may be task-dependent: fine-grained reading tasks like TextVQA might need later cross-modal pruning than holistic hallucination checks like POPE.
  • The success of a conservative-then-aggressive schedule suggests that pruning guidance should match the maturity of vision–language fusion at each layer, a principle that could apply to other modalities such as audio or depth tokens in future multimodal models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes STAR, a training-free, plug-and-play method to reduce visual tokens in large vision-language models in two stages: Stage I prunes tokens after the vision encoder using averaged visual self-attention scores, and Stage II prunes a second batch of tokens at an intermediate decoder layer using cross-modal attention from visual tokens to the concatenated question and response tokens. The authors report experiments on LLaVA-1.5-7B/13B and LLaVA-NeXT-7B across seven VQA benchmarks, claiming up to 95% token pruning, a 28.7% FLOPs reduction, and better accuracy preservation than FastV, FasterVLM, and SparseVLM at high pruning ratios. A FLOPs analysis in Section 4.3 and a latency comparison in Table 3 are presented to support the efficiency claims.

Significance. If the method worked as described, the two-stage design would be a useful contribution: it is training-free, introduces no learned parameters, and the empirical tables show STAR maintaining near-baseline scores at much higher pruning ratios than the single-stage baselines. The paper is clearly written, and the experimental breadth across model scales and benchmarks is a strength. However, the feasibility of Stage II is unresolved: Eq. (5) uses response tokens that do not exist at prefill and an image-to-text attention block that the causal mask never computes. Until this is specified and costed, the significance of the accuracy and speedup numbers is conditional, and the FLOPs accounting in Section 4.3 is inconsistent with the stated pruning schedule.

major comments (3)
  1. [Section 4.2, Eq. (5)] The second-stage importance score is not computable in the inference protocol claimed by the paper. In LLaVA-1.5 and LLaVA-NeXT, the decoder is a causal transformer with input order [image tokens; text prompt], so the causal mask prevents image tokens from attending to text tokens; the map C_K = Softmax(H_v H_tilde_q^T / sqrt(d)) is not produced by the standard forward pass. Moreover, H_resp does not exist during prefill. If H_resp is obtained from a draft pass or a gold-answer pass, that pass costs roughly the full baseline FLOPs, and the reported 28.7% FLOPs reduction and the latencies in Table 3 cannot describe single-pass inference. The manuscript must specify exactly how the cross-modal score is computed, where H_resp comes from, and the end-to-end cost of that computation; otherwise the central speedup and accuracy claims are not verifiable.
  2. [Section 4.3, Eqs. (7)-(9)] The FLOPs derivation assumes that N_i = R L0_v tokens are pruned at every decoder layer i=1..K and N_i = P L0_v tokens at every layer i=K+1..Omega. This does not match the described schedule of one prune immediately after the projector and one prune at layer K. Under the actual schedule, Stage I reduces the sequence length before the decoder, so Stage II operates on (1-R)L0_v remaining tokens, and pruning at one layer affects that layer and all subsequent layers rather than being charged independently at every layer. The formula therefore overcounts the FLOPs savings, and the abstract's claim of a 28.7% reduction is not supported by the method as described.
  3. [Table 3 and Section 5.3] The latency and throughput numbers do not support the claimed acceleration. At 288 remaining tokens, STAR reduces FLOPs by 28.7% relative to baseline but increases latency by 0.7% (134.68 ms vs. 135.58 ms); even at 29 tokens, the latency is 121.80 ms, which is slower than FasterVLM at 288 tokens (103.35 ms). The authors should report latency for the exact pipeline including the cross-modal scoring computation, and explain why the FLOPs savings do not translate to wall-clock speedups.
minor comments (4)
  1. [Section 4.1, Eq. (3)] The threshold definition can retain fewer than the intended (1-R)L_v tokens when multiple tokens share the same score at the boundary; the tie-handling rule should be stated.
  2. [Section 5.1] The text lists eight benchmarks including MM-Vet, but the experimental tables report only seven; the abstract also says seven tasks, so either MM-Vet results should be added or the list corrected.
  3. [Figure 2] The two panels lack axis labels and a description of the underlying evaluation setup, which makes the motivating claims about pruning ratio and pruning layer difficult to verify.
  4. [Throughout] There are several naming inconsistencies, such as 'LLaVA-NeXT' versus 'LLaV A-NeXT', and the affiliation 'De Artificial Intelligence Lab' appears malformed; these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: STAR's importance scores come from the model's own attention and its FLOPs savings are arithmetic over pruned token counts; no fitted parameter is renamed as a prediction.

full rationale

The paper's derivation chain is self-contained rather than circular. Stage I importance (Eq. 1 and the row-averaged score r_i) is defined directly from the visual encoder's self-attention map, and Stage II importance (Eq. 5) is defined from a cross-modal attention map; neither definition presupposes the benchmark outcomes reported later. The pruning thresholds R and P are user-specified token-retention ratios rather than parameters fitted to the evaluation data, and the FLOPs reduction in Section 4.3 is exact arithmetic: with N_i visual tokens pruned at layer i, the per-layer saving 6*N_i*D^2 + 2*N_i^2*D follows from the stated 6*L*D^2 + 2*L^2*D baseline. Accuracy and latency are external measurements, not consequences of the definitions. The only self-referential element is that the same model's attention is used both to choose tokens and to measure quality; that is inherent to training-free attention-based pruning and is not a circular reduction. A separate concern is that Eq. (5) requires H_resp, the generated response, which is not available during single-pass prefill; this is an implementation/feasibility gap and a possible target-information leak if H_resp is a gold answer, but it does not make any derived quantity equal to its input by construction, so it does not change the circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method introduces no free parameters beyond the hand-set R, K, and unreported P. The key assumptions are the attention-as-importance proxy and the availability of response tokens. The FLOPs formula adds a stylized transformer assumption.

free parameters (3)
  • Stage-1 pruning ratio R = 0.1 (Table 1)
    The dynamic threshold in Eq. (3) retains the top (1-R)L_v tokens; R is fixed at 0.1 for all reported experiments without ablation.
  • Pivot layer K = 14 (Table 1)
    The cross-modal pruning is applied at decoder layer K; K is set to 14 for both LLaVA-1.5 and LLaVA-NeXT without a sensitivity study.
  • Stage-2 pruning ratio P = not reported
    P controls how many of the remaining tokens are pruned at layer K to reach the final token counts (288, 115, 58, 29); the values are never listed, so the experiments cannot be reproduced exactly.
assumptions (3)
  • domain assumption Attention weights are a valid proxy for visual token importance.
    The entire method scores tokens by averaged self-attention in Eq. (3) and cross-attention in Eq. (5); the paper does not validate this proxy beyond its own experiments.
  • ad hoc to paper Generated response embeddings H_resp are available when computing the cross-modal attention in Eq. (5).
    This is required by Section 4.2 but is not explained for autoregressive inference, where the response is not yet generated.
  • standard math The transformer FLOPs per layer follow F_base = 6LD^2 + 2L^2D with FFN inner size equal to hidden width D.
    This stylized model is assumed in Section 4.3; actual LLaVA models have larger FFN dimensions, so the theoretical FLOPs numbers are approximate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference." pith.science (2026). https://pith.science/paper/KMFTUXUW

@misc{pith2026250512359,
  author       = {Pith},
  title        = {Pith review of: STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KMFTUXUW}},
  note         = {Machine review of arXiv:2505.12359}
}
read the original abstract

Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.

Figures

Figures reproduced from arXiv: 2505.12359 by the authors.

Figure 1
Figure 1. The framework of STAR. The two-stage pruning includes visual self-attention pruning to remove redundant tokens, followed by cross-modal attention pruning to discard task-irrelevant ones. early-stage pruning within the decoder may hinder the model’s ability to fully exploit visual cues via cross-modal attention [26, 28]. In this paper, we propose STAR (Stage-wise Attention-guided token Reduction), a novel framework f… view at source ↗
Figure 2
Figure 2. (a) Model performance under different pruning ratio of image tokens before the LLM [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The implementation of STAR mechanism. In the first stage, an importance score indicator is computed to guide the removal of low-level redundant features. In the second stage, we use the cross-modal importance indicator to select important visual tokens, which is strongly aligned to input query and generated response. 4.1 Visual Self-Attention The visual encoder produces a set of token embeddings Hv ∈ R Lv×d and the … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the STAR method: Examples generated using the Vicuna-7B model [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

    cs.CV 2026-07 conditional novelty 5.5 of 10

    Restructuring visual tokens via cross-modal prune–merge–refine consistently lowers hallucination rates on MME, POPE and AMBER across four 7B LVLMs without any training.

Reference graph

Works this paper leans on

46 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  5. [5]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  6. [6]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  7. [7]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng ...

    arXiv 2024

  8. [8]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    arXiv 2024

Show all 46 references
  1. [9]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  2. [10]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  3. [11]

    Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    2023

  4. [12]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–6913, 2017. 10

    2017

  6. [14]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608–3...

    2018

  7. [15]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700–6709, 2019

    2019

  8. [16]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8317–8326, 2019

    2019

  9. [17]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35: 2507–...

    2022

  10. [18]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021

    2021

  11. [19]

    Not all patches are what you need: Expediting vision transformers via token reorganizations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800, 2022

    2022 arXiv

  12. [20]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  13. [21]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35: 237...

    2022

  14. [22]

    Spvit: Enabling faster vision transformers via latency- aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency- aware soft token pruning. In European conference on computer vision, pages 620–640. Springer, 2022

    2022

  15. [23]

    Tokenpacker: Efficient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024

    2024 arXiv

  16. [24]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022

    2022 arXiv

  17. [25]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024

    2024

  18. [26]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision , pages 19–35. Springer, 2024

    2024

  19. [27]

    [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818, 2024

    2024 arXiv

  20. [28]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024. 11

    arXiv

  21. [29]

    Inference optimal vlms need only one visual token but larger models

    Kevin Y Li, Sachin Goyal, Joao D Semedo, and J Zico Kolter. Inference optimal vlms need only one visual token but larger models. arXiv preprint arXiv:2411.03312, 2024

    2024 arXiv

  22. [30]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  23. [31]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  24. [32]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/...

    2023

  25. [33]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023

    2023

  26. [34]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  27. [35]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  28. [36]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  29. [37]

    Dymu: Dynamic merging and virtual unmerging for efficient vlms

    Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, and Ran Xu. Dymu: Dynamic merging and virtual unmerging for efficient vlms. arXiv preprint arXiv:2504.17040, 2025

    2025 arXiv

  30. [38]

    Lookupvit: Compressing visual information to a limited number of tokens

    Rajat Koner, Gagan Jain, Prateek Jain, V olker Tresp, and Sujoy Paul. Lookupvit: Compressing visual information to a limited number of tokens. In European Conference on Computer Vision, pages 322–337. Springer, 2024

    2024

  31. [39]

    Diffrate: Differentiable compression rate for efficient vision transformers

    Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 17164–1...

    2023

  32. [40]

    Token compensator: Altering inference cost of vision transformer without re-tuning

    Shibo Jie, Yehui Tang, Jianyuan Guo, Zhi-Hong Deng, Kai Han, and Yunhe Wang. Token compensator: Altering inference cost of vision transformer without re-tuning. In European Conference on Computer Vision, pages 76–94. Springer, 2024

    2024

  33. [41]

    Pumer: Pruning and merging tokens for efficient vision language models

    Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi. Pumer: Pruning and merging tokens for efficient vision language models. arXiv preprint arXiv:2305.17530, 2023

    2023 arXiv

  34. [42]

    Ivtp: Instruction-guided visual token pruning for large vision-language models

    Kai Huang, Hao Zou, Ye Xi, BoChen Wang, Zhen Xie, and Liang Yu. Ivtp: Instruction-guided visual token pruning for large vision-language models. In European Conference on Computer Vision, pages 214–230. Springer, 2024

    2024

  35. [43]

    Pact: Pruning and clustering-based token reduction for faster visual language models

    Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, and Aymen Shabou. Pact: Pruning and clustering-based token reduction for faster visual language models. arXiv preprint arXiv:2504.08966, 2025

    2025 arXiv

  36. [44]

    Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer

    Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yansong Tang, Jiwen Lu, and Tao Chen. Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages...

    2024

  37. [45]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247, 2024

    arXiv

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 13

    2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.