Pith. sign in

REVIEW 17 cited by

Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.01818 v2 pith:Z2QMCJLC submitted 2024-12-02 cs.CV cs.AI

Qizhe Zhang , Aosong Cheng , Ming Lu , Renrui Zhang , Zhiyong Zhuo , Jiajun Cao , Shaobo Guo , Qi She
show 1 more author
Shanghang Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords tokensvisualattentionpruningvisprunertext-visualtokencues
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large vision-language models (LVLMs) generally contain significantly more visual tokens than their textual counterparts, resulting in a considerable computational burden. Recent efforts have been made to tackle this issue by pruning visual tokens early within the language model. Most existing works use attention scores between text and visual tokens to assess the importance of visual tokens. However, in this study, we first analyze the text-visual attention in the language model and find that this score is not an ideal indicator for token pruning. Based on the analysis, We propose VisPruner, a plug-and-play method that utilizes visual cues for more effective token pruning in LVLMs. Specifically, we first use visual attention to select a limited number of significant tokens. Then, we remove duplicate tokens from the remaining ones based on their similarity. By retaining diverse tokens alongside the initially selected important tokens, we maximally preserve the visual information of the input image. Experimental results demonstrate that our VisPruner sustains strong performance across various VLM architectures and reduction ratios, significantly outperforming existing methods based on text-visual attention. Notably, without any training, VisPruner can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%, while maintaining comparable performance. Our code is available at https://github.com/Theia-4869/VisPruner.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    SFPruner performs redundancy-aware visual token pruning in a single non-iterative pass using semantics-guided ridge leverage scores and directional masking, cutting selection latency from 112.4 ms to 2.5 ms at 512 tok...

  2. When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A break-even study of Qwen2.5-VL on two GPUs finds that a post-vision pruner and a pre-vision resolution rule both cut end-to-end latency, but pre-vision routing wins by skipping vision encoding, not by removing the m...

  3. Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CaRe retains about 96% of full-token VLM benchmark performance while pruning about 94% of visual tokens, by calibrating retained anchors with confidence-gated signals from pruned tokens.

  4. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

  5. Training-free Token Reduction for Vision Mamba

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MTR uses Mamba's timescale parameter Δ as a token importance score to merge unimportant tokens, giving training-free inference speedups with small accuracy loss.

  6. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  7. Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Pruning visual tokens degrades visual grounding because position IDs become misaligned; preserving the original position IDs recovers most of the lost accuracy with no extra cost.

  8. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  9. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  10. Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Tracking cross-layer attention momentum lets training-free pruning reactivate late-rising visual tokens, cutting final-layer tokens to ~23 at 77.8% prune while keeping near-baseline multimodal accuracy.

  11. SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Using the separator token as a position-free query, SepPrune prunes 80% of vision tokens while retaining 96.3% of Qwen2.5-VL-7B's original accuracy.

  12. LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression

    cs.CV 2025-08 conditional novelty 5.0 of 10

    LightVLM accelerates vision-language model inference with pyramid token merging and KV cache compression, preserving about 98% accuracy with only 3% of image tokens.

  13. GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A training-free token pruning method that combines cosine-similarity saliency with greedy redundancy removal to preserve accuracy at high compression ratios.

  14. Do Concept Replacement Techniques Really Erase Unacceptable Concepts?

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Weight-editing concept erasure methods fail to prevent reconstruction of unacceptable images in an image-to-image pipeline; a face-targeted editing method, AntiMirror, replaces celebrity likenesses with better fidelit...

  15. FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.

  16. AdaTP: Attention-Debiased Token Pruning for Video Large Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    AdaTP prunes visual tokens in video LLMs by debiasing attention scores, reducing FLOPs to about a quarter of the vanilla model at matching benchmark accuracy.

  17. A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    The paper claims 92.6% dynamic visual-token pruning with retained (or 110% of) baseline VQA performance, but the full text supplied is a different paper, so the claim is unverifiable here.

Pith tools