REVIEW 20 cited by
Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
read the original abstract
Vision tokens in multimodal large language models often dominate huge computational overhead due to their excessive length compared to linguistic modality. Abundant recent methods aim to solve this problem with token pruning, which first defines an importance criterion for tokens and then prunes the unimportant vision tokens during inference. However, in this paper, we show that the importance is not an ideal indicator to decide whether a token should be pruned. Surprisingly, it usually results in inferior performance than random token pruning and leading to incompatibility to efficient attention computation operators.Instead, we propose DART (Duplication-Aware Reduction of Tokens), which prunes tokens based on its duplication with other tokens, leading to significant and training-free acceleration. Concretely, DART selects a small subset of pivot tokens and then retains the tokens with low duplication to the pivots, ensuring minimal information loss during token pruning. Experiments demonstrate that DART can prune 88.9% vision tokens while maintaining comparable performance, leading to a 1.99$\times$ and 2.99$\times$ speed-up in total time and prefilling stage, respectively, with good compatibility to efficient attention operators. Our codes are available at https://github.com/ZichenWen1/DART.
Forward citations
Cited by 20 Pith papers
-
Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs
SFPruner performs redundancy-aware visual token pruning in a single non-iterative pass using semantics-guided ridge leverage scores and directional masking, cutting selection latency from 112.4 ms to 2.5 ms at 512 tok...
-
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
LearnPruner prunes vision tokens to 5.5% of the original count while retaining about 95% of VLM performance and delivering 3.2 times faster inference by fixing attention sink in encoders and using unbiased middle-laye...
-
CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference
A training-free, text-driven pre-LLM pruning method that keeps ~99% of LVLM performance while retaining only ~11-22% of visual tokens.
-
Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models
A training-free VLM token pruning method that diffuses importance on a kNN graph, allocates budgets via quadtree partitioning, and adds sink tokens to retain pruned information.
-
MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs
MS-Resampler deploys multiple scope-specific resamplers with explicit spatial priors and adaptive fusion to outperform single-scope global cross-attention in MLLMs on ten benchmarks with minimal added cost.
-
MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving
MVPruner is a two-stage dynamic token pruning technique that uses view diversity for initial budget allocation and instruction text for task-aligned selection, delivering 87.3% FLOPs reduction and 4.97x prefilling spe...
-
When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics
STS is a two-stage pruning framework that decouples structural diversity via repulsion sampling from semantic filtering via cross-attention to reduce redundancy in visual tokens for VLMs.
-
Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
SPpruner reduces visual tokens in VLMs via focus identification followed by context-aware scanning, retaining 22.2% tokens for 2.53x speedup on Qwen2.5-VL with negligible accuracy loss.
-
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
A unified learnable KV eviction policy with cross-layer calibration reduces memory and matches or exceeds full-cache performance on long-context tasks by retaining useful tokens and limiting attention dilution.
-
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
RTPrune prunes visual tokens in DeepSeek-OCR via a reading-twice two-stage process, retaining 84.25% tokens for 99.47% accuracy and 1.23x faster prefill on OmniDocBench.
-
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
ST-Prune is a training-free spatio-temporal token pruning framework for VLMs in autonomous driving that achieves near-lossless results at 90% token reduction by exploiting motion volatility, temporal recency, and mult...
-
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
HAWK is a training-free method that prunes over 80% of visual tokens in MLLMs while retaining 96% accuracy by using head importance weights and text-guided attention to select task-relevant tokens.
-
ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention
Frozen visual tokens with sparse cross-attention at selected layers preserve 98% accuracy while reducing vision-side FLOPs to 4% in LLaVA-1.5 models.
-
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Using the separator token as a position-free query, SepPrune prunes 80% of vision tokens while retaining 96.3% of Qwen2.5-VL-7B's original accuracy.
-
Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models
Training-free SpecFlow condenses VLM visual tokens via kNN heat diffusion, adaptive quadtree budgets, and coreset sinks, retaining 95.6% LLaVA-1.5 performance after pruning 88.9% of tokens.
-
MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving
MVPruner is a two-stage adaptive token pruning technique for multi-view VLMs that achieves 87.3% FLOPs reduction and 4.97x prefilling speedup while retaining 98.5% accuracy on DriveLM.
-
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
RTPrune introduces a reading-twice inspired two-stage pruning technique for DeepSeek-OCR that retains 84.25% tokens while delivering 99.47% accuracy and 1.23x faster prefill on OmniDocBench.
-
EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling
EvoComp compresses visual tokens in MLLMs by 3x while retaining 99.3% accuracy via an evolutionary labeling strategy that searches for low-loss, semantically diverse token subsets.
-
Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
AOT reduces visual tokens in VLLMs via intra-frame and inter-frame anchors with local-global optimal transport, delivering competitive benchmark performance and efficiency gains in a training-free way.
-
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
RTPrune delivers 99.47% accuracy and 1.23x faster prefill on OmniDocBench for DeepSeek-OCR-Large by retaining only 84.25% of tokens through a reading-twice inspired two-stage pruning process.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.