REVIEW 13 cited by
Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The vision tokens in multimodal large language models usually exhibit significant spatial and temporal redundancy and take up most of the input tokens, which harms their inference efficiency. To solve this problem, some recent works were introduced to drop the unimportant tokens during inference where the importance of each token is decided only by the information in either the vision encoding stage or the prefilling stage. In this paper, we propose Multi-stage Token Dropping (MustDrop) to measure the importance of each token from the whole lifecycle, including the vision encoding stage, prefilling stage, and decoding stage. Concretely, in the visual encoding stage, MustDrop merges spatially adjacent tokens with high similarity, and establishes a key token set to retain the most vision-critical tokens, preventing them from being discarded in later stages. In the prefilling stage, MustDrop further compresses vision tokens by the guidance of text semantics, with a dual-attention filtering strategy. In the decoding stage, an output-aware cache policy is proposed to further reduce the size of the KV cache. By leveraging tailored strategies in the multi-stage process, MustDrop can more precisely recognize the important and redundant tokens, thus achieving an optimal balance between performance and efficiency. For instance, MustDrop reduces about 88.5\% FLOPs on LLaVA with a compression ratio of 92.2\% while maintaining comparable accuracy. Our codes are available at \url{https://github.com/liuting20/MustDrop}.
Forward citations
Cited by 13 Pith papers
-
SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
Visual-token attention filtering improves structured pruning of vision-language models, keeping 94% of average benchmark accuracy after removing 20% of parameters.
-
CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference
A training-free, text-driven pre-LLM pruning method that keeps ~99% of LVLM performance while retaining only ~11-22% of visual tokens.
-
Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
A grounding-guided attack that concentrates perturbation on text-matched image regions and disrupts global and local semantic alignment consistently improves adversarial transferability across multiple vision-language models.
-
Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
ACCM recovers information lost in high-rate visual token pruning by generating a question-guided caption from discarded tokens and selecting the best candidate, improving pruned LVLM accuracy with fewer FLOPs.
-
METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.
-
LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.
-
Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs
Post-training an LLM on economic reasoning problems improves accuracy on economic benchmarks and, without game-specific training, raises its Nash equilibrium frequency and win rates in strategic games.
-
ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs
A question-aware, per-image budget decides which visual tokens to keep, letting OCR-focused multimodal models hold accuracy at about half the tokens.
-
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Using the separator token as a position-free query, SepPrune prunes 80% of vision tokens while retaining 96.3% of Qwen2.5-VL-7B's original accuracy.
-
Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
ALTR keeps model accuracy nearly intact while pruning up to ~89% of visual tokens, using entropy ranking plus stride sampling on an MLP consistency signal, without attention maps or pairwise comparisons.
-
GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
A training-free token pruning method that combines cosine-similarity saliency with greedy redundancy removal to preserve accuracy at high compression ratios.
-
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
CDPruner prunes visual tokens in MLLMs by maximizing instruction-conditioned diversity via a determinantal point process, preserving accuracy at high reduction ratios.
-
Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Models
FlashVLA, a training-free plug-in, reuses stable actions and prunes visual tokens to cut VLA model inference FLOPs by 55.7% and latency by 36% with only a 0.7% success-rate drop on LIBERO.
Discussion (0). Sign in to comment.