REVIEW 25 cited by
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
read the original abstract
In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named CDPruner, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by 95\% and CUDA latency by 78\%, while maintaining 94\% of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.
Forward citations
Cited by 25 Pith papers
-
Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs
SFPruner performs redundancy-aware visual token pruning in a single non-iterative pass using semantics-guided ridge leverage scores and directional masking, cutting selection latency from 112.4 ms to 2.5 ms at 512 tok...
-
RADIO1D: Elastic Representations for Condensed Vision Modeling
RADIO1D produces elastic hierarchical 1D visual tokens via multi-teacher distillation that match or beat fixed 2D encoders in VLMs at lower token counts.
-
X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding
X-Stream benchmark shows SOTA MLLMs score ~50% on concurrent multi-stream tasks and lack proactive ability, using a dual-verification pipeline to avoid single-stream bias.
-
X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding
X-Stream benchmark shows state-of-the-art MLLMs achieve only about 50% on multi-stream video tasks and exhibit poor proactive ability.
-
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Sink-Token-aware Pruning (SToP) suppresses semantically uninformative sink tokens during visual token pruning in Video LLMs, boosting fine-grained performance even at 90% pruning rates across hallucination, reasoning,...
-
Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding
Representing bounding-box coordinates as axis-specific hundreds, tens, and ones tokens, plus a geometry-aware GRPO reward, improves generative visual grounding accuracy.
-
PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.
-
WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation
WaveZip, a training-free wavelet method for video token condensation, reports about 10x token reduction while retaining roughly 98-99% accuracy on multi-benchmark LVLM evaluation.
-
WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation
Wavelet-based temporal and spatial token condensation preserves 99.6% of full-token video question-answering accuracy at 10x compression without retraining.
-
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
Ground-truth boxes, when used as a privileged visual hint for a frozen teacher during training, improve autoregressive visual grounding in multimodal LLMs by about 4 Acc@0.7 points with no inference-time overhead.
-
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
TOPS formulates visual token pruning as constructing Token Optimal Preservation Sets using three information-theoretic principles and demonstrates superior performance on MLLM benchmarks.
-
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
CLSE prunes tokens in MLLMs by quantifying cross-layer spectral redistribution in the frequency domain to preserve semantically active tokens and reduce compute.
-
TGV-KV: Text-Grounded KV Eviction for Vision-Language Models
TGV-KV uses text-vision budgeting, weighted ranking, and prioritised retention to evict KV cache in VLMs while retaining 99.2% accuracy at 5% budget on VizWiz-VQA.
-
SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation
SAFE-Pruner forecasts deep-layer visual-token saliency from historical attention maps and refreshes at subtask boundaries, enabling up to 1.89x faster VLA inference with minimal success-rate drop.
-
Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
SPpruner reduces visual tokens in VLMs via focus identification followed by context-aware scanning, retaining 22.2% tokens for 2.53x speedup on Qwen2.5-VL with negligible accuracy loss.
-
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
RTPrune prunes visual tokens in DeepSeek-OCR via a reading-twice two-stage process, retaining 84.25% tokens for 99.47% accuracy and 1.23x faster prefill on OmniDocBench.
-
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Sink-Token-aware Pruning (SToP) uses a sink score to suppress attention-sink tokens during visual token pruning, improving fine-grained video understanding in Video LLMs at high pruning rates.
-
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
HAWK is a training-free method that prunes over 80% of visual tokens in MLLMs while retaining 96% accuracy by using head importance weights and text-guided attention to select task-relevant tokens.
-
Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
A 3D-aware VLM, RoboTracer, generates metric-grounded spatial traces for robot manipulation using scale supervision and metric-sensitive reinforcement rewards.
-
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Using the separator token as a position-free query, SepPrune prunes 80% of vision tokens while retaining 96.3% of Qwen2.5-VL-7B's original accuracy.
-
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
Training a coordinate-generating VLM with an IoU-aware distillation loss from a teacher that sees the ground-truth box marked on the image improves referring-expression grounding by ~3-4 accuracy points.
-
SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation
SAFE-Pruner forecasts deep-layer token saliency in VLA models via semantic attention consistency and adaptive subtask detection to achieve up to 1.89x speedup with under 1.7% success rate loss.
-
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
RTPrune introduces a reading-twice inspired two-stage pruning technique for DeepSeek-OCR that retains 84.25% tokens while delivering 99.47% accuracy and 1.23x faster prefill on OmniDocBench.
-
Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning
EADP filters textual noise via statistical entropy then casts token selection as submodular maximization with spatial prior to preserve fine-grained cues in VLMs under strict budgets.
-
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
RTPrune delivers 99.47% accuracy and 1.23x faster prefill on OmniDocBench for DeepSeek-OCR-Large by retaining only 84.25% of tokens through a reading-twice inspired two-stage pruning process.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.