REVIEW 7 cited by
Dynamic Token Reduction during Generation for Vision Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Vision-Language Models (VLMs) have achieved notable success in multimodal tasks but face practical limitations due to the quadratic complexity of decoder attention mechanisms and autoregressive generation. Existing methods like FASTV and VTW have achieved notable results in reducing redundant visual tokens, but these approaches focus on pruning tokens in a single forward pass without systematically analyzing the redundancy of visual tokens throughout the entire generation process. In this paper, we introduce a dynamic pruning strategy tailored for VLMs, namedDynamic Rate (DyRate), which progressively adjusts the compression rate during generation. Our analysis of the distribution of attention reveals that the importance of visual tokens decreases throughout the generation process, inspiring us to adopt a more aggressive compression rate. By integrating a lightweight predictor based on attention distribution, our approach enables flexible adjustment of pruning rates based on the attention distribution. Our experimental results demonstrate that our method not only reduces computational demands but also maintains the quality of responses.
Forward citations
Cited by 7 Pith papers
-
Attending to Multimodal Generation One Token at a Time
MLLMs attend to the relevant modality at the exact generation steps that need it; causal blocking of those peaks breaks answers while timed boosting improves multimodal performance.
-
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
CLSE prunes tokens in MLLMs by quantifying cross-layer spectral redistribution in the frequency domain to preserve semantically active tokens and reduce compute.
-
Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
PriorTR estimates model-induced prior attention via a null token in one forward pass and contrasts it with task-conditioned attention to improve visual token pruning accuracy-efficiency trade-offs in MLLMs.
-
Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models
DiffPrune reformulates visual token pruning as continuous control of token information using an Information Throttler with importance-conditioned variance-preserving noise, enabling fully differentiable learning of sc...
-
DiffPrune: differentiable information throttling for token pruning in vision-language models
DiffPrune replaces Gumbel-Softmax surrogate gradients with a differentiable noise-throttling path for visual token scoring, achieving high accuracy retention under aggressive pruning on three VLM families.
-
A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
The paper claims 92.6% dynamic visual-token pruning with retained (or 110% of) baseline VQA performance, but the full text supplied is a different paper, so the claim is unverifiable here.
-
Toward Native Multimodal Modeling: A Roadmap
A roadmap that defines architectural nativity for multimodal models and categorizes them into Multi-to-Text, Multi-to-Target, and Multi-to-Multi types while outlining an industrial pipeline toward unified transformer-...
Discussion (0). Sign in to comment.