REVIEW 6 cited by
Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Vision transformer has emerged as a new paradigm in computer vision, showing excellent performance while accompanied by expensive computational cost. Image token pruning is one of the main approaches for ViT compression, due to the facts that the complexity is quadratic with respect to the token number, and many tokens containing only background regions do not truly contribute to the final prediction. Existing works either rely on additional modules to score the importance of individual tokens, or implement a fixed ratio pruning strategy for different input instances. In this work, we propose an adaptive sparse token pruning framework with a minimal cost. Specifically, we firstly propose an inexpensive attention head importance weighted class attention scoring mechanism. Then, learnable parameters are inserted as thresholds to distinguish informative tokens from unimportant ones. By comparing token attention scores and thresholds, we can discard useless tokens hierarchically and thus accelerate inference. The learnable thresholds are optimized in budget-aware training to balance accuracy and complexity, performing the corresponding pruning configurations for different input instances. Extensive experiments demonstrate the effectiveness of our approach. Our method improves the throughput of DeiT-S by 50% and brings only 0.2% drop in top-1 accuracy, which achieves a better trade-off between accuracy and latency than the previous methods.
Forward citations
Cited by 6 Pith papers
-
Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detection
FocusMamba uses event-camera activity to adaptively prune uninformative tokens in both RGB and event streams, improving detection accuracy and cutting FLOPs.
-
Sparsified State-Space Models are Efficient Highway Networks
Simba prunes tokens hierarchically in pre-trained SSMs, creating sparse upper layers that act as highways, improving the accuracy-FLOPs trade-off and long-context perplexity.
-
DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
A training-free framework that dynamically merges similar visual tokens and reconstructs attention for the full sequence, yielding 32-85% token reduction with near-parity performance on VLM benchmarks.
-
AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning
AIM reduces multi-modal LLM inference cost by merging similar visual tokens and progressively pruning visual tokens based on attention importance, cutting FLOPs about 7x with near-parity accuracy and improving long-vi...
-
Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
Token reduction for vision transformers is recast as a matrix transformation, and a many-to-many weighted merging is used for training-free acceleration.
-
freePruner: A Training-free Approach for Large Multimodal Model Acceleration
freePruner selects 50 percent of visual tokens using attention-based importance and keeps accuracy close to the original model, enabling a training-free about 2x acceleration for LMMs.
Discussion (0). Continue with ORCID to comment.