REVIEW 13 cited by
PPT: Token Pruning and Pooling for Efficient Vision Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Vision Transformers (ViTs) have emerged as powerful models in the field of computer vision, delivering superior performance across various vision tasks. However, the high computational complexity poses a significant barrier to their practical applications in real-world scenarios. Motivated by the fact that not all tokens contribute equally to the final predictions and fewer tokens bring less computational cost, reducing redundant tokens has become a prevailing paradigm for accelerating vision transformers. However, we argue that it is not optimal to either only reduce inattentive redundancy by token pruning, or only reduce duplicative redundancy by token merging. To this end, in this paper we propose a novel acceleration framework, namely token Pruning & Pooling Transformers (PPT), to adaptively tackle these two types of redundancy in different layers. By heuristically integrating both token pruning and token pooling techniques in ViTs without additional trainable parameters, PPT effectively reduces the model complexity while maintaining its predictive accuracy. For example, PPT reduces over 37% FLOPs and improves the throughput by over 45% for DeiT-S without any accuracy drop on the ImageNet dataset. The code is available at https://github.com/xjwu1024/PPT and https://github.com/mindspore-lab/models/
Forward citations
Cited by 13 Pith papers
-
Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
Heterogeneity-aware depth pruning of attention and activation layers, guided by a polynomial model-accuracy predictor, delivers up to 1.58× speedup on DeiT-B and 5.19× when combined with width pruning.
-
Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purging
Removing tokens whose embeddings diverge most from a source or CLS-token prototype, selected by output entropy, improves 3D point cloud classification under distribution shift without backpropagation.
-
Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
ACCM recovers information lost in high-rate visual token pruning by generating a question-guided caption from discarded tokens and selecting the best candidate, improving pruned LVLM accuracy with fewer FLOPs.
-
ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference
ToFe freezes unimportant ViT tokens temporarily and reuses them later, achieving about 50% FLOPs reduction with less than 2% top-1 accuracy drop on ImageNet.
-
ToSA: Token Merging with Spatial Awareness
A training-free token merging method that adds depth-derived spatial similarity to ToMe's bipartite soft matching, improving VQA accuracy at high token reduction rates.
-
Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers
GRAT accelerates pretrained diffusion transformers by grouping tokens and restricting each group's attention to neighboring blocks or criss-cross rows and columns, achieving large speedups with near-full-attention quality.
-
DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
A training-free framework that dynamically merges similar visual tokens and reconstructs attention for the full sequence, yielding 32-85% token reduction with near-parity performance on VLM benchmarks.
-
ST$^3$: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
ST3 progressively removes inattentive visual tokens across network layers and across generation steps, achieving about 2x faster MLLM inference with roughly 30 to 50 percent of the original KV cache memory.
-
Token Cropr: Faster ViTs for Quite a Few Tasks
Cropr uses removable auxiliary heads to learn task-relevant token pruning in ViTs, achieving 1.5-4x speedups with small accuracy drops across image classification, semantic segmentation, and object detection.
-
Importance-Based Token Merging for Efficient Image and Video Generation
A token-merging method that anchors computation on high-CFG-importance tokens improves generation quality at fixed inference speedups.
-
GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
A training-free token pruning method that combines cosine-similarity saliency with greedy redundancy removal to preserve accuracy at high compression ratios.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
-
Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
Token reduction for vision transformers is recast as a matrix transformation, and a many-to-many weighted merging is used for training-free acceleration.
Discussion (0). Continue with ORCID to comment.