REVIEW 6 cited by
Vision Transformer Pruning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Vision transformer has achieved competitive performance on a variety of computer vision applications. However, their storage, run-time memory, and computational demands are hindering the deployment to mobile devices. Here we present a vision transformer pruning approach, which identifies the impacts of dimensions in each layer of transformer and then executes pruning accordingly. By encouraging dimension-wise sparsity in the transformer, important dimensions automatically emerge. A great number of dimensions with small importance scores can be discarded to achieve a high pruning ratio without significantly compromising accuracy. The pipeline for vision transformer pruning is as follows: 1) training with sparsity regularization; 2) pruning dimensions of linear projections; 3) fine-tuning. The reduced parameters and FLOPs ratios of the proposed algorithm are well evaluated and analyzed on ImageNet dataset to demonstrate the effectiveness of our proposed method.
Forward citations
Cited by 6 Pith papers
-
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
FastTPS accelerates LLM token-phase inference via reloading-free static KV-cache management, tiled fused RoPE attention, and interlaced-weight MLP fusion, yielding up to 6× speedup at 93% bandwidth on AMD NPUs.
-
Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
Heterogeneity-aware depth pruning of attention and activation layers, guided by a polynomial model-accuracy predictor, delivers up to 1.58× speedup on DeiT-B and 5.19× when combined with width pruning.
-
L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training
A distilled 99M-parameter CLIP gives L-CLIPScore, a lightweight caption metric that matches CLIPScore on human correlation and best improves captioning models when mixed with CIDEr.
-
Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework
SNOWS prunes vision networks in one shot by optimizing a K-step nonlinear reconstruction objective with Hessian-free Newton steps, improving accuracy over layer-wise least-squares methods.
-
QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models
QuarterMap prunes spatial activations before VMamba's four-directional scan and upsamples after, yielding up to 1.11x throughput with under 1% accuracy loss on ImageNet classification.
-
V"Mean"ba: Visual State Space Models only need 1 hidden dimension
VMeanba speeds up VMamba's selective scan by averaging its internal channel dimension down to 1, achieving up to 1.12x end-to-end speedup with under 3% accuracy loss on ImageNet and ADE20k.
Discussion (0). Continue with ORCID to comment.