REVIEW 7 cited by
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Despite transformers' impressive accuracy, their computational cost is often prohibitive to use with limited computational resources. Most previous approaches to improve inference efficiency require a separate model for each possible computational budget. In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training. We train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines a sequence length at each layer. We then conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget. Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into token-level classification with Drop-and-Restore process that drops word-vectors temporarily in intermediate layers and restores at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including span-based question answering and text classification. Code is available at https://github.com/clovaai/length-adaptive-transformer.
Forward citations
Cited by 7 Pith papers
-
Training-free Token Reduction for Vision Mamba
MTR uses Mamba's timescale parameter Δ as a token importance score to merge unimportant tokens, giving training-free inference speedups with small accuracy loss.
-
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding
Static per-head thresholds calibrated on a small dataset can replace top-k selection in transformer attention, achieving 3 to 10x sparsity with negligible accuracy loss.
-
ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
ImagePiece merges the bottom 30% of low-attention image tokens into semantically meaningful chunks before pruning, reporting 54% faster DeiT-S inference with a 0.39% ImageNet accuracy gain.
-
AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning
AIM reduces multi-modal LLM inference cost by merging similar visual tokens and progressively pruning visual tokens based on attention importance, cutting FLOPs about 7x with near-parity accuracy and improving long-vi...
-
Training Noise Token Pruning
A noise-allocation training trick learns token relevance scores that can prune vision transformer tokens at test time, beating some baselines in some regimes but not all claimed settings.
-
LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression
LightVLM accelerates vision-language model inference with pyramid token merging and KV cache compression, preserving about 98% accuracy with only 3% of image tokens.
-
LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning
LeMo reduces long-context fine-tuning memory by eliminating low-informativeness tokens, predicting sparsity patterns, and optimizing kernels, while keeping perplexity close to LoRA.
Discussion (0). Continue with ORCID to comment.