REVIEW 2 cited by
Fair Comparison between Efficient Attentions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformers have been successfully used in various fields and are becoming the standard tools in computer vision. However, self-attention, a core component of transformers, has a quadratic complexity problem, which limits the use of transformers in various vision tasks that require dense prediction. Many studies aiming at solving this problem have been reported proposed. However, no comparative study of these methods using the same scale has been reported due to different model configurations, training schemes, and new methods. In our paper, we validate these efficient attention models on the ImageNet1K classification task by changing only the attention operation and examining which efficient attention is better.
Forward citations
Cited by 2 Pith papers
-
NAE: Normalizing AutoEncoder
A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...
-
The Linear Attention Resurrection in Vision Transformer
L2ViT alternates ReLU-based linear attention with a depthwise-convolution local concentration module and window attention, reaching 84.4% ImageNet-1K top-1 accuracy at linear-complexity attention.
Discussion (0). Continue with ORCID to comment.