Pith. sign in

REVIEW 4 major objections 6 minor 85 references

The Linear Attention Resurrection in Vision Transformer

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that linear attention can match softmax attention in vision transformers when attention weights stay non-negative and locally concentrated, and demonstrates the resulting architecture, L2ViT, with 84.4% ImageNet-1K top-1…

desk verdict L2ViT is a credible linear-complexity backbone that beats Swin on three benchmarks, but the 'attention concentration' mechanism is argued from visuals, not measurements, and the paper needs cleanup before it is citable. read the letter →

arxiv 2501.16182 v1 pith:Z3D2D42U submitted 2025-01-27 cs.CV cs.AI

Chuanyang Zheng This is my paper · ORCID
classification cs.CVcs.AI
keywords linearattentionvisiontransformerlocalconcentrationmoduleReLUfeaturemapnon-negativeglobal-localinteractionimageclassificationobjectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that linear attention can be made competitive with softmax attention in vision transformers, rather than merely a cheap approximation. The author argues that softmax attention owes its success to two properties, non-negative attention weights and a concentrated, locally peaked attention distribution, and shows that restoring these two properties to kernel-based linear attention closes most of the performance gap. The resulting architecture, L2ViT, alternates local window attention with linear global attention blocks and reaches 84.4% top-1 accuracy on ImageNet-1K and 87.0% after ImageNet-22K pretraining at 384 resolution, with favorable object detection and segmentation results. The paper's own limitations note that the fixed local concentration module does not fix global dispersion in the deepest layers, which is where the central mechanism is weakest.

What carries the argument

The load-bearing object is the decomposable kernel attention identity $\phi(Q_i)\sum_j \phi(K_j)^\top V_j / \sum_k \phi(Q_i)\phi(K_k)^\top$, whose associativity turns the quadratic softmax sum into $O(N)$ compute, and the local concentration module (LCM), a two-layer 7x7 depthwise convolution with GELU and batch norm inserted after linear attention. The ReLU feature map supplies the non-negativity property of softmax, and the LCM supplies the concentration property by mixing attention rows over a local window; a denominator clamp and learnable scale keep the linear-attention variance stable during training. The architecture then alternates local window attention blocks and these enhanced linear-attention blocks in a hierarchical four-stage layout.

What would settle it

Measure, on ImageNet validation images, the fraction of attention mass inside a 7x7 neighborhood of each query before and after the LCM in every linear global attention block. If the LCM does not increase local concentration in the deepest stage (comparable to layer 12 in the paper's Figure 2), then the claimed mechanism is absent precisely where the paper assumes it, and the architecture's gains would have to be reassigned to the window blocks, conv stem, or positional encodings.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that kernel-based linear attention fails in vision transformers not because it lacks global reach, but because it distributes attention too evenly across patches: without softmax's reweighting, nearby, object-bearing patches are not emphasized, so the model is distracted by background. The fix is a ReLU feature map, which keeps every attention weight non-negative, plus a local concentration module (LCM) that applies two depthwise 7x7 convolutions to the attention output, re-aggregating neighboring rows of the attention matrix so local structure is preserved. Combining enhanced linear attention with window attention yields L2ViT, which the paper reports beats window-only transformers on ImageNet-1K (84.4% vs. Swin-B's 83.5%) and performs competitively on COCO detection and ADE20K segmentation, all at linear complexity in the number of patches.

Load-bearing premise

The load-bearing premise is that a single fixed 7x7 convolution placed after every linear-attention block is enough to restore local concentration at all depths and for all image content; the paper itself notes that deep layers keep a global, unconcentrated attention pattern, yet the architecture still relies on this module in every linear attention block.

Editorial extensions

If this is right

  • With linear complexity in patch count, L2ViT can process high-resolution inputs and dense prediction tasks without aggressive early downsampling, which the paper argues is a key advantage over window-only and reduced-key attention variants.
  • Non-negativity of attention weights matters: on DeiT-Tiny, ReLU feature maps (69.3%) beat L1 norm (68.6%) and LeakyReLU (67.6%), and adding LCM brings enhanced linear attention to 73.3%, above the softmax baseline's 72.2%.
  • L2ViT-B reaches 84.4% ImageNet-1K top-1, outperforming Swin-B (83.5%) and Twins-SVT-L (83.7%); ImageNet-22K pretraining and 384 resolution push it to 87.0%.
  • LCM kernel size is a factor: using 3x3 instead of 7x7 drops ImageNet accuracy and COCO AP, showing the local concentration range is part of the mechanism.
  • The same enhanced linear attention also improves plain ViTs (DeiT-Tiny +0.6%) when mixed with softmax blocks, suggesting the design transfers beyond L2ViT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the non-negativity plus concentration diagnosis is general, a content-adaptive concentration module, such as gating or input-dependent kernels, should outperform the fixed 7x7 LCM, especially in deep layers where the paper itself observes global dispersion remains.
  • The same idea could be tested in long-context language transformers, where linear attention is a common efficiency tool; inserting a local smoothing or concentration step on the sequence dimension may restore performance lost by flattening softmax.
  • Because the architecture also includes conditional positional encodings, a convolutional stem, and window attention, a careful reader should treat the reported gains as evidence about the whole L2ViT configuration; isolating the LCM's contribution to deep-layer behavior is a direct next experiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes L2ViT, a hierarchical vision transformer that alternates local window attention (LWA) and linear global attention (LGA). LGA uses ReLU feature maps to enforce non-negative attention values and a local concentration module (LCM) consisting of two 7×7 depthwise convolutions with a residual connection, plus a clamped denominator and a learnable scale parameter. The authors argue that linear attention underperforms softmax attention in vision because its attention maps are too dispersed, and that LCM restores the missing 'concentration' needed for visual recognition. They report 84.4% ImageNet-1K top-1 accuracy for L2ViT-B, 87.0% after ImageNet-22K pretraining and 384² fine-tuning, and favorable COCO detection and ADE20K segmentation results relative to Swin and Twins at matched model sizes.

Significance. If the mechanistic story were established, this would be a significant contribution: it shows that a simple, global, linear-complexity attention can match or exceed window-based softmax attention in standard vision benchmarks. The empirical work is broad and internally consistent, covering classification, detection, and segmentation, and includes careful ablations (Tables 6, 7, 8, 10, 12), a Pytorch-style code snippet for LCM, training details, and a public code link. The proposed architecture itself appears competitive. However, the central explanatory claim—that linear attention fails because of dispersed attention maps and that LCM restores concentration—is supported only qualitatively, and the paper's own Limitations section concedes that deep-layer dispersion is not compensated by the convolution. The architecture contribution is solid; the mechanism contribution needs quantitative support or a substantial reframing.

major comments (4)
  1. [§4.2, Fig. 2] The paper's central mechanistic claim—that linear attention underperforms because its attention map is dispersed and that LCM restores the missing concentration—is supported only by qualitative visualization (Fig. 1 and Fig. 2) and by the aggregate accuracy gain in Table 6. No quantitative concentration metric (e.g., row-wise entropy, participation ratio, or the fraction of attention mass within a 7×7 neighborhood) is reported for softmax, linear, and enhanced linear attention. Please add such a metric, averaged over layers and a sample of ImageNet images, and report it for the three conditions; without it, the 'concentration' mechanism is not testable and the title/abstract claim that LCM 'concentrates' the attention map is not established.
  2. [Appendix, Limitations; §4.2] The Limitations section explicitly states that in deep layers (e.g., layer 12 of Fig. 2) the attention dispersion is global and 'may not be compensated by convolution.' Because LGA blocks with LCM appear in every stage, including the deepest stage (Fig. 3 and Table 13), the proposed concentration mechanism cannot explain the behavior of the very blocks that contribute to the reported gains. The authors should either provide per-layer quantitative evidence that shallow and middle layers carry the concentration effect, or revise the mechanism statement to a weaker claim about local smoothing of the attention output. This is load-bearing because the paper's central narrative attributes the 0.6-point Top-1 and 0.7 APb improvements in Table 6 to restored concentration.
  3. [§4.2, Eqs. (5)–(8), Table 6] The ablation in Table 6 does not isolate the 'concentration' effect from the additional depthwise convolutional parameters introduced by LCM (about 1M parameters and 0.2 GFLOPs in L2ViT-T). Eq. (5) is an algebraic identity showing that a convolution over attention outputs is equivalent to row mixing of the attention matrix; it does not demonstrate that the learned conv weights actually make the effective attention rows more concentrated than the unenhanced rows. Please add a control that injects a comparable number of depthwise conv parameters at a different location (e.g., in the MLP or on the value projection) and compares the effective attention concentration under LCM versus the control. Without such a control, the accuracy gain can be attributed to extra convolutional inductive bias, and the mechanism claim remains underdetermined.
  4. [§4.1, Fig. 2] The text asserts that ReLU-based linear attention 'is sufficient to extract short-range and long-range interactions as softmax attention' and supports this with Fig. 2. Since Fig. 2 shows only a handful of selected attention maps without quantitative agreement, the claim is stronger than the evidence. Please report a distributional similarity measure (e.g., average KL divergence, cosine distance of attention rows, or rank correlation) between softmax and ReLU-linear attention maps across layers, or soften the claim to 'captures similar qualitative patterns.'
minor comments (6)
  1. [Table 3 caption] The caption contains the typo 'ImaegNet-22k'; it should be 'ImageNet-22k'.
  2. [Eq. (4)] In Eq. (4), the denominator should be written explicitly as a scalar dot product, e.g., Σ_k φ(Q_i)·φ(K_k)^T; the current typesetting is ambiguous about where the summation and normalization apply.
  3. [Table 6] The checkmark layout in Table 6 is difficult to parse; use one row per configuration with clear column entries so that the reader can unambiguously compare Enhanced Swin-T-V1, Enhanced Swin-T-V2, and the L2ViT-T variants.
  4. [Throughout] The architecture name is inconsistently typeset as 'L2ViT', 'L 2ViT', and 'L²ViT' in the abstract, figures, and tables; please standardize.
  5. [§5.4] The text calls the 3×3 LCM kernel result a 'dramatic drop,' but the difference in Table 6 is 82.7% vs. 83.1% Top-1; a neutral wording such as 'a drop' would be more proportionate.
  6. [Appendix, Limitations] Because the Limitations section directly bears on the paper's mechanistic claim, a summary of this limitation should appear in the main text rather than only in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are supported by controlled ablations and external benchmarks; the mechanism is underdetermined but not definitionally circular.

full rationale

The paper's central derivation is an empirical design loop: it observes that linear attention underperforms softmax attention, attributes this to a lack of attention concentration based on qualitative visualizations (Fig. 1, Fig. 2), introduces a local concentration module (LCM) given by Eqs. (5)-(8), and then confirms the design with ablations (Tab. 1 and Tab. 6) and comparisons against external baselines (Tabs. 2-5). No load-bearing step reduces to its own input by construction. The non-negative property is tested through controlled feature-map variants in Tab. 1, with ReLU outperforming L1 norm and LeakyReLU; this is an empirical comparison, not a tautology. The LCM is not a fitted parameter renamed as a prediction: its benefit is established by an ablation (without LCM: 82.5% Top-1; with 7x7 LCM: 83.1%) and by comparison with alternative local enhancements (Tab. 12). The paper does not invoke self-citations or an imported uniqueness theorem; all cited prior work is external and none is claimed as the sole justification for the architecture. The weakest point is that the concentration mechanism is qualitatively inferred rather than quantitatively measured, and the Limitations section admits that deep-layer dispersion is global and 'may not be compensated by convolution.' That concern undermines the explanatory completeness of the mechanism and could be raised as a correctness or evaluation risk, but it is not circularity: the accuracy improvements are independently tested against baselines and ablations, and no equation or fitted value is defined in terms of the conclusion it is supposed to support. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two empirical design axioms (non-negativity of attention values, and recoverability of concentration by a fixed 7x7 depthwise conv) plus one tuned numerical constant (Cmin=1e2) and one tuned architectural choice (kernel size 7x7); no new physical entities are introduced. The non-negativity axiom is supported by Tab. 1 but is an empirical regularity, and the concentration axiom is explicitly acknowledged as failing in deep layers in the Limitations section.

free parameters (3)
  • LCM depthwise kernel size = 7x7
    Chosen over 3x3 after ablation on ImageNet-1K (Tab. 6): 7x7 gives 83.1% vs 82.7% for 3x3. Central claim's performance depends on this choice.
  • Denominator clamp lower bound Cmin = 1e2
    Selected after sweep over {1e-6, 1e-1, 1e0, 1e1, 1e2, 1e3} in Tab. 10; 1e2 yields 83.1% Top-1 and 45.5 AP_b. The linear-attention formulation in Eq. (4) requires this clamp to train stably.
  • Learnable scale parameter s = initialized as sqrt(C), learned
    Introduced in Section 4.2 to control variance of the key-value product in Eq. (4); affects training stability and final accuracy.
assumptions (4)
  • standard math Matrix multiplication is associative, so (phi(Q) phi(K)^T) V can be regrouped as phi(Q) (phi(K)^T V).
    Basis of the O(N) complexity claim in Section 3, Eq. (4).
  • domain assumption Non-negative attention values are necessary for good vision performance; ReLU is a sufficient feature map.
    Empirically supported by Tab. 1 (ReLU 69.3% vs L1 68.6%, LeakyReLU 67.6% on DeiT-Tiny) and by Fig. 2, but treated as a design axiom for the architecture.
  • ad hoc to paper A fixed, translation-invariant 7x7 depthwise convolution can restore the local concentration that linear attention lacks.
    The LCM in Eq. (5)-(8) assumes a single shared convolution weight pattern sufficiently focuses attention for all queries and layers; the paper's own Limitations section states deep-layer dispersion (global patterns) is not compensated.
  • domain assumption Variance of the unnormalized attention output is controlled by clamping the denominator and scaling by s.
    Section 4.2 and Appendix Section 8; the specific lower bound 1e2 is found by validation sweep, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Linear Attention Resurrection in Vision Transformer." pith.science (2026). https://pith.science/paper/Z3D2D42U

@misc{pith2026250116182,
  author       = {Pith},
  title        = {Pith review of: The Linear Attention Resurrection in Vision Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3D2D42U}},
  note         = {Machine review of arXiv:2501.16182}
}
abstract

Vision Transformers (ViTs) have recently taken computer vision by storm. However, the softmax attention underlying ViTs comes with a quadratic complexity in time and memory, hindering the application of ViTs to high-resolution images. We revisit the attention design and propose a linear attention method to address the limitation, which doesn't sacrifice ViT's core advantage of capturing global representation like existing methods (e.g. local window attention of Swin). We further investigate the key difference between linear attention and softmax attention. Our empirical results suggest that linear attention lacks a fundamental property of concentrating the distribution of the attention matrix. Inspired by this observation, we introduce a local concentration module to enhance linear attention. By incorporating enhanced linear global attention and local window attention, we propose a new ViT architecture, dubbed L$^2$ViT. Notably, L$^2$ViT can effectively capture both global interactions and local representations while enjoying linear computational complexity. Extensive experiments demonstrate the strong performance of L$^2$ViT. On image classification, L$^2$ViT achieves 84.4% Top-1 accuracy on ImageNet-1K without any extra training data or label. By further pre-training on ImageNet-22k, it attains 87.0% when fine-tuned with resolution 384$^2$. For downstream tasks, L$^2$ViT delivers favorable performance as a backbone on object detection as well as semantic segmentation.

Figures

Figures reproduced from arXiv: 2501.16182 by the authors.

Figure 2
Figure 2. The attention maps of softmax, linear attention using ReLU as ϕ, and local enhanced linear attention. x and y axes indicate the patches. The deeper the network, the longer-range dependency the attention mechanism extracts. 4.2. Local Concentration Module Although linear attention can capture similar correla￾tions as softmax attention, there is still a significant per￾formance gap, as shown in Tab. 1. We discover tha… view at source ↗
Figure 3
Figure 3. Left: the overall architecture of our proposed L2ViT. Right: the illustration of the Local Window Attention block (LWA) and Linear Global Attention block (LGA). MLP indicates the Multi-Layer Perceptron. WA indicates Window Attention, and LA indicates Linear Attention. can update these features by computing Y = LCM(LN(X)) + X ∈ R N×C , (8) The details of LCM and our implementation are summa￾rized in Appendix. We call… view at source ↗
Figure 4
Figure 4. Comparison of different MLP layers as Plain MLP in [54] (left, used by L2ViT), PVTv2 (middle), and locally im￾proved MLP (right). Compared to Other Local Enhancements Similar to our work, EfficientViT [3] proposes to insert a depth￾wise convolution in the MLP layer to improve the local￾ity of feature maps generated by linear attention layers [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The Pytorch-style code for LCM. interactions and maintaining linear complexity. Both lin￾ear attention and channel attention calculate the multiplica￾tion of key and value first. They differ in several aspects: First, linear attention still models the spatial relations…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 58 canonical work pages

  1. [1]

    Xcit: Cross-covariance image transformers

    Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bo- janowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Na- talia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. NeurIPS, 34:20014– 20027, 2021. 3, 6, 7, 8, 13

    2021

  2. [2]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 3

    arXiv 2004

  3. [3]

    Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction. In ICCV, pages 17302–17313,

  4. [4]

    MMDetection: Open mmlab detection tool- box and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. MMDetection: Open mmlab detection tool- box and benchmark. arXiv preprint arXiv:1906.07155, 2019. 6

    arXiv 1906

  5. [5]

    Region- vit: Regional-to-local attention for vision transformers

    Richard Chen, Rameswar Panda, and Quanfu Fan. Region- vit: Regional-to-local attention for vision transformers. In ICLR, 2022. 3, 6, 7

    2022

  6. [6]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 3

    arXiv 1904

  7. [7]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In ICLR, 2021. 3, 4

    2021

  8. [8]

    Twins: Revisiting the design of spatial attention in vision transformers

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haib- ing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NeurIPS, 34:9355–9366, 2021. 6, 7

    2021

Show all 85 references
  1. [9]

    Conditional po- sitional encodings for vision transformers

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xi- aolin Wei, Huaxia Xia, and Chunhua Shen. Conditional po- sitional encodings for vision transformers. arXiv preprint arXiv:2102.10882, 2021. 5

    2021 arXiv

  2. [10]

    MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark

    MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https : / / github . com / open - mmlab/mmsegmentation, 2020. 6

    2020

  3. [11]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPRW, pages 702–703,

  4. [12]

    Coatnet: Marrying convolution and attention for all data sizes

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 34:3965–3977, 2021. 6

    2021

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009. 4

    2009

  6. [14]

    Davit: Dual attention vision transform- ers

    Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transform- ers. arXiv preprint arXiv:2204.03645, 2022. 3, 6, 13

    2022 arXiv

  7. [15]

    Scaling up your kernels to 31x31: Revisiting large kernel design in cnns

    Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, pages 11963–11975, 2022. 1

    2022

  8. [16]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, pages 12124–12134,

  9. [17]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020. 1, 2

    2020

  10. [18]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, pages 6824–6835,

  11. [19]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, pages 12175–12185, 2022. 7, 8

    2022

  12. [20]

    Flatten transformer: Vision transformer using fo- cused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using fo- cused linear attention. In ICCV, pages 5961–5971, 2023. 3, 6, 7

    2023

  13. [21]

    Agent attention: On the in- tegration of softmax and linear attention

    Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, and Gao Huang. Agent attention: On the in- tegration of softmax and linear attention. arXiv preprint arXiv:2312.08874, 2023. 6

    2023 arXiv

  14. [22]

    Fastervit: Fast vision transformers with hierarchical attention

    Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06189, 2023. 6

    2023 arXiv

  15. [23]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 6

    2017

  16. [24]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 1, 7

    2016

  17. [25]

    Fair comparison between efficient attentions

    Jiuk Hong, Chaehyeon Lee, Soyoun Bang, and Heechul Jung. Fair comparison between efficient attentions. arXiv preprint arXiv:2206.00244, 2022. 1, 3

    2022 arXiv

  18. [26]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. InECCV, pages 646–661. Springer, 2016. 12

    2016

  19. [27]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Kop- pula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. 3

    2021 arXiv

  20. [28]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, pages 5156–

  21. [29]

    Sima: Simple softmax-free attention for vision transformers

    Soroush Abbasi Koohpayegani and Hamed Pirsiavash. Sima: Simple softmax-free attention for vision transformers. arXiv preprint arXiv:2206.08898, 2022. 3, 4, 8

    2022 arXiv

  22. [30]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. NeurIPS, 25:1097–1105, 2012. 1

    2012

  23. [31]

    Set transformer: A frame- work for attention-based permutation-invariant neural net- works

    Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Se- ungjin Choi, and Yee Whye Teh. Set transformer: A frame- work for attention-based permutation-invariant neural net- works. In ICML, pages 3744–3753. PMLR, 2019. 3

    2019

  24. [32]

    Mpvit: Multi-path vision transformer for dense pre- diction

    Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense pre- diction. In CVPR, pages 7287–7296, 2022. 6

    2022

  25. [33]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In ICCV, pages 2980–2988, 2017. 6

    2017

  26. [34]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 6

    2014

  27. [35]

    Dynamic group transformer: A general vision transformer backbone with dynamic group attention

    Kai Liu, Tianyi Wu, Cong Liu, and Guodong Guo. Dynamic group transformer: A general vision transformer backbone with dynamic group attention. IJCAI, 2022. 3

    2022

  28. [36]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021. 1, 2, 5, 6, 7, 12, 13

    2021

  29. [37]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pages 11976–11986, 2022. 5, 6, 12

    2022

  30. [38]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 12

    2017 arXiv

  31. [39]

    How do vision transformers work? In ICLR, 2022

    Namuk Park and Songkuk Kim. How do vision transformers work? In ICLR, 2022. 1

    2022

  32. [40]

    Random feature attention

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In ICLR, 2021. 3, 4

    2021

  33. [41]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. 12

    1992

  34. [42]

    cosformer: Rethinking softmax in attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In ICLR, 2022. 1, 3, 4, 8

    2022

  35. [43]

    Amixer: Adaptive weight mixing for self-attention free vi- sion transformers

    Yongming Rao, Wenliang Zhao, Jie Zhou, and Jiwen Lu. Amixer: Adaptive weight mixing for self-attention free vi- sion transformers. In ECCV, pages 50–67. Springer, 2022. 1

    2022

  36. [44]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618–626, 2017. 2, 4

    2017

  37. [45]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In WACV, pages 3531–3539, 2021. 3, 8

    2021

  38. [46]

    Inception transformer

    Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. arXiv preprint arXiv:2205.12956, 2022. 3

    2022 arXiv

  39. [47]

    Segmenter: Transformer for semantic segmenta- tion

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmenta- tion. In ICCV, pages 7262–7272, 2021. 1

    2021

  40. [48]

    Augmenting self-attention with persistent memory

    Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470 ,

    1907 arXiv

  41. [49]

    Rethinking the inception ar- chitecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In CVPR, pages 2818–2826,

  42. [50]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105–6114. PMLR, 2019. 1, 6

    2019

  43. [51]

    Efficient transformers: A survey

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6), 2022. 3

    2022

  44. [52]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In ICML, pages 10347–10357. PMLR, 2021. 2, 4, 5, 8, 12, 13

    2021

  45. [53]

    Are convolutional neural networks or transformers more like human vision? arXiv preprint arXiv:2105.07197,

    Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L Griffiths. Are convolutional neural networks or transformers more like human vision? arXiv preprint arXiv:2105.07197,

    arXiv

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017. 2, 12

    2017

  47. [55]

    Repvit: Revisiting mobile cnn from vit perspective

    Ao Wang, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Repvit: Revisiting mobile cnn from vit perspective. In CVPR, pages 15909–15920, 2024. 6

    2024

  48. [56]

    Convolutional embedding makes hierarchical vision transformer stronger

    Cong Wang, Hongmin Xu, Xiong Zhang, Li Wang, Zhitong Zheng, and Haifeng Liu. Convolutional embedding makes hierarchical vision transformer stronger. arXiv preprint arXiv:2207.13317, 2022. 6, 7, 8

    2022 arXiv

  49. [57]

    Scaled relu matters for training vision transformers

    Pichao Wang, Xue Wang, Hao Luo, Jingkai Zhou, Zhipeng Zhou, Fan Wang, Hao Li, and Rong Jin. Scaled relu matters for training vision transformers. In AAAI, volume 36, pages 2495–2503, 2022. 8

    2022

  50. [58]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 3

    2006 arXiv

  51. [59]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pages 568–578,

  52. [60]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415–424, 2022. 13

    2022

  53. [61]

    Crossformer: A versatile vision transformer hinging on cross-scale attention

    Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: A versatile vision transformer hinging on cross-scale attention. In ICLR, 2022. 6, 7

    2022

  54. [62]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, pages 22–31,

  55. [63]

    Pale transformer: A general vision transformer backbone with pale-shaped attention

    Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: A general vision transformer backbone with pale-shaped attention. In AAAI, volume 36, pages 2731– 2739, 2022. 2, 8

    2022

  56. [64]

    Unified perceptual parsing for scene understand- ing

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understand- ing. In ECCV, pages 418–434, 2018. 6

    2018

  57. [65]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pages 1492–1500, 2017. 7

    2017

  58. [66]

    Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention. In AAAI, volume 35, pages 14138–14148,

  59. [67]

    Moat: Alternating mobile convolution and attention brings strong vision models

    Chenglin Yang, Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Yuille, Hartwig Adam, and Liang- Chieh Chen. Moat: Alternating mobile convolution and attention brings strong vision models. arXiv preprint arXiv:2210.01820, 2022. 13

    arXiv

  60. [68]

    Focal attention for long-range interactions in vision transformers

    Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. NeurIPS, 34:30008–30022, 2021. 2, 6, 7

    2021

  61. [69]

    Metaformer baselines for vision

    Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. arXiv preprint arXiv:2210.13452, 2022. 8

    2022 arXiv

  62. [70]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In ICCV, pages 6023–6032, 2019. 12

    2019

  63. [71]

    Shvit: Single-head vision transformer with memory efficient macro design

    Seokju Yun and Youngmin Ro. Shvit: Single-head vision transformer with memory efficient macro design. In CVPR, pages 5756–5767, 2024. 6

    2024

  64. [72]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. NeruIPS, 33:17283– 17297, 2020. 3

    2020

  65. [73]

    mixup: Beyond empirical risk minimiza- tion

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. arXiv preprint arXiv:1710.09412, 2017. 12

    2017 arXiv

  66. [74]

    Poolingformer: Long document modeling with pooling attention

    Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. Poolingformer: Long document modeling with pooling attention. In ICML, pages 12437–12446. PMLR, 2021. 3

    2021

  67. [75]

    Multi-scale vision long- former: A new vision transformer for high-resolution image encoding

    Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision long- former: A new vision transformer for high-resolution image encoding. In ICCV, pages 2998–3008, 2021. 1

    2021

  68. [76]

    Rest v2: Simpler, faster and stronger

    Qing-Long Zhang and Yu-Bin Yang. Rest v2: Simpler, faster and stronger. arXiv preprint arXiv:2204.07366, 2022. 1, 3, 6

    2022 arXiv

  69. [77]

    Random erasing data augmentation

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, vol- ume 34, pages 13001–13008, 2020. 12

    2020

  70. [78]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633–641, 2017. 6, 7

    2017

  71. [79]

    Long-short transformer: Efficient transformers for language and vision

    Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. NeurIPS, 34:17723–17736, 2021. 1, 2 Appendix

    2021

  72. [80]

    Training Details We follow the training strategy in [36, 37] and show the setting in Tab. 9. When fine-tuning the 22k pre-trained model on ImageNet-1k, we use the same fine-tuning strat- egy as Swin [36]. Specifically, we fine-tune the models for 30 epochs with a batch size of...

  73. [81]

    Comparison of different lower-bound Cmin when clamping the denominator of linear attention into the range [Cmin, +∞)

    Implement Details for Clamping Lower-bound Cmin of Clamp ImaegNet 1k COCO Top-1 (%) AP b AP m 1e-6 fail fail fail 1e-1 82.7 44.6 40.6 1e0 82.9 45.0 41.0 1e1 82.9 45.1 41.0 1e2 83.1 45.5 41.2 1e3 82.8 45.2 41.0 Table 10. Comparison of different lower-bound Cmin when clamping th...

  74. [82]

    Apply our proposed enhanced linear attention on the plain ViT architectures

    Additional Ablation Experiments Model # Params.(M) FLOPs(G) Top-1 (%) DeiT-T [52] 5.7 1.3 72.2 DeiT-T [52] + Enhanced Linear Attention5.7 1.3 72.8 Table 11. Apply our proposed enhanced linear attention on the plain ViT architectures. Linear Attention on Vanilla ViT Tab. 11 sho...

  75. [83]

    Local Concentration Module Here we provide a detailed implementation of the local concentration module (LCM) as shown in Fig. 5. All the source code and pre-trained models will be publicly avail- able. Due to attention operation, we keep the feature map as F ∈ RB×N ×C, B is th...

  76. [84]

    may not be compensated by convolution since they show global patterns instead of local patterns

    Limitations While our proposed local concentration module en- hances the linear attention to a large extent, we notice that the dispersive attention in deeper layers like layer 12 in Fig- ure 2. may not be compensated by convolution since they show global patterns instead of l...

  77. [85]

    Unlike the non-overlapping patchify stem in Swin [36], we adopt a two-layer convolu- tional stem to extract more important local structure infor- mation for each patch

    Model Configurations Table 13 shows the detailed model configurations for L 2ViT-Tiny/Small/Base. Unlike the non-overlapping patchify stem in Swin [36], we adopt a two-layer convolu- tional stem to extract more important local structure infor- mation for each patch. In ith sta...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.