Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that generalized attention inevitably disperses as token count grows, and builds SEMA, a linear-complexity vision block that uses a simple average for the global term and window attention for focus, reaching 83.7% on…

desk verdict A clean dispersion theorem (with a fixable proof gap) motivating a simple window-plus-average attention that gives small but consistent gains; worth refereeing after tightening the theory and adding reproducibility. read the letter →

arxiv 2506.08297 v2 pith:XWWC6R6I submitted 2025-06-10 cs.CV cs.AI

Nhat Thanh Tran , Fanghui Xue , Shuai Zhang , Jiancheng Lyu , Yunling Zheng , Yingyong Qi , Jack Xin This is my paper
classification cs.CVcs.AI
keywords attentiondispersiongeneralizedlinearwindowMamba-likevisiontransformerImageNetclassificationscalable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that any global attention mechanism in the softmax/linear family must lose focus as the number of keys grows: in the limit, a query assigns equal weight to every key. Instead of engineering around this dispersion, the authors use it as design guidance, replacing the global attention term with a plain arithmetic average of the value vectors and keeping focus through local window attention. The resulting block, SEMA, is placed in a Mamba-like macro-architecture and is reported to reach 83.7% top-1 accuracy on ImageNet-1K with 26M parameters, outperforming comparable vision Mamba models and improving further when fine-tuned to 384, 672, and 768 pixel inputs. If the claim holds, it offers a provably motivated, linear-complexity attention whose global component is consistent with the asymptotic behavior of full attention.

What carries the argument

The load-bearing object is the $\Phi$-normalized attention operator (Definition 3.3), which assigns weight $\phi(\text{similarity})/\sum \phi(\cdot)$ with a continuous $\phi$. The argument runs on the dispersion lemma (Lemma 3.2), which bounds each coefficient between $\phi(a)/(n\phi(b))$ and $\phi(b)/(n\phi(a))$, forcing $\Theta(1/n)$ behavior, and on Theorem 3.4, which transports this through continuous layers to all transformer heads. SEMA is then defined as window softmax attention plus the broadcast average of the value vectors (Eq. 13), placed in the MILA macro-architecture with rotary positional embeddings and a local positional-encoding branch (RoPE and LePE), so the local window term preserves focus while the average supplies global context at $O(1)$ cost.

What would settle it

Measure the per-query maximum softmax weight in a trained vision transformer on 224×224 ImageNet patches across token counts from 49 to 3136: if the average maximum weight stays far above $1/n$ (say, greater than $10/n$), attention has not dispersed at operating scale, so SEMA's averaging term is not approximating the global softmax term. A second check is to fine-tune SEMA at 768×768 with the averaging term removed and see whether the reported accuracy gap disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is that dispersion is unavoidable for a broad class of attention mechanisms. It defines $\Phi$-normalized attention with a continuous normalization function $\phi$, proves that each attention coefficient is $\Theta(1/n)$ whenever the logit vector is bounded (Lemma 3.2), and extends this to networks of feedforward and self-attention layers under compactness (Theorem 3.4). Consequently softmax attention, linear attention, focused attention, and MILA's attention all give each key weight $\Theta(1/n)$ in the large-$n$ limit, while window attention escapes because each query only sees a fixed set of keys. The proposed SEMA attention is exactly the sum of a window softmax attention term and the arithmetic mean of all value vectors broadcast to every token. On ImageNet-1K this design reaches 83.7% top-1 at 26M parameters, and fine-tuning to 384, 672, and 768 pixels gives 84.1, 84.1, and 84.2, which the paper reports as 0.5–0.7% above VMamba-T at each resolution.

Load-bearing premise

The load-bearing premise is that at the token counts SEMA actually uses (49 to 3136), softmax attention has already flattened enough that averaging all value vectors is a faithful stand-in for the global attention term; the paper says the regime is 'close but not exactly like theory' and provides no quantitative check of how close.

Editorial extensions

If this is right

  • At sufficiently large token counts, every head of softmax, linear, focused, and differential attention must give each key weight $\Theta(1/n)$; models relying on global attention in long-context settings should expect this flattening.
  • SEMA's global term costs $O(d)$ per token rather than $O(n^2 d)$, so the block is linear in sequence length and can scale to larger images or longer sequences.
  • SEMA reaches 83.7% top-1 on ImageNet-1K with 26M parameters, and at 384, 672, and 768 pixel inputs it scores 84.1, 84.1, and 84.2, each 0.5–0.7% above VMamba-T at the same resolution.
  • The ablation shows the averaging term alone adds 0.2% top-1 over the same model without it, which the paper reads as direct support for the dispersion-motivated design.
  • SEMA is competitive on COCO instance segmentation with Mask R-CNN under both 1x and 3x training, indicating the local-global block transfers beyond classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not measure how close finite-$n$ softmax weights are to uniform; measuring attention entropy on trained models at $n=196$, $784$, and $3136$ would directly test whether the averaging term is faithful at operating scales.
  • If dispersion is the active mechanism, the global average could be made adaptive (for example a learned convex combination of the average and a sharper statistic) without changing the theoretical motivation, an extension the authors list as future work.
  • The same theorem should apply to fixed-vocabulary language models, so SEMA's average-plus-window recipe is a candidate template for long-context attention beyond vision.
  • Because the ablation gain is small (0.2%), an alternative explanation for SEMA's accuracy is the added value projection and parameter count; a controlled comparison at matched capacity would separate the two.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a generalized Φ-normalized attention framework that subsumes softmax attention, linear attention, focused attention, MILA, and differential attention, and proves (or attempts to prove) that under compactness assumptions the attention coefficients decay as Θ(1/n) as the number of keys grows. Based on this 'dispersion' property, the authors propose SEMA, a Mamba-like vision backbone that combines window attention for local focusing with a global uniform average of value vectors, and report ImageNet-1K classification and COCO detection/segmentation results. The claimed contributions are a theoretical justification of dispersion, a scalable O(n) local-global attention design, and empirical gains over MILA and VMamba at similar parameter counts.

Significance. If the theoretical claim and the empirical results both held, SEMA would offer a simple, efficient local-global attention mechanism with a principled asymptotic motivation, and the unified treatment of several attention variants in a single Φ-normalized framework would be a useful conceptual contribution. The paper ships a formal-looking theorem and explicit dispersion bounds for multiple attention types, and the ablation in Table 5 cleanly isolates the averaging term. However, the central theoretical link between dispersion and arithmetic averaging is not actually established, the proof has a uniformity gap, and the empirical margins are small, single-run, and not backed by released code. The significance of the paper therefore hinges on whether these gaps can be closed; in its current form the main claims are overstated.

major comments (3)
  1. [Section 3.1 / Appendix B.2] The proof of Theorem 3.4 does not establish the uniform-in-n bound on the logits that Lemma 3.2 requires. For each fixed n, compactness of X^n and continuity of the layers give a bound M_n on |e_ij^(n)|, but Lemma 3.2 needs a single M independent of n (and k) for the Θ(1/n) conclusion to have constants independent of n. The recursive layer structure could supply such a uniform bound by induction, since each feedforward map is continuous on a fixed compact domain and each attention layer is a convex combination that stays in the convex hull of the previous layer's output set; however, this induction is not stated in the proof. Without it, Theorem 3.4 is not proved as written. Please either add the explicit induction or weaken the theorem to a per-n statement with constants that may depend on n.
  2. [Section 4, Eq. (13)] The design replaces the global part of full attention with (1/n)Σ v_j based on the dispersion theorem, but the theorem only shows that each attention weight is Θ(1/n); it does not show that the attention output is close to the uniform average. With bounded logits, the weights may remain concentrated on a subset of keys: for example, if half of the keys have logit b and half have logit a with b−a large, each high-weight coefficient is about 2/n and each low-weight coefficient is near 0, so every individual weight tends to 0 while the softmax output is not close to (1/n)Σ v_j. Thus the phrase 'theoretically consistent arithmetic averaging' is not supported by Theorem 3.4. The paper needs either a stronger theorem that controls ∥softmax(QK^T)V − (1/n)Σ v_j∥, or an explicit empirical measurement of this approximation error at the token counts used in the experiments (49 to 3136). Without one of these, the 0.2%–0.7% gains cannot be attributed to faithfulness to the dispersion limit.
  3. [Section 5, Tables 2 and 5] The reported gains are small (83.7 vs 83.5 over MILA in Table 2, and 0.2% in the Table 5 ablation) and all experiments appear to be single-run, with no error bars or multiple seeds. Given typical ImageNet-1K run-to-run variance at this model scale, a 0.2% margin does not by itself establish that SEMA outperforms MILA. Additionally, the larger-resolution comparison in Table 3 is against VMamba-T only, not against MILA at those resolutions. Please provide multiple-seed statistics and, if possible, code or checkpoints, and extend the larger-scale comparison to include MILA.
minor comments (5)
  1. [Definition 3.1 / Table 1] Definition 3.1 requires ϕ:R→R+, but Table 1 lists ϕ(x)=x for linear and related attentions; over the reals this ϕ is not positive. Please clarify that the domain is restricted to values where ϕ is nonnegative, or adjust the definition.
  2. [Theorem 3.4] The theorem's conclusion is phrased as 'there exists an n∈N such that Φ(e^(n))_k < ε', which does not express an asymptotic statement. If the intended claim is eventual dispersion, it should read 'there exists N such that for all n ≥ N'.
  3. [Section 5] There are typos in the notation: '4×10 3' should be 4×10^−3, and '6722,7682' should be 672^2, 768^2.
  4. [Section 6] The limitations paragraph concedes that finite token counts are 'close but not exactly like theory', but it gives no quantitative sense of the approximation error; adding the measurement called for in Major Comment 2 would make this limitation concrete.
  5. [General] The manuscript does not mention whether code or trained checkpoints will be released; for reproducibility of the 0.2%–0.7% claims, a code release statement would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dispersion theorem is a standalone compactness argument, and SEMA's averaging branch is a design choice motivated by, not fitted to, the theorem.

full rationale

Following the paper's derivation chain: Definition 3.1 and Lemma 3.2 establish that any Phi-normalized weight with bounded logits decays as Theta(1/n); Theorem 3.4 supplies compactness conditions under which the logits of the stated layer compositions are uniformly bounded. This is a standalone mathematical statement whose hypotheses (compactness, continuity, convex-combination attention layers) do not contain the conclusion of equal attention. The design step in Section 4 then chooses arithmetic averaging to 'match this asymptotic behavior in the simplest manner' (Eq. 13); the averaging term is not fitted to data, nor is it a renamed version of the theorem's conclusion. The theorem implies equal weights asymptotically, and averaging is one particular efficient implementation of that limit, with the finite-n gap explicitly conceded in Section 6. The empirical claims (Tables 2-5) are obtained by training and evaluation against external benchmarks, and the ablation (Table 5) measures the contribution of averaging rather than deriving it from the theorem. There is no load-bearing self-citation: the compactness argument is proved in the paper and adapted from the external result [29], and the MILA architecture [13] is by another group. The paper's real weakness, that the asymptotic bound does not quantify approximation error at n=49-3136, is a correctness and evidence gap, not a circular reduction of a predicted quantity to a fitted input. Hence no circular step is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The proof of dispersion requires bounded attention logits and a continuous positive kernel. The SEMA design additionally assumes that the infinite-token limit (uniform averaging) is a faithful proxy at finite token counts, an assumption the paper flags in Limitations but never validates quantitatively.

free parameters (1)
  • window size w = 7 (ImageNet), 12 (COCO)
    Selected by hand; the COCO experiments report trying different window sizes and choosing the best. The dispersion theorem does not determine w.
assumptions (3)
  • domain assumption Attention logits are uniformly bounded independently of the number of tokens n.
    Needed for Lemma B.3 and Theorem B.4. The paper claims this follows from compactness of the input feature space and continuity of the layers, but the proof does not explicitly demonstrate a uniform bound across n.
  • domain assumption The normalization kernel phi is continuous and strictly positive on the relevant domain.
    Required for the dispersion bounds in Lemma B.3. Softmax (exp) and linear attention (identity on positive features) satisfy this; the paper defines phi: R to R+ in Definition 3.1.
  • ad hoc to paper The infinite-token limit of attention, uniform averaging, is a sufficient proxy at the finite token counts used in experiments.
    The SEMA design relies on replacing full attention by arithmetic averaging, which is only exact as n goes to infinity. The paper acknowledges in Section 6 that the number of keys is finite and 'close but not exactly like theory,' but no empirical measurement of the approximation error is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging." pith.science (2026). https://pith.science/paper/XWWC6R6I

@misc{pith2026250608297,
  author       = {Pith},
  title        = {Pith review of: SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWWC6R6I}},
  note         = {Machine review of arXiv:2506.08297}
}
read the original abstract

Attention is the critical component of a transformer. Yet the quadratic computational complexity of vanilla full attention in the input size and the inability of its linear attention variant to focus have been challenges for computer vision tasks. We provide a mathematical definition of generalized attention and formulate both vanilla softmax attention and linear attention within the general framework. We prove that generalized attention disperses, that is, as the number of keys tends to infinity, the query assigns equal weights to all keys. Motivated by the dispersion property and recent development of Mamba form of attention, we design Scalable and Efficient Mamba like Attention (SEMA) which utilizes token localization to avoid dispersion and maintain focusing, complemented by theoretically consistent arithmetic averaging to capture global aspect of attention. We support our approach on Imagenet-1k where classification results show that SEMA is a scalable and effective alternative beyond linear attention, outperforming recent vision Mamba models on increasingly larger scales of images at similar model parameter sizes.

Figures

Figures reproduced from arXiv: 2506.08297 by the authors.

Figure 1
Figure 1. SEMA integrates window attention and homogeneous mixing (averaging), to be placed in a [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Homogeneous mixing (averaging) in SEMA attention (bottom) vs. window attention (top). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Mamba like transformers: MILA vs. SEMA, green blocks show our innovations. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The 4 stage architecture of SEMA similar to Swin [22] and MILA [13]. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VideoSEMA: a scalable and efficient Mamba-like attention for video understanding

    cs.CV 2026-07 conditional novelty 4.0 of 10

    VideoSEMA uses SEMA spatial attention plus softmax temporal attention to outperform larger video transformers and Mamba models on K400/SSv2 and degrade less at 1024² resolution.

Reference graph

Works this paper leans on

36 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. ArXiv:2004.05150, 2020

    arXiv 2004

  2. [2]

    Mixformer: Mixing features across windows and dimensions.CVPR, 2022

    Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions.CVPR, 2022

    work page 2022

  3. [3]

    Conditional positional encodings for vision transformers.ICLR, 2023

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers.ICLR, 2023

    work page 2023

  4. [4]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 18613–18624. Curran Associates, Inc., 2020

    work page 2020

  5. [5]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  6. [6]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows.CVPR, 2022

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows.CVPR, 2022

    work page 2022

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    2021

  8. [8]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024

    2024

Show all 36 references
  1. [9]

    Hippo: Recurrent memory with optimal polynomial projections.NeurIPS, 2020

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Re. Hippo: Recurrent memory with optimal polynomial projections.NeurIPS, 2020

    2020

  2. [10]

    M. Guo, C. Lu, Z. Liu, M. Cheng, and S. Hu. Visual attention network.Computational Visual Media, 9(4):733–752, 2023

    2023

  3. [11]

    Flatten transformer: Vision transformer using focused linear attention.ICCV, 2023

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention.ICCV, 2023

    2023

  4. [12]

    Bridging the divide: Reconsidering softmax and linear attention

    Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. Bridging the divide: Reconsidering softmax and linear attention. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in ...

    2024

  5. [13]

    Demystify Mamba in Vision: A Linear Attention Perspective

    Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify Mamba in Vision: A Linear Attention Perspective. NeurIPS, 2024

    2024

  6. [14]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6185–6194, June 2023

    2023

  7. [15]

    Localmamba: Visual state space model with windowed selective scan.arXiv:2403.09338, 2024

    Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan.arXiv:2403.09338, 2024

    2024 arXiv

  8. [16]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention.ICML, 2020

    2020

  9. [17]

    Linear attention mechanism: An efficient attention for semantic segmentation.arXiv:2007.14902, 2020

    Rui Li, Jianlin Su, Chenxi Duan, and Shunyi Zheng. Linear attention mechanism: An efficient attention for semantic segmentation.arXiv:2007.14902, 2020

    2007 arXiv

  10. [18]

    Rethinking vision transformers for MobileNet size and speed.ICCV, 2023

    Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for MobileNet size and speed.ICCV, 2023

    2023

  11. [19]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors,Computer Vision – ECCV 2014, pages...

    2014

  12. [20]

    Def- Mamba: Deformable visual state space model.arXiv:2504.05794, 2025

    Leiye Liu, Miao Zhang, Jihao Yin, Tingwei Liu, Wei Ji, Yongri Piao, and Huchuan Lu. Def- Mamba: Deformable visual state space model.arXiv:2504.05794, 2025

    2025 arXiv

  13. [21]

    VMamba: Visual State Space Model.NeurIPS, 2024

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba: Visual State Space Model.NeurIPS, 2024

    2024

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, October 2021

    2021

  15. [23]

    A ConvNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s . In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11966–11976, Los Alamitos, CA, USA, June 2022. IEEE Computer Society

    2022

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  17. [25]

    Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors,Pro- ceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages ...

    2015

  18. [26]

    Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. Do massively pretrained language models make better storytellers? In Mohit Bansal and Aline Villavicencio, editors,Proceedings of the 23rd Conference on Computational Natural Language Learnin...

    2019

  19. [27]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    2024

  20. [28]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 11

    2017

  21. [29]

    softmax is not enough (for sharp out-of-distribution).arXiv:2410.01104, 2024

    Petar Veliˇckovi´c, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distribution).arXiv:2410.01104, 2024

    2024 arXiv

  22. [30]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415–424, 2022

    2022

  23. [31]

    Low-resolution self-attention for semantic segmentation

    Yu-Huan Wu, Shi-Chen Zhang, Yun Liu, Le Zhang, Xin Zhan, Daquan Zhou, Jiashi Feng, Ming-Ming Cheng, and Liangli Zhen. Low-resolution self-attention for semantic segmentation. arXiv Preprint, arXiv:2310.05026, 2023

    2023 arXiv

  24. [32]

    Differential transformer.arXiv:2410.05258, 2024

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer.arXiv:2410.05258, 2024

    2024 arXiv

  25. [33]

    Mambaout: Do We Really Need Mamba for Vision?CVPR, 2025

    Weihao Yu and Xinchao Wang. Mambaout: Do We Really Need Mamba for Vision?CVPR, 2025

    2025

  26. [34]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022–6031, 2019

    2019

  27. [35]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. InInternational Conference on Learning Representations, 2018

    2018

  28. [36]

    dim64 head2 window size7 # ×2 2 28×28 downsampling, 128

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation.Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):13001– 13008, Apr. 2020. 12 A Ablation Study In this section, we conduct experiments to show that adding the h...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.