Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Modality Agnostic Efficient Long Range Encoder

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MAELRE claims that a single transformer encoder can process very long inputs across text, audio, time series, and vision on one commodity GPU by merging tokens progressively and switching from O(D^2) kernel attention to exact O(N^2)…

desk verdict Useful hybrid of linear attention and token merging with real efficiency wins, but the accuracy and memory-complexity claims are overstated; deserves review after correction. read the letter →

arxiv 2507.19409 v1 pith:2YEIVSFE submitted 2025-07-25 cs.CV

Toufiq Parag , Ahmed Elgammal This is my paper · ORCID
classification cs.CV
keywords long-contexttransformertokenmerginglinearattentionapproximationmodality-agnosticencoderefficientsingle-deviceinferencemultimodalclassificationO(min(ND))memory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a single transformer encoder can handle very long inputs on one ordinary GPU by attacking both quadratic costs at once: shrink the token count as computation deepens and use a cheaper attention whenever tokens still outnumber feature dimensions. It claims this combination, not token merging alone and not approximation alone, preserves or improves accuracy while using a fraction of the memory and FLOPs of existing long-context transformers. If true, long-context classification becomes feasible on a single 16GB device, and the same encoder applies to text, time series, audio, and images with only the input convolution changed.

What carries the argument

The load-bearing object is the attention switch rule and the token-reduction schedule that makes it possible. Each of four blocks contains MHSA/MLP layers, then a strided-convolution reduction (factor $\nu=4$, kernel $2\nu+1$) that cuts token count and doubles dimensionality; when $N_b>D_b$ the block uses kernel attention with a nonnegative feature map $\psi\in\{1+\mathrm{ELU},\mathrm{ReLU},\mathrm{Softplus}\}$ and scaling $\tau=\sqrt{N_b}$, and when $N_b<D_b$ it uses exact dot-product softmax attention. This makes the attention matrix cost $O(D_b^2)$ early and $O(N_b^2)$ late, so the overall memory is $O(\min(N,D)^2)$ rather than $O(N^2)$.

What would settle it

Replace every shallow-layer approximate attention in the MeSH or VGGSound training runs with exact dot-product attention, keeping token merging and all other settings fixed. If the exact-attention model's EBF or top-1 accuracy exceeds the approximate version by more than the margin seen in the all-approximate ablation (about 3.5 points on MeSH, 1.8 on VGGSound), the claim that the finite feature map faithfully approximates early-layer attention is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quadratic cost of self-attention can be tamed by making the attention type follow the token-to-dimension ratio. In shallow blocks, where $N>D$, MAELRE replaces softmax attention with a kernel-based linear attention whose memory is $O(D^2)$; in deeper blocks, after each block's strided-convolution token reduction has cut $N$ by a factor of 4 and doubled $D$, it switches back to standard dot-product attention with $O(N^2)$ memory. The combined memory cost is $O(M^2)$ with $M=\min(N,D)$. The paper reports that this switch, with token merging, gives higher accuracy than sparse-attention long-context baselines and lower cost than multiscale vision transformers, with accuracy above mViTv1 and below mViTv2 on ImageNet, on classification tasks in four modalities.

Load-bearing premise

The load-bearing premise is that a cheap, nonnegative kernel feature map with an empirically chosen scaling factor can stand in for exact dot-product attention in the early layers without losing the information later layers need.

Editorial extensions

If this is right

  • A single 16GB GPU can hold 24 sequences of 17,000 tokens under MAELRE's base configuration, so long-context classification no longer requires high-end hardware or multi-device parallelism.
  • Doubling the input length beyond $D$ no longer squares attention memory in shallow layers; cost grows only through the linear projection terms until merging brings $N$ below $D$.
  • Using approximate attention over all tokens in early layers yields higher accuracy on MeSH text classification than sparse attention over a subset, supporting the paper's hypothesis about early-layer information consolidation.
  • The same four-block encoder reaches state-of-the-art accuracy on VGGSound audio (57.85%) and MosquitoSound time series (95.42%) while using roughly half the memory and FLOPs of the closest competitor.
  • On ImageNet the design matches pooling-based vision transformers at lower cost, but closing the gap to mViTv2 would require adding relative position embedding and residual pooling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because $\tau=\sqrt{N_b}$ was chosen by training convergence rather than derived, a natural extension is to sweep or learn $\tau$ per block; if accuracy is sensitive to it, transfer to new modalities would need per-task tuning.
  • The same $O(M^2)$ attention switch can be stacked under sequence-parallel or ring-attention schemes, so the single-device claim is probably a floor, not a ceiling, for context length.
  • The all-approximate ablation loses about 3.5 EBF points on MeSH and about 1.8 points on VGGSound, suggesting exact deep-layer attention is the accuracy anchor; a testable hypothesis is that the deep-layer switch, not the merging, preserves accuracy.
  • The learned-projection token reduction performs worse than convolution, but this may be a training-difficulty artifact; with longer schedules or warm starts, learned projections could close the gap and remove the convolution prior.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MAELRE, a transformer encoder that combines progressive token merging with two attention modes: an approximate 'linear attention' in shallow layers where the token count exceeds the feature dimension, and standard dot-product attention in deeper layers after merging has reduced the token count. The authors argue that this hybrid gives O(M^2) memory with M = min(N, D), enabling long-context classification on a single device. They evaluate the architecture on four modalities: MeSH multilabel text classification, UCR MosquitoSound time series, VGGSound audio, and ImageNet vision, comparing against RoBERTa, Longformer, BigBird, OPT, mViTv1/v2, and other efficient transformers, and report large reductions in memory and FLOPs with competitive or superior accuracy in most settings. Ablations examine the attention combination, the choice of feature map, and the token merging operator.

Significance. If the empirical results hold, the contribution is valuable: a simple, modality-agnostic recipe that couples progressive token merging with a switch between approximate and exact attention, with meaningful efficiency gains on long inputs and a broad evaluation across four modalities. The paper's strength is its scope: MeSH with over twelve thousand labels, a 280k-sample time-series dataset, VGGSound, and ImageNet, plus ablations on the feature map, the attention combination, and the merging operator. However, the central complexity claim is overstated, and the abstract-level accuracy claim is contradicted by the paper's own tables. The underlying design is still likely salvageable, but the paper as written overclaims in two load-bearing places.

major comments (4)
  1. [§3.1.2, Eq. (2); Introduction and Fig. 1] The claimed O(M^2) memory cost with M = min(N, D) is not what Eq. (2) delivers. Computing phi(Q), phi(K), and phi(V) requires O(N_b D_b) memory before the D_b x D_b product is formed, so a single shallow block costs O(N_b D_b + D_b^2) memory, not O(D_b^2). For N_b > D_b the O(N_b D_b) term dominates and grows linearly in N_b. Table 1 is consistent with this: doubling the token count from 2048 to 4096 at D = 96 halves the maximum batch size from 180 to 90, which is a linear-in-N scaling, not a quadratic-in-D scaling. Please restate the per-block memory as O(N_b D_b + D_b^2) and the compute as O(N_b D_b^2), and remove the O(M^2) formulation from the abstract, introduction, and Figure 1 unless a different accounting is intended.
  2. [§4.4, Table 4; also Abstract and §1] The abstract's 'superior accuracy' claim is contradicted by Table 4: MAELRE-base reaches 82.97% on ImageNet versus 83.78% for mViTv2-base, and MAELRE-small reaches 82.21% versus 83.23% for mViTv2-small. The text in §4.4 acknowledges inferiority to mViTv2, but the paper-level claims in the abstract and introduction are not qualified accordingly. Similarly, Table 7 shows that removing token merging improves text EBF from 43.28 to 45.47, so the claim that token merging uniformly maintains or improves accuracy is also overstated. The defensible claims are lower memory/FLOPs and accuracy advantages on MeSH, MosquitoSound, and VGGSound; the 'superior accuracy' language should be restricted to those comparisons.
  3. [§3.1.2 and Tables 5-6] The approximate attention relies on the unproven assumption that a finite nonnegative feature map with scaling tau = sqrt(N_b) faithfully replaces dot-product attention in shallow layers. The paper states that tau was chosen because it 'led to training convergence' and provides no error bound, unbiasedness guarantee, or sensitivity analysis. Since the all-approximate ablations in Tables 5 and 6 lose 3.48 EBF and 1.84 accuracy points respectively, this assumption is load-bearing. Please add a sensitivity analysis over tau and, ideally, a formal or empirical bound on the approximation error relative to the softmax kernel in the regimes used.
  4. [Tables 1-4; §4.2 and §4.4] All accuracy results appear to be single runs, with no error bars, standard deviations, or seed counts reported. Since many claimed gains are around one point (e.g., 95.42% vs 94.45% on MosquitoSound, 57.85% vs 57.30% on VGGSound), run-to-run variance could change the conclusions. Moreover, the proposed and baseline models are trained with different learning rates and batch sizes by design (§4.2, §4.4), so part of the observed difference may be attributable to training protocol rather than architecture. Please report multiple seeds with intervals and a short hyperparameter search for baselines under a comparable protocol.
minor comments (5)
  1. [§4.4] The dataset description says '1.2M training and 50K training images from 100 categories'; it should read '1.2M training and 50K validation images from 1000 classes.'
  2. [§4.5.2] The stride is called tau after tau was already defined as the attention scaling factor in §3.1.2; rename the stride to avoid notation collision.
  3. [Eq. (2)] The normalization matrix Z' is not defined precisely in the factored form; please state how the row sums are computed without materializing the N x N attention matrix.
  4. [Table 6] The All-dot-base memory entry '1347.38G' appears to mix units with the other entries given in 'M'; also state the memory units explicitly in the caption.
  5. [Table 4] PoolFormer rows lack memory and FLOPs entries; consider filling these for a complete comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are benchmarked externally, and the design choices are validated by ablations rather than fitted to reproduce the reported results.

full rationale

The paper's derivation chain is not circular. It builds on an external kernel attention approximation [Katharopoulos et al., 2020] and is evaluated against external datasets (MeSH, UCR MosquitoSound, VGGSound, ImageNet) and external baseline models, rather than against constants fitted to a target output. The design choices — tau = sqrt(N_b), the block/depth schedules, and stride-4 convolution merging — are either described as empirically selected for convergence or directly tested through ablations (all-approximate, no-merging, pooling vs. linear merging), so they do not function as hidden inputs that force the reported accuracies. The all-approximate and no-merging ablations are honest internal comparisons that could have contradicted the design, and they are not circular validations. There are also no load-bearing self-citations: the authors do not invoke their own prior theorems to justify the central architecture. The main concern in the manuscript is a complexity-accounting issue in Section 3.1.2: Equation (2) still requires materializing phi(Q) and phi(K) of size N_b x D_b, so the per-block memory is O(N_b D_b + D_b^2), not O(D_b^2), and Table 1's batch-size halving as token count doubles is consistent with a linear-in-N term. That is a correctness issue about an overstated theoretical selling point, not circularity, because the claimed bound does not reduce to a fitted parameter, a self-citation, or the paper's own empirical inputs. The central empirical contribution remains self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the fidelity of the linear-attention surrogate, the information-preserving nature of convolutional token merging, and the modality-agnostic input stem. These are domain assumptions rather than derived results. The main hand-set free parameters are the scaling factor tau and the architecture schedule, neither of which is fitted to a benchmark target.

free parameters (3)
  • attention scaling factor tau = sqrt(N_b)
    Chosen empirically in Section 3.1.2 because it led to training convergence; no theoretical derivation is provided, and it directly scales the approximate attention.
  • architecture hyperparameters = B=4, nu=4, D=96, M=[1,3,16,3] base, M=[1,2,11,2] small
    Set by hand in Section 4 and used across all experiments; no search or sensitivity analysis is reported, yet these values determine when the N-versus-D switch happens and hence the efficiency-accuracy tradeoff.
  • feature map psi = 1+ELU, with ReLU and Softplus tested
    Three nonnegative feature maps are tested and yield similar accuracy, but the choice is an input to the approximation, not derived from a kernel error bound.
assumptions (4)
  • domain assumption A finite nonnegative feature map can approximate the exponential dot-product softmax kernel well enough for early-layer attention.
    Invoked in Section 3.1.2 following Katharopoulos et al. [33], but no error bound is provided, and the all-approximate ablations in Tables 5 and 6 show accuracy loss.
  • domain assumption Progressive token merging via strided convolution preserves the information needed for the downstream classification task.
    Section 3.2 and the ablations in Section 4.5.2 test this; the Nored-ELU ablation shows merging is not strictly accuracy-preserving on text, so the assumption is that the efficiency gain justifies any accuracy loss.
  • domain assumption A modality-agnostic input stem with convolution converts text, time series, audio, and images into a common token format without losing task-relevant structure.
    Section 3 states this and uses modality-dependent kernels and strides. It is an empirical assumption validated only by the reported tasks.
  • standard math Standard matrix multiplication associativity and softmax normalization identities hold in the rewritten attention equation.
    Used to rewrite phi(Q)phi(K)^T V as phi(Q)(phi(K)^T V) in Equation 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modality Agnostic Efficient Long Range Encoder." pith.science (2026). https://pith.science/paper/2YEIVSFE

@misc{pith2026250719409,
  author       = {Pith},
  title        = {Pith review of: Modality Agnostic Efficient Long Range Encoder},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YEIVSFE}},
  note         = {Machine review of arXiv:2507.19409}
}
read the original abstract

The long-context capability of recent large transformer models can be surmised to rely on techniques such as attention/model parallelism, as well as hardware-level optimizations. While these strategies allow input lengths to scale to millions of tokens, they do not fundamentally mitigate the quadratic computational and memory complexity of the core attention mechanism. In this paper, we address the challenge of long-context processing on a single device using generic implementations by reducing the quadratic memory footprint and inference cost. Existing approaches to extend the context length for generic single device implementations -- such as token merging and modified attentions -- are often modality specific and attain a suboptimal tradeoff between accuracy and efficiency. To overcome these limitations, we propose MAELRE (Modality Agnostic Efficient Long Range Encoder), a unified and efficient transformer architecture designed for long-range encoding across diverse modalities. MAELRE integrates token merging with attention approximation, progressively merging tokens at different stages of internal computational blocks. It employs a lightweight attention approximation when the number of tokens is large, and switches to standard dot-product attention as the sequence becomes shorter through successive aggregation. We demonstrate that MAELRE achieves superior accuracy while reducing computational cost compared to existing long-context models on classification tasks spanning multiple modalities, including text, time series, audio, and vision.

Figures

Figures reproduced from arXiv: 2507.19409 by the authors.

Figure 1
Figure 1. Illustrating advantage of the proposed MAELRE through attention matrix size with respect to token number and dimensions. The number and dimensions of tokens, and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 53 canonical work pages

  1. [1]

    GQA: Training gener- alized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyan- skiy, Federico Lebron, and Sumit Sanghai. GQA: Training gener- alized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 3

    work page 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Ana Recasens, Antoni Miech, Marjan Ghazvininejad, Marc’Aurelio Ranzato, Yuval Avidor, Yuandong Zhai, Minseok Lee, Naman Goyal, Wei-Cheng Hsu, et al. Flamingo: a visual language model for few-shot learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. 4

    work page 2022

  3. [3]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 1, 2, 5, 6

    arXiv 2004

  4. [4]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 2

    arXiv 2022

  5. [5]

    Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with rmt. arXiv preprint arXiv:2304.11062, 2023. 2, 3

    arXiv 2023

  6. [6]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 721–725. IEEE, 2020. 7

    work page 2020

  7. [7]

    Classification of long sequential data using circular dilated convolutional neural networks

    Lei Cheng, Ruslan Khalitov, Tong Yu, Jing Zhang, and Zhirong Yang. Classification of long sequential data using circular dilated convolutional neural networks. Neurocomputing, 518:50–59, 2023. 6, 7

    work page 2023

  8. [8]

    Gen- erating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Gen- erating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 2

    arXiv 1904

Show all 70 references
  1. [9]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Alex Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Col- well, and Adrian Weller. Rethinking attention with performers. In International Conference on...

    2021

  2. [10]

    Ran- daugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Ran- daugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Sys- tems (NeurIPS) Workshops, 2020. 8

    2020

  3. [11]

    Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V . Le. Funnel- transformer: Filtering out sequential redundancy for efficient lan- guage processing. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3

    2020

  4. [12]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Rep- resentations (ICLR), 2024. 3

    2024

  5. [13]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Sys- tems (NeurIPS), volume 35, pages 16344–16359, 2022. 1, 3, 6, 7

    2022

  6. [14]

    The ucr time series classifica- tion archive, 2019

    Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin- Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series classifica- tion archive, 2019. Accessed: 2024-12-29. 6

    2019

  7. [15]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv, 2405.04434, 2024. 3

    2024 arXiv

  8. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009. 8

    2009

  9. [17]

    fvcore: A collection of core libraries for com- puter vision

    FacebookAI. fvcore: A collection of core libraries for com- puter vision. https://github.com/facebookresearch/ fvcore, 2019. Accessed: 2024-12-27. 5 9

    2019

  10. [18]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 6824–6835, 2021. 1, 2, 4, 5, 7, 8

    2021

  11. [19]

    Fu, Tri Dao, Khaled K

    Daniel Y . Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher R ´e. Hungry hungry hippos: To- wards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. 3

    2022 arXiv

  12. [20]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023. 2, 3

    2023 arXiv

  13. [21]

    Liu, David Har- wath, Leonid Karlinsky, Hilde Kuehne, and James R

    Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Har- wath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Con- trastive audio-visual masked autoencoder. In The Eleventh Interna- tional Conference on Learning Representations, 2023. 7

    2023

  14. [22]

    Choudhury, Saurabh M

    Saurabh Goyal, Anamitra R. Choudhury, Saurabh M. Raje, Venkate- san T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power- bert: Accelerating bert inference via progressive word-vector elimi- nation. In Proceedings of the 37th International Conference on Ma- chine Learni...

    2020

  15. [23]

    Levit: a vision transformer in convnet’s clothing for faster inference

    Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv ´e J ´egou, and Matthijs Douze. Levit: a vision transformer in convnet’s clothing for faster inference. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pag...

    2021

  16. [24]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 3

    2023 arXiv

  17. [25]

    Longt5: Efficient text-to-text transformer for long sequences

    Mandy Guo, Joshua Ainslie, David Uthus, and Santiago Onta ˜n´on. Longt5: Efficient text-to-text transformer for long sequences. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa...

    2022

  18. [26]

    Flatten transformer: Vision transformer using focused linear atten- tion

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear atten- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 3

    2023

  19. [27]

    Masked autoencoders are scalable vision learn- ers

    Kaiming He, Xiangyu Chen, Saining Xie, Yanghao Li, Piotr Doll ´ar, and Ross Girshick. Masked autoencoders are scalable vision learn- ers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022. 8

    2022

  20. [28]

    Rethinking spatial dimensions of vision transformers

    Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Jun- suk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11936–11945, 2021. 2, 4

    2021

  21. [29]

    Thomas Hofmann, Bernhard Sch ¨olkopf, and Alexander J. Smola. Kernel methods in machine learning. The MIT Press, 2008. 4

    2008

  22. [30]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Wein- berger. Deep networks with stochastic depth. In European Confer- ence on Computer Vision (ECCV) , pages 646–661. Springer, 2016. 8

    2016

  23. [31]

    Transformers: State-of-the-art machine learning for pytorch, tensorflow, and jax

    HuggingFace. Transformers: State-of-the-art machine learning for pytorch, tensorflow, and jax. https : / / github . com / huggingface/transformers, 2023. 5

    2023

  24. [32]

    An empirical survey of data augmentation for time series classification with neural networks

    Brian Kenji Iwana and Seiichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. PLOS ONE, 16(7):e0254841, jul 2021. 6

    2021

  25. [33]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020. 1, 2, 3, 4, 8

    2020

  26. [34]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Rep- resentations, 2020. 2, 7

    2020

  27. [35]

    Sequence parallelism: Long sequence training from sys- tem perspective

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from sys- tem perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2391–2404, T...

    2023

  28. [36]

    Mvitv2: Im- proved multiscale vision transformers for classification and detec- tion

    Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Im- proved multiscale vision transformers for classification and detec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Rec...

    2022

  29. [37]

    Efficientformer: Vi- sion transformers at mobilenet speed

    Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vi- sion transformers at mobilenet speed. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2022. 1, 2, 9

    2022

  30. [38]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023. 1, 2, 3, 6

    2023 arXiv

  31. [39]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 4, 5, 6

    1907 arXiv

  32. [40]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10012–10022, 2021. 1, 2, 4

    2021

  33. [41]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Repre- sentations, 2017. 5

    2017

  34. [42]

    Decoupled weight decay regular- ization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regular- ization. In International Conference on Learning Representations ,

  35. [43]

    Token pooling in vi- sion transformers for image classification

    Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vi- sion transformers for image classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023. 2

    2023

  36. [44]

    Lo- cating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Lo- cating and editing factual associations in gpt. In Advances in Neural Information Processing Systems , volume 35, pages 17204–17219,

  37. [45]

    Scalable vision transformers with hierarchical pooling

    Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transformers with hierarchical pooling. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 377–386, October 2021. 2

    2021

  38. [46]

    Investigating efficiently ex- tending transformers for long input summarization

    Jason Phang, Yao Zhao, and Peter Liu. Investigating efficiently ex- tending transformers for long input summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023. 1, 7

    2023

  39. [47]

    Rabe and Charles Staats

    Markus N. Rabe and Charles Staats. Self-attention does not need o(n2) memory. arXiv preprint arXiv:2112.05682, 2021. 1, 3

    2021 arXiv

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. arXiv preprint arX...

    2021 arXiv

  41. [49]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. 3

    1911 arXiv

  42. [50]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(56):1929–1958, 2014. 8

    1929

  43. [51]

    Cox, Ruchir Puri, and Rameswar Panda

    Matt Stallone, Vaibhav Saxena, Leonid Karlinsky, Bridget McGinn, Tim Bula, Mayank Mishra, Adriana Meza Soria, Gaoyuan Zhang, 10 Aditya Prasad, Yikang Shen, Saptha Surendran, Shanmukha Guttula, Hima Patel, Parameswaran Selvam, Xuan-Hong Dang, Yan Koyf- man, Atin Sood, Rogerio F...

    2024 arXiv

  44. [52]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016. 6

    2016

  45. [53]

    Sparse sinkhorn attention

    Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In Proceedings of the 37th International Conference on Machine Learning, pages 9438–9447, 2020. 3

    2020

  46. [54]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Long range arena: A benchmark for efficient transformers. InInternational Conference on Learning Representations (ICLR), 2021. 1, 7

    2021

  47. [55]

    Atten- tion is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atten- tion is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 4

    2017

  48. [56]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. 2, 3

    2022 arXiv

  49. [57]

    Huang, Krzysztof Choromanski, Valerii Likhosherstov, and Adrian Weller

    Sinong Wang, Belinda Z. Huang, Krzysztof Choromanski, Valerii Likhosherstov, and Adrian Weller. Linformer: Self-attention with linear complexity. In Advances in Neural Information Process- ing Systems, volume 33, pages 5290–5300. Curran Associates, Inc.,

  50. [58]

    Meshup: A corpus for full text biomedical document indexing

    Xindi Wang, Robert E Mercer, and Frank Rudzicz. Meshup: A corpus for full text biomedical document indexing. arXiv preprint arXiv:2204.13604, 2022. 5, 6

    2022 arXiv

  51. [59]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github. com/rwightman/pytorch-image-models , 2019. 8

    2019

  52. [60]

    Ef- fective long-context scaling of foundation models

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhos- ale, Sergey Edunov, Mike Lewis,...

    2024

  53. [61]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 10856–10865,

  54. [62]

    Megabyte: Predicting million- byte sequences with multiscale transformers

    Lili Yu, D ´aniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million- byte sequences with multiscale transformers. arXiv preprint arXiv:2305.07185, 2023. 2, 3

    2023 arXiv

  55. [63]

    Metaformer is actually what you need for vision

    Weihao Yu, Cheng Si, Yujun Yu, Minghao Luo, Tao Zhou, and Jiashi Du. Metaformer is actually what you need for vision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 10819–10829, 2022. 1, 2, 8, 9

    2022

  56. [64]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Jun- suk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. InProceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 6023–6032, 2019. 8

    2019

  57. [65]

    Big bird: Trans- formers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems (NeurIPS),...

    2020

  58. [66]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Interna- tional Conference on Learning Representations, 2018. 6

    2018

  59. [67]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 5, 6

    2022 arXiv

  60. [68]

    Adavit: Adaptive vision transformers for efficient image recognition

    Zhize Zhao, Hong Xu, Bo Zhang, Haoyi Zhang, Zhe Wang, Xiaohui Zhai, Lijun Zhang, and Zhiqiang Wang. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10650–10...

    2022

  61. [69]

    Long-short transformer: Efficient transformers for language and vision

    Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. In Ad- vances in Neural Information Processing Systems, volume 34, pages 16191–16203, 2021. 3

    2021

  62. [70]

    Multiscale audio spectro- gram transformer for efficient audio classification

    Wentao Zhu and Mohamed Omar. Multiscale audio spectro- gram transformer for efficient audio classification. arXiv preprint arXiv:2303.10757, 2023. 7 11

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.