Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Leaner Transformers: More Heads, Less Depth

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that multi-head attention improves the conditioning of the attention block, making it possible to trade depth for heads and cut transformer parameter counts by 30–50% without losing accuracy.

desk verdict The empirical head-for-depth sweep is a genuinely useful finding, but the theory is a repackaged width result and the experiments don't isolate head count from total width. read the letter →

arxiv 2505.20802 v1 pith:TQPGT6PU submitted 2025-05-27 cs.LG cs.CV

Hemanth Saratchandran , Damien Teney , Simon Lucey This is my paper
classification cs.LGcs.CV MSC 60B2015A1268T07
keywords transformersmulti-headattentionconditionnumberrandommatrixtheorydepthvswidthtrade-offparameterefficiencyImageNet-1kGLUE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper contends that many transformers are over-sized because their depth is doing work that could be done by extra attention heads. Its theoretical claim is that concatenating many attention-head matrices improves the condition number of the attention block, driving it close to 1 when the model is much wider than the sequence length. The authors use this to propose a design rule: add heads, subtract layers. They re-train a range of vision and language transformers and report matching or better accuracy with 30–50% fewer parameters, which matters because it suggests a practical, architecture-agnostic way to shrink models.

What carries the argument

The load-bearing object is the condition number $\kappa(A) = \sigma_1(A)/\sigma_k(A)$ of the concatenated attention block $A=[A_1,\ldots,A_h]$, together with the random-matrix singular-value estimates for Gaussian matrices. The paper's argument is that because each head is $N \times D/h$, the concatenated block is $N \times D$, so when the total width $D$ greatly exceeds the sequence length $N$, the extreme singular values concentrate near $\sqrt{D} \pm \sqrt{N}$ and the ratio approaches 1. This provides Theorem 3.2. The mechanism is then promoted from a mathematical observation to a design principle: multi-head attention acts as an 'implicit conditioner,' making extra heads a substitute for extra layers.

What would settle it

Hold a transformer's total width $D = h \cdot d$ fixed and vary only the number of heads $h$ (for example, $h=4$ with $d=64$ vs. $h=8$ with $d=32$). If the measured condition number of the attention block and the downstream accuracy do not improve as $h$ grows, the claim that head count itself conditions the block—rather than the mere increase in total width—is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that multi-head attention functions as an implicit conditioner of the attention matrix. For the concatenated block $A = [A_1, \ldots, A_h]$ of $h$ identically distributed Gaussian head matrices, Theorem 3.2 shows that the condition number $\kappa(A) = \sigma_1(A)/\sigma_k(A)$ is close to 1 when the total width $D$ is much larger than the sequence length $N$, and that with a fixed head dimension $d$, $\kappa(A) \to 1$ as $h \to \infty$. The argument relies on standard random-matrix estimates $\sigma_{\min} \approx \sqrt{D} - \sqrt{N}$ and $\sigma_{\max} \approx \sqrt{D} + \sqrt{N}$, which make the ratio tend to 1. The paper then verifies empirically that measured condition numbers of attention layers in a trained ViT fall as head count rises, and treats this as evidence that heads can substitute for depth across image and language benchmarks.

Load-bearing premise

The whole argument depends on treating the outputs of real attention heads as independent, identically distributed Gaussian random matrices; actual heads produce softmax-weighted value vectors with normalized rows and strong correlations through shared query, key, and value projections, so the singular-value bounds used in the proof need not apply.

Editorial extensions

If this is right

  • Vision transformers (ViT-B, DeiT-B, XCiT-M, TNT-B, VOLO-d3, DaViT-B) re-designed with more heads and fewer layers match or surpass baseline accuracy with 30–50% fewer parameters on ImageNet-1k.
  • The same trade-off holds for language models: a 10-layer/24-head Crammed BERT matches the 16-layer/12-head original on GLUE at 84M vs 119M parameters, and a 4-layer/16-head GPT-2 beats the 12-layer/12-head baseline on TinyStories at 64M vs 89M parameters.
  • Nyströmformers on Long-Range Arena tasks improve accuracy with 1 layer and 4 heads over the original 2-layer/2-head configuration, with equal or fewer parameters.
  • Measured condition numbers of attention layers in a trained ViT-B fall as the number of heads increases, directly supporting Theorem 3.2's prediction.
  • The design rule is architecture-agnostic and complementary to attention-efficiency methods like Nyström approximation, since conditioning benefits appear on top of those mechanisms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theorem is proved for i.i.d. Gaussian head matrices; actual attention heads are softmax-weighted value projections and correlated across heads, so whether the same conditioning bound holds for real attention is an open empirical question. A width-matched ablation (varying head count at fixed total width) would separate the effect of head count from the effect of width.
  • If improved conditioning is indeed the mechanism, then other interventions that improve attention-block conditioning—weight normalization, spectral normalization, better initialization—might achieve the same depth-for-heads trade-off without changing architecture.
  • The paper's limitations note that no quantitative theory predicts the depth–head trade-off; predicting that trade-off from condition numbers alone is a natural next step the framework does not yet provide.
  • All empirical results are on models up to about 200M parameters; whether the benefit survives at billion-parameter scale is untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes that multi-head attention improves the conditioning of the attention block, based on Theorem 3.2, which states that concatenated i.i.d. Gaussian attention head matrices have condition number near 1 when total width D >> N and approach 1 as the number of heads h tends to infinity at fixed head dimension. The authors use this principle to redesign several transformer architectures (ViT, DeiT, XCiT, TNT, VOLO, DaViT, Crammed BERT, GPT-2, and Nyströmformer), reporting parameter reductions of 30–50% with maintained or improved accuracy. Experiments include conditioning measurements, head/depth sweeps, MLP width comparisons, and benchmarks on ImageNet-1k, GLUE, TinyStories, and Long-Range Arena.

Significance. If the empirical design rule were established, the practical payoff would be real: many standard transformers appear overparameterized and could be made substantially smaller with maintained accuracy across architectures and tasks. The paper deserves credit for a broad and falsifiable empirical program, including direct conditioning measurements, and for candidly stating in the Limitations section that a theoretical explanation of the depth/head trade-off is still missing. However, the central theoretical principle as stated is not supported for actual transformers because the Gaussian assumption does not match softmax attention, and the headline experiments are confounded by simultaneous changes in MLP width and by single-run evaluation. The result is therefore best understood as a promising empirical finding that requires a corrected theoretical framing and stronger ablations, rather than as a demonstrated theoretical discovery.

major comments (4)
  1. [§3.2, Theorem 3.2 and Eq. (3)] Theorem 3.2 models attention head outputs Ai as i.i.d. Gaussian matrices, but actual attention head outputs are A_i = softmax(Q_i K_i^T) V_i, whose rows are convex combinations of value vectors and are therefore nonnegative, bounded, and row-stochastic. The singular-value asymptotics in Eq. (3) are for i.i.d. Gaussian matrices and do not apply to such structured matrices. Consequently, Theorem 3.2 does not establish that real multi-head attention blocks are well-conditioned, and the paper's central theoretical claim that multi-head attention acts as an implicit conditioner is unsupported.
  2. [§3.2, Observation and Eq. (13)] In the fixed-head-dimension case, the number of heads h enters the theorem only through the total width D = dh, and the authors themselves observe that letting d grow gives the same result. The theorem is therefore a statement about wide random matrices, not about the number of heads per se. The empirical protocol in Fig. 2 and Fig. 3 also varies heads while keeping head dimension fixed, so the measured conditioning improvement is inseparable from a pure width effect. Moreover, for low head counts the regime D >> N is not even satisfied in the experiments (for ViT-B with N=196 tokens and d=64, h=1 gives D=64 < N), so the theorem's assumptions do not cover the full experimental range.
  3. [§4.1.2, Table 4 and §4.1.1, Fig. 3] The best configurations change the MLP width simultaneously with heads and depth; for example, ViT-B goes from MLP dimension 3072 to 1536, and VOLO-d3 and DaViT-B also reduce MLP dimensions in their redesigned variants. The reported parameter reductions and accuracy gains therefore cannot be attributed to the head/depth trade-off alone. An ablation that holds MLP width constant while varying heads and depth, together with multiple seeds or error bars for the points in Fig. 3, is required to support the proposed design rule.
  4. [Limitations and Open Questions] The manuscript explicitly states that a theoretical explanation for the depth/head trade-off is still missing and that the effect on task accuracy rests on empirical results. Combined with the mismatch between Theorem 3.2 and actual attention outputs, this concedes the key inference the paper claims to draw. The paper should be reframed as an empirical study of the head/depth trade-off, with the Gaussian result presented as a heuristic or analogy rather than a proof for transformer attention.
minor comments (5)
  1. [§3.2, Eq. (3)] In Eq. (3), the second stated approximation should refer to the maximum singular value sigma_1(X), not sigma_m(X); as written, both lines claim the same quantity.
  2. [§3.1, Definition 3.1] The phrase "full-rank,n×m matrix" contains a formatting error; it should read "full-rank n×m matrix" or "full-rank, n×m matrix".
  3. [Fig. 9 caption and §4.3 text] The caption of Fig. 9 says "with a single layers" and the surrounding text repeats the phrase; it should be "a single layer".
  4. [Table 1 caption] The caption says "For each task our learner variant achieves comparable performance"; "learner" should be "leaner".
  5. [§3.2, Proof of Theorem 3.2] The proof refers to "each A1" where it should refer to each Ai; this is a typo that does not affect the argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; Theorem 3.2 is an independent random-matrix result and the empirical tests are new training runs.

full rationale

The paper's central derivation is Theorem 3.2, which states that a concatenation of i.i.d. Gaussian random matrices has condition number near 1 when the total width D satisfies D >> N and, with fixed head dimension d, as h goes to infinity. The proof uses standard singular-value bounds for random matrices (Eq. 3) from Vershynin's textbook, not any fitted parameter or the paper's own empirical data. The conditioning measurements in Fig. 2 and the accuracy comparisons on ImageNet-1k, GLUE, TinyStories, and LRA are new training runs, so no prediction is a renamed fit. The paper explicitly acknowledges that the same convergence would occur by increasing d rather than h (Observation after Theorem 3.2), which reveals a width/head confound but does not make the theorem circular. Likewise, the Gaussian assumption on attention matrices is an idealization whose mismatch with softmax(qk^T)v attention is a validity concern, not a circularity concern. There is no load-bearing self-citation: the cited random-matrix facts are standard, parameter-free, and independent of the present paper's results. Consequently, the claimed derivation does not reduce to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a Gaussian idealization of attention outputs that is not justified for real softmax attention, plus the unproven assumptions that conditioning drives accuracy and that depth and heads are substitutable. No new physical or mathematical entities are introduced.

free parameters (2)
  • attention head dimension d = 64
    Set to 64 in the ImageNet experiments; with head dimension fixed, increasing the number of heads increases total attention width, which is the actual driver of the conditioning effect in Theorem 3.2.
  • MLP ratio in leaner variants = 2 (baseline 4)
    Tables 4 and 5 pair more heads and fewer layers with a halved MLP hidden width, so the reported parameter savings are not attributable to head count alone.
assumptions (5)
  • ad hoc to paper Attention head outputs are i.i.d. Gaussian random matrices.
    Assumed in Theorem 3.2; real attention outputs are softmax(qk^T)v and are correlated, bounded, and not Gaussian.
  • domain assumption The attention block is much wider than the number of tokens (D >> N).
    Required for the random matrix singular value bounds; true for ViT-B with 196 tokens and width 768, but not guaranteed for all transformers.
  • standard math Singular value concentration bounds for i.i.d. Gaussian matrices.
    The proof of Theorem 3.2 uses the known bounds sigma_max approx sqrt(n)+sqrt(m) and sigma_min approx sqrt(n)-sqrt(m), citing Vershynin [37].
  • domain assumption A lower condition number of attention layers improves optimization and downstream accuracy.
    Asserted in Section 3; the paper's Limitations section states that the effect on accuracy rests on empirical results and that the link to training dynamics is open.
  • domain assumption Depth can be traded for width (heads) in transformers without losing performance.
    Introduced in Section 3.3 as a hypothesis ('We therefore hypothesize'); Limitations states no theoretical explanation is available for this balance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leaner Transformers: More Heads, Less Depth." pith.science (2026). https://pith.science/paper/TQPGT6PU

@misc{pith2026250520802,
  author       = {Pith},
  title        = {Pith review of: Leaner Transformers: More Heads, Less Depth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TQPGT6PU}},
  note         = {Machine review of arXiv:2505.20802}
}
read the original abstract

Transformers have reshaped machine learning by utilizing attention mechanisms to capture complex patterns in large datasets, leading to significant improvements in performance. This success has contributed to the belief that "bigger means better", leading to ever-increasing model sizes. This paper challenge this ideology by showing that many existing transformers might be unnecessarily oversized. We discover a theoretical principle that redefines the role of multi-head attention. An important benefit of the multiple heads is in improving the conditioning of the attention block. We exploit this theoretical insight and redesign popular architectures with an increased number of heads. The improvement in the conditioning proves so significant in practice that model depth can be decreased, reducing the parameter count by up to 30-50% while maintaining accuracy. We obtain consistent benefits across a variety of transformer-based architectures of various scales, on tasks in computer vision (ImageNet-1k) as well as language and sequence modeling (GLUE benchmark, TinyStories, and the Long-Range Arena benchmark).

Figures

Figures reproduced from arXiv: 2505.20802 by the authors.

Figure 1
Figure 1. We redesign popular transformers models with an in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Empirical measurement of the condition number of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Accuracy on ImageNet-1k of variants of ViT-B with the original depth (12 layers, left) or reduced to 8 layers (right). Each point [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Similar experiments as Fig [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Additional variants of ViT-B with different numbers of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Other vision transformer architectures. We plot improve [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Accuracy on the ListOps task of the LRA benchmark with variants of the Nystr [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Accuracy on the text classification task of the LRA benchmark with variants of the Nystr [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 32 canonical work pages

  1. [1]

    A deep conditioning treatment of neural networks

    Naman Agarwal, Pranjal Awasthi, and Satyen Kale. A deep conditioning treatment of neural networks. In Algorithmic Learning Theory, pages 249–305. PMLR, 2021. 1, 2, 4

    work page 2021

  2. [2]

    Xcit: Cross-covariance image transformers

    Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bo- janowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Na- talia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014–20027, 2021. 2, 5

    work page 2021

  3. [3]

    On the op- timization of deep networks: Implicit acceleration by over- parameterization

    Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the op- timization of deep networks: Implicit acceleration by over- parameterization. In International conference on machine learning, pages 244–253. PMLR, 2018. 1, 2, 4

    work page 2018

  4. [4]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 1

    2020

  5. [5]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Alex Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations (ICLR), 2021. 2

    work page 2021

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1

    arXiv 2018

  7. [7]

    Davit: Dual attention vision transform- ers

    Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transform- ers. In European conference on computer vision, pages 74–

    work page

  8. [8]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1, 3, 4

    arXiv 2010

Show all 45 references
  1. [9]

    Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023. 2, 4, 5, 7, 1

    2023 arXiv

  2. [10]

    Drive like a human: Rethinking au- tonomous driving with large language models

    Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking au- tonomous driving with large language models. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 910–919, 2024. 1

    2024

  3. [11]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Aadi Thite, Eric Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2021. 5

    2021 arXiv

  4. [12]

    Cramming

    Jonas Geiping. Cramming. https://github.com/ JonasGeiping/cramming, 2023. 1

    2023

  5. [13]

    Cramming: Training a language model on a single gpu in one day

    Jonas Geiping and Tom Goldstein. Cramming: Training a language model on a single gpu in one day. In Inter- national Conference on Machine Learning , pages 11117– 11143. PMLR, 2023. 5

    2023

  6. [14]

    Transformer in transformer

    Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in neural information processing systems, 34:15908–15919,

  7. [15]

    Neu- ral tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl ´ement Hongler. Neu- ral tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing sys- tems, 31, 2018. 1, 2, 4

    2018

  8. [16]

    On the size of convolutional neural networks and generalization per- formance

    Maya Kabkab, Emily Hand, and Rama Chellappa. On the size of convolutional neural networks and generalization per- formance. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 3572–3577. IEEE, 2016. 1

    2016

  9. [17]

    Re- former: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Re- former: The efficient transformer. In International Confer- ence on Learning Representations (ICLR), 2020. 2

    2020

  10. [18]

    The depth-to-width interplay in self-attention

    Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Am- non Shashua. The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467, 2020. 1, 2

    2006 arXiv

  11. [19]

    Limits to depth efficiencies of self-attention

    Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Am- non Shashua. Limits to depth efficiencies of self-attention. NeurIPS, 33:22640–22651, 2020. 1, 2

    2020

  12. [20]

    On tighter generalization bound for deep neu- ral networks: Cnns, resnets, and beyond

    Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis Haupt, and Tuo Zhao. On tighter generalization bound for deep neu- ral networks: Cnns, resnets, and beyond. arXiv preprint arXiv:1806.05159, 2018. 1

    2018 arXiv

  13. [21]

    Loss land- scapes and optimization in over-parameterized non-linear systems and neural networks

    Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss land- scapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85–116, 2022. 1, 4

    2022

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 1, 2

    2021

  15. [23]

    The expressive power of neural networks: A view from the width

    Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  16. [24]

    Transfusion: Multi-modal fusion network for semantic segmentation

    Abhisek Maiti, Sander Oude Elberink, and George V ossel- man. Transfusion: Multi-modal fusion network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6536– 6546, 2023. 1

    2023

  17. [25]

    Numerical optimiza- tion

    Jorge Nocedal and Stephen J Wright. Numerical optimiza- tion. Springer, 1999. 2, 3

    1999

  18. [26]

    The impact of depth and width on transformer language model generalization

    Jackson Petty, Sjoerd van Steenkiste, Fei Sha, Ishita Das- gupta, Dan Garrette, and Tal Linzen. The impact of depth and width on transformer language model generalization. openreview, 2023. 1, 2

    2023

  19. [27]

    Exponential expressivity in deep neural networks through transient chaos

    Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl- Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016. 2

    2016

  20. [28]

    Tiny-stories-gpt

    Praveen Raja. Tiny-stories-gpt. https://github.com/ PraveenRaja42/Tiny-Stories-GPT. 1

  21. [29]

    Trajectron++: Multi-agent generative trajec- 10 tory forecasting with heterogeneous data for control

    Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Multi-agent generative trajec- 10 tory forecasting with heterogeneous data for control. arXiv preprint arXiv:2001.03093, 2, 2020. 1

    2001 arXiv

  22. [30]

    Representational strengths and limitations of transformers

    Clayton Sanford, Daniel J Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. NeurIPS, 36:36677–36707, 2023. 1, 2

    2023

  23. [31]

    Real analysis: measure theory, integration, and Hilbert spaces

    Elias M Stein and Rami Shakarchi. Real analysis: measure theory, integration, and Hilbert spaces. Princeton University Press, 2009. 1

    2009

  24. [32]

    How to train your vit? data, augmentation, and regularization in vision transformers

    Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270, 2021. 2, 4, 5, 1

    2021 arXiv

  25. [33]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Julian Heinrich, Dai Hua, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006,

    2011 arXiv

  26. [34]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 1, 2, 5

    2021

  27. [35]

    Width is less important than depth in relu neural networks

    Gal Vardi, Gilad Yehudai, and Ohad Shamir. Width is less important than depth in relu neural networks. In Conference on learning theory, pages 1249–1281. PMLR, 2022. 2

    2022

  28. [36]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 1, 3, 6

    2017

  29. [37]

    High-dimensional probability: An intro- duction with applications in data science

    Roman Vershynin. High-dimensional probability: An intro- duction with applications in data science . Cambridge uni- versity press, 2018. 3

    2018

  30. [38]

    Glue: A multi-task benchmark and analysis platform for natural language un- derstanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language un- derstanding. arXiv preprint arXiv:1804.07461, 2018. 2, 5

    2018 arXiv

  31. [39]

    Linformer: Self-attention with linear complex- ity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complex- ity. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2

    2020

  32. [40]

    Github repository, 2021

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Fei Tan, Glenn Fung, Vikas Singh, Xiaodong Yuan, Sung- soo Ahn Wang, Dimitris Papailiopoulos, and Katerina Fragkiadaki. Github repository, 2021. 1

    2021

  33. [41]

    Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximat- ing self-attention. Proceedings of the AAAI Conference on Artificial Intelligence, 2021. 2, 6, 8, 9, 1

    2021

  34. [42]

    V olo: Vision outlooker for visual recog- nition

    Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. V olo: Vision outlooker for visual recog- nition. IEEE transactions on pattern analysis and machine intelligence, 45(5):6575–6586, 2022. 2, 5

    2022

  35. [43]

    cosformer: rethinking softmax in attention

    Q Zhen, W Sun, H Deng, D Li, Y Wei, B Lv, J Yan, L Kong, and Y Zhong. cosformer: rethinking softmax in attention. In International Conference on Learning Representations ,

  36. [44]

    Understanding generalization and optimization performance of deep cnns

    Pan Zhou and Jiashi Feng. Understanding generalization and optimization performance of deep cnns. In International Conference on Machine Learning, pages 5960–5969. PMLR,

  37. [45]

    A robustly optimized bert pre-training approach with post-training

    Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized bert pre-training approach with post-training. In Proceedings of the 20th chinese national conference on com- putational linguistics, pages 1218–1227, 2021. 1 11 Leaner Transformers: More Heads, Less Depth Suppl...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.