Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Doubly Stochastic attention is the most robust self-attention mechanism for vision transformers under data corruption.

desk verdict Useful empirical comparison with a real confound: DS attention's robustness lead under fog is visible in these runs, but per-mechanism hyperparameter tuning means the paper does not prove the attention mechanism is the cause. read the letter →

arxiv 2507.20453 v3 pith:S45YUVT2 submitted 2025-07-28 cs.LG

Camilo Tamayo-Rousseau , Yunjia Zhao , Yiqun Zhang , Randall Balestriero This is my paper
classification cs.LG
keywords self-attentionvisiontransformersrobustnessdatacorruptionfogdoublystochasticattentionSinkhornnormalizationrelativeaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the internal normalization of a vision transformer's self-attention mechanism changes how much accuracy the model loses when its images are corrupted. It compares Softmax, Sigmoid, Linear, Cosine, and Doubly Stochastic attention under fog corruption on CIFAR-10, CIFAR-100, and Imagenette, measuring both absolute accuracy and accuracy relative to each model's clean baseline. The central claim is that Doubly Stochastic attention—whose attention matrix is normalized so that every row and every column sums to one—is the most robust choice, consistently keeping the highest relative accuracy when training data, or both training and testing data, are corrupted. The authors argue that absolute accuracy alone misleads, because Softmax often reaches higher absolute scores yet degrades more sharply under fog. This matters for real-world deployment, where the choice of attention mechanism could serve as an architectural lever for reliability on imperfect data.

What carries the argument

The load-bearing object is Doubly Stochastic attention, defined by applying Sinkhorn's algorithm to the scaled query-key product $\frac{QK^T}{\sqrt{d}}$ before the multiplication by values $V$. Sinkhorn iteration alternates row and column normalizations until every row and every column of the attention matrix sums to one, with iteration count and step size $\epsilon$ as hyperparameters. The paper credits this double normalization with preventing any single token from dominating the attention budget, which it argues stabilizes training and inference when inputs are corrupted. The other load-bearing piece is the evaluation metric, relative accuracy, defined as corrupted-case accuracy divided by clean-baseline accuracy, which is what makes the robustness ranking visible despite Softmax's higher absolute scores.

What would settle it

Train all five attention mechanisms in an otherwise identical Vision Transformer (fixed hidden size, MLP width, heads, batch size, learning rate, and gradient clipping) on fog-corrupted CIFAR-10 and compare relative accuracy against the tuned-configuration results; if Doubly Stochastic attention no longer consistently leads in the corrupted-training scenarios, the ranking is driven by the tuning confounds rather than the normalization scheme. A second check is to rerun the original protocol with multiple seeds and verify that the reported 0.1–3.8 percentage-point margins exceed run-to-run variability.

Watch

Extended reading notes

Core claim

Across three datasets, Doubly Stochastic attention retains the largest fraction of its clean-data accuracy when fog corrupts the training set, the test set, or both. The mechanism applies Sinkhorn's algorithm to the scaled query-key product $\frac{QK^T}{\sqrt{d}}$, alternating row and column normalizations so that the attention matrix is doubly stochastic. In the combined train-and-test corruption setting, its relative-accuracy margin over the next-best mechanism is 3.0 percentage points on CIFAR-10, 1.9 on CIFAR-100, and 3.8 on Imagenette. Softmax attention often wins on absolute accuracy yet loses more of that performance under corruption, while Linear attention collapses to roughly 1% absolute accuracy in the corrupted-training settings. The paper interprets the row-and-column normalization constraints as the stabilizing factor that gives Doubly Stochastic attention its edge.

Load-bearing premise

The ranking assumes that independently tuning each attention mechanism's hyperparameters on clean data makes them fairly comparable, even though the tuned models differ in hidden width, MLP width, batch size, learning rate, head count, and gradient clipping.

Editorial extensions

If this is right

  • When training data are unreliable, Doubly Stochastic attention should be preferred over Softmax, Sigmoid, Cosine, and Linear attention in vision transformer image classification.
  • Linear attention should be avoided for noisy training regimes: it collapses to about 1% absolute accuracy on CIFAR-10 and CIFAR-100 when its training set is fog-corrupted.
  • Robustness comparisons should report relative accuracy alongside absolute accuracy, since the absolute leader (often Softmax) can be the one that degrades the most.
  • Attention-mechanism choice is a viable architectural robustness intervention, complementary to augmentation and training-time regularization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the study only uses fog, an immediate testable extension is whether the Doubly Stochastic advantage persists under Gaussian noise, blur, or other common corruptions; the paper itself flags this as open.
  • The reported margins in some settings are only 0.1–1.9 percentage points and every configuration uses a single seed, so multi-seed replication is needed to confirm the fine-grained ranking; the qualitative collapse of Linear attention is likely the most stable effect.
  • If the normalization is the causal mechanism, the same doubly stochastic constraint could be applied to other token-mixing architectures or to attention in language models, transferring the robustness benefit beyond image classification.
  • The corrupted-training results suggest the constraint acts during learning, not just at test time; a direct test would be to freeze a clean-trained Doubly Stochastic model and corrupt only at test time, where its margin is smaller.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper compares five self-attention mechanisms (Softmax, Sigmoid, Linear, Doubly Stochastic, Cosine) in Vision Transformers on CIFAR-10, CIFAR-100, and Imagenette under clean and fog-corrupted training/testing conditions. It reports absolute and relative accuracies, and claims that Doubly Stochastic attention is the most robust, particularly when both training and test data are corrupted, while Linear attention collapses under training corruption. The authors conclude that attention mechanism choice is a critical determinant of robustness and provide code.

Significance. If the central claim were established, the paper would provide useful guidance for architecture selection under distribution shift, and the relative-accuracy normalization is a sensible way to separate predictive strength from resilience. The paper also connects to prior theoretical work on doubly stochastic Gaussian kernel normalization, making the hypothesis scientifically motivated. However, the current experimental design does not yet support the causal claim that the attention normalizer drives the observed robustness ranking, and the evidence is weakened by a single seed and a post-hoc corruption choice.

major comments (5)
  1. The comparison does not isolate the attention mechanism because hyperparameters were independently tuned per mechanism and differ substantially in capacity and optimization. For example, on CIFAR-10 the Doubly Stochastic model uses 8 heads, batch size 256, and learning rate 4.5e-4, while the Linear model uses 12 heads, batch size 512, and learning rate 1e-3; on Imagenette the Sigmoid model has hidden dimension 768 and MLP hidden 3072, while the Softmax model has 384/384. Since only the number of layers, optimizer, and weight decay are held constant, the observed robustness ordering (e.g., Table 3's 99.9% vs 94.8% relative accuracy under train+test corruption) could be due to capacity, regularization, or optimization differences rather than row/column normalization. The central claim requires either matching all non-attention hyperparameters across mechanisms or demonstrating that the ranking is invariant to reasonable hyperparameter variation.
  2. The claim that Doubly Stochastic attention 'consistently maintains the highest relative accuracy' is contradicted by the CIFAR-100 test-only corruption row: Doubly Stochastic achieves 49.0% relative accuracy, behind Cosine (51.6%) and Softmax (51.5%). The abstract and Introduction bullet therefore overclaim the consistency of the result. The paper should either qualify the claim to the settings where it holds (train corruption, and train+test corruption) or provide an explanation for the exception.
  3. The absence of multiple seeds and error bars is a load-bearing issue, not merely a routine limitation. Several reported advantages are small: on CIFAR-10 train corruption, Doubly Stochastic's relative accuracy is 92.0% versus Softmax's 91.4%; on CIFAR-100 train corruption, it is 77.7% versus Cosine's 77.4%. With only one run per configuration, these differences are within plausible run-to-run variability. Without repeated runs or significance measures, the paper cannot support the strength of its conclusion about the ranking.
  4. The corruption type was selected post-hoc: the text states that initial Gaussian noise 'lacked significant performance variation' and that the study 'shifted to fog corruption', with fog severity tuned to produce noticeable validation drops. This selection bias, combined with the lack of a pre-registered protocol, means the reported ranking is conditioned on the corruption that happened to differentiate the mechanisms. The title and framing promise robustness to 'noise and spurious correlations', but only one corruption type (fog) is evaluated; no spurious-correlation experiment is performed. Either broaden the corruption suite or revise the claims to be explicitly about fog-like corruptions.
  5. The manuscript's title and conclusion invoke 'spurious correlations', but no experiment in the paper measures or manipulates spurious correlations. Fog is a common corruption, not a spurious correlation. This mismatch between the stated scope and the actual evidence should be corrected, for example by removing 'spurious correlations' from the title and claims or by adding a spurious-correlation benchmark.
minor comments (4)
  1. The reported margin of improvement is inconsistent: the abstract at the top of the full text says '0.1%−3.8%', while the separate abstract block says '0.1%−5.1%'. Please reconcile these numbers.
  2. The notation for the Sinkhorn iteration is unclear: the relationship between the matrix C in Eq. (7), the cost c_ij in Eq. (8), and the iterates K_l, u_l, v_l is not specified precisely. A reader cannot reconstruct the algorithm from the text alone.
  3. The linear attention formula uses ψ(Q)(KV) in the numerator and ψ(Q)K^T + ε in the denominator; the associativity and dimension of these terms should be clarified, as the current expression is ambiguous without knowing the intended order of operations.
  4. There is a typo in the running title 'Y our Attention Matters' (extra space), and figure captions 2–11 are largely repetitive; condensing them would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the robustness ranking is an empirical benchmark of measured accuracies, not a quantity derived from or fitted to its own inputs.

full rationale

The paper makes an empirical claim: Doubly Stochastic attention retains the highest relative accuracy under fog corruption across CIFAR-10, CIFAR-100, and Imagenette. There is no derivation chain in which an output equation reduces to an input equation or to a fitted constant. The relative accuracy numbers are computed directly from measured validation accuracies, and no attention mechanism is defined in terms of the robustness metric it is claimed to predict. The only theoretical reference to doubly stochastic normalization, Landa et al. [20], is used as motivation ('This leads one to wonder if this advantage holds in the context of Transformers'), not as a load-bearing uniqueness theorem, and it is not authored by the present paper's authors. The paper contains no self-citations, so no self-citation chain is invoked to justify the central claim. The hyperparameter tuning protocol independently tunes each mechanism, which raises a legitimate experimental-design concern about capacity and optimization confounds, but that is a missing control and an external-validity limitation, not circularity: the reported accuracies are not statistically forced by the tuned hyperparameters in the sense of a fitted parameter renamed as a prediction. The paper itself flags the fog-only corruption choice and the single-seed protocol as limitations in the conclusion, and those are candid scope restrictions rather than circular moves. In sum, the central ranking is an honest measurement against an external benchmark, with no step that reduces to its own input.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

This is an empirical study, so the ledger contains experimental design choices rather than mathematical postulates. The load-bearing choices are the per-mechanism hyperparameters (learning rates, capacities, Sinkhorn settings), the hand-selected fog severity, and the metric of relative accuracy. These choices, not the attention mechanism alone, determine the reported ranking.

free parameters (6)
  • Per-attention learning rate and batch size = varying by model, Tables 4-6
    Tuned on clean data for each mechanism; optimization differences (e.g., LR 5e-5 for DS-Imagenette vs 2e-4 for Cosine-Imagenette) confound the robustness comparison.
  • Per-attention architectural dimensions (heads, hidden, MLP) = varying, Tables 4-6
    DS-Imagenette uses hidden 768/MLP 3072 while Cosine-Imagenette uses hidden 768/MLP 384; capacity differences could drive the robustness ranking.
  • Sinkhorn max_iter and epsilon = C10: (20, 2), C100: (6, 1.0), Imagenette: (6, 0.2)
    Doubly stochastic attention-specific hyperparameters, stated in Tables 1-3; they affect convergence and numerical behavior of the attention output.
  • Sigmoid bias b = not reported
    Equation 4 introduces a bias term b; the chosen value is not listed in the paper, so exactly how Sigmoid attention was configured is ambiguous.
  • Fog severity = tuned to cause noticeable validation drops
    Section 2.1 reports the shift from Gaussian noise to fog after the former showed no effects; the corruption level is hand-selected, affecting the operational range of the claim.
  • Training epochs per mechanism = varies by mechanism
    Appendix C says total epochs vary across mechanisms while held constant across corruption settings for each mechanism; unequal training length affects clean baselines and relative accuracy.
assumptions (4)
  • domain assumption Relative accuracy is a fair measure of robustness across models with different clean baselines.
    Introduced in Section 2.2 as the primary robustness metric; if baseline normalization hides or creates differences, the ranking may be an artifact of the chosen metric.
  • domain assumption Independent hyperparameter tuning on clean data yields a fair comparison across attention mechanisms.
    Invoked in Section 2.1; the method produces unequal model capacities across mechanisms, which is a known confound when attributing robustness to the attention rule.
  • ad hoc to paper Fog corruption represents the noise and spurious correlations promised in the title.
    The title and abstract claim robustness to noise and spurious correlations, but only fog is tested; Gaussian noise was tried and abandoned, and no spurious correlation experiment is described.
  • ad hoc to paper A single random seed per configuration is sufficient to establish the robustness ranking.
    The Conclusion states this limitation but the paper uses the ranking as the central finding; a single seed means the margins (often under 1%) may be within run-to-run variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations." pith.science (2026). https://pith.science/paper/S45YUVT2

@misc{pith2026250720453,
  author       = {Pith},
  title        = {Pith review of: Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S45YUVT2}},
  note         = {Machine review of arXiv:2507.20453}
}
abstract

Self-attention mechanisms are foundational to Transformer architectures, supporting their impressive success in a wide range of tasks. While there are many self-attention variants, their robustness to noise and spurious correlations has not been well studied. This study evaluates Softmax, Sigmoid, Linear, Doubly Stochastic, and Cosine attention within Vision Transformers under different data corruption scenarios. Through testing across the CIFAR-10, CIFAR-100, and Imagenette datasets, we show that Doubly Stochastic attention is the most robust. It consistently outperformed the next best mechanism by $0.1\%-5.1\%$ when training data, or both training and testing data, were corrupted. Our findings inform self-attention selection in contexts with imperfect data. The code used is available at https://github.com/ctamayor/NeurIPS-Robustness-ViT.

Figures

Figures reproduced from arXiv: 2507.20453 by the authors.

Figure 1
Figure 1. Relative validation accuracy (as a percentage of clean baseline) for Cosine ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of absolute valida￾tion accuracy (%) of five attention mecha￾nisms—Softmax, Linear, Sigmoid, Cosine, and Doubly Stochastic—on the CIFAR-10 dataset un￾der four corruption settings. Doubly Stochastic attention achieves the highest absolute accuracy on CIFAR-10 when both training and test data are corrupted, while Softmax performs best on clean data and Linear attention collapses in the settings involving co… view at source ↗
Figure 5
Figure 5. Visualization of relative valida￾tion accuracy (%) of five attention mecha￾nisms—Softmax, Linear, Sigmoid, Cosine, and Doubly Stochastic—on the CIFAR-100 dataset under four corruption settings. Doubly Stochas￾tic attention demonstrates superior robustness to fog corruption on CIFAR-100 in settings involv￾ing corrupted training data, while Softmax is best on test-only corruption. See fig. 4 for an analysis of absolut… view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Analysis of the absolute validation accuracy (%) of five attention mechanisms (x￾axis) on the Imagenette dataset under four dis￾tinct corruption scenarios (y-axis). Softmax at￾tention achieves the highest absolute accuracy on clean data and train-only corruption while …
Figure 8
Figure 8. Figure 8: Validation accuracy learning curves for Vision Transformers using Cosine ( [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Absolute validation accuracy learning curves for Cosine ( [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Relative validation accuracy (as a percentage of clean baseline) for Cosine ( [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Relative validation accuracy learning curves for Cosine ( [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

  2. CLIMP: Contrastive Language-Image Mamba Pretraining

    cs.CV 2026-01 conditional novelty 6.0 of 10

    A fully Mamba-based (VMamba + Mamba LLM) CLIP model matches or beats transformer baselines on retrieval and OOD benchmarks, and natively supports high resolutions and dense captions.

Reference graph

Works this paper leans on

40 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Understanding robustness of transformers for image classification

    Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. InProceedings of the IEEE/CVF international conference on computer vision, pages 10231–10241, 2021

    work page 2021

  3. [3]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuropean conference on computer vision, pages 213–229. Springer, 2020

    2020

  4. [4]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020

    arXiv 2009

  5. [5]

    Autoaugment: Learning augmentation strategies from data

    Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113–123, 2019

    2019

  6. [6]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  7. [7]

    Improved regularization of convolutional neural networks with cutout.arXiv preprint arXiv:1708.04552, 2017

    Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout.arXiv preprint arXiv:1708.04552, 2017

    arXiv 2017

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

Show all 40 references
  1. [9]

    Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. InInternational conference on learning representations, 2018

    2018

  2. [10]

    Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778, 2021

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778, 2021

    2021 arXiv

  3. [11]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  5. [13]

    Benchmarking neural network robustness to common corruptions and perturbations.arXiv preprint arXiv:1903.12261, 2019

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations.arXiv preprint arXiv:1903.12261, 2019

    1903 arXiv

  6. [14]

    Imagenette: A smaller subset of 10 easily classified classes from imagenet

    Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from imagenet. https: //github.com/fastai/imagenette, October 2019

    2019

  7. [15]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. InInternational conference on machine learning, pages 448–456. pmlr, 2015

    2015

  8. [16]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  9. [17]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. InInternational conference on machine learning, pages 5583–5594. PMLR, 2021

    2021

  10. [18]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  11. [19]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. InAdvances in Neural Information Processing Systems, volume 25, pages 1097–1105, 2012. 9

    2012

  12. [20]

    Doubly stochastic normalization of the gaussian kernel is robust to heteroskedastic noise.SIAM journal on mathematics of data science, 3(1):388–413, 2021

    Boris Landa, Ronald R Coifman, and Yuval Kluger. Doubly stochastic normalization of the gaussian kernel is robust to heteroskedastic noise.SIAM journal on mathematics of data science, 3(1):388–413, 2021

    2021

  13. [21]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  15. [23]

    Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  16. [24]

    Cottention: Linear transformers with cosine attention

    Gabriel Mongaras, Trevor Dohm, and Eric C Larson. Cottention: Linear transformers with cosine attention. arXiv preprint arXiv:2409.18747, 2024

    2024

  17. [25]

    Rethinking self-attention: Towards interpretability in neural parsing.arXiv preprint arXiv:1911.03875, 2019

    Khalil Mrini, Franck Dernoncourt, Quan Tran, Trung Bui, Walter Chang, and Ndapa Nakashole. Rethinking self-attention: Towards interpretability in neural parsing.arXiv preprint arXiv:1911.03875, 2019

    1911 arXiv

  18. [26]

    Vision transformers are robust learners

    Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. InProceedings of the AAAI conference on Artificial Intelligence, volume 36, pages 2071–2081, 2022

    2022

  19. [27]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  20. [28]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  21. [29]

    Theory, analysis, and best practices for sigmoid self-attention.arXiv preprint arXiv:2409.04431, 2024

    Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention.arXiv preprint arXiv:2409.04431, 2024

    2024 arXiv

  22. [30]

    A generalist agent

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

    2022 arXiv

  23. [31]

    Sinkformers: Transformers with doubly stochastic attention

    Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. InInternational Conference on Artificial Intelligence and Statistics, pages 3515–3530. PMLR, 2022

    2022

  24. [32]

    Very deep convolutional networks for large-scale image recogni- tion.arXiv preprint arXiv:1409.1556, 2014

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion.arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  25. [33]

    Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1): 1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1): 1929–1958, 2014

    1929

  26. [34]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

    2015

  27. [35]

    Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.Advances in neural information processing systems, 30, 2017

    Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.Advances in neural information processing systems, 30, 2017

    2017

  28. [36]

    Going deeper with image transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 32–42, 2021

    2021

  29. [37]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  30. [38]

    Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  31. [39]

    Xlnet: Generalized autoregressive pretraining for language understanding.Advances in neural information processing systems, 32, 2019

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding.Advances in neural information processing systems, 32, 2019. 10

    2019

  32. [40]

    mixup: Beyond empirical risk minimization.arXiv preprint arXiv:1710.09412, 2017

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization.arXiv preprint arXiv:1710.09412, 2017. A Compute Resources All experiments were conducted on a high-performance computing (HPC) cluster. Each job was run on a compute ...

    2017 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.