Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The local Lipschitz constant of transformer self-attention is governed by the shape of the attention distribution, and the softmax Jacobian's spectral norm never exceeds 1/2.

desk verdict A promising new Lipschitz bound and regularizer for attention, but the key theorem's proof has a load-bearing gap that must be repaired. read the letter →

arxiv 2507.07814 v1 pith:KGZ7HE3O submitted 2025-07-10 cs.LG cs.NAmath.NA

Nikolay Yudin , Alexander Gaponov , Sergei Kudriashov , Maxim Rakhuba This is my paper
classification cs.LGcs.NAmath.NA MSC 68T0715A4215A18
keywords localLipschitzboundself-attentionsoftmaxJacobianspectralnormattentiondistributionadversarialrobustnessregularizationtransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to prove a sharper, distribution-aware local Lipschitz bound for dot-product self-attention. Its central technical result is an interlacing bound on the singular values of the softmax Jacobian $M(x)=\mathrm{diag}(x)-xx^{\top}$, giving a universal bound $\|J_{\mathrm{sm}}(z)\|_2 \leq g_1(\mathrm{sm}(z)) \leq 1/2$, a factor of two tighter than the commonly used 1-Lipschitz assumption. Plugging this into the exact Jacobian of a self-attention head yields $\|J_{\mathrm{Attn}}(X)\|_2 \leq \|W^V\|_2 (\|P^h\|_2 + 2\|X\|_2^2 \|A_h\|_2 \max_i \|M(P^h_{i,:})\|_2)$, making the dependence on attention distributions explicit. The authors argue this explains why near-uniform and near-categorical attention maps are locally smoother, and they introduce JaSMin (Jacobian Softmax norm Minimization), a lightweight regularizer that penalizes this term and improves adversarial robustness. The payoff, if the results hold, is a direct and cheap way to read robustness off the attention probability vectors.

What carries the argument

The central object is $M(x)=\mathrm{diag}(x)-xx^{\top}$, the Jacobian of the softmax map, written as $M(P^h_{i,:})$ for a row of the attention map. The carrying mechanism is a rank-one interlacing argument: $M(x)$ is a symmetric rank-one perturbation of $\mathrm{diag}(x)$, and the paper extends the classical interlacing theorem with an eigenvector sign-pattern argument to obtain the chain $x_{(1)} \geq g_1(x) \geq \sigma_1(A) \geq x_{(2)} \geq \cdots$, hence $\|M(x)\|_2 \leq g_1(x) \leq 1/2$. This softmax spectral bound is then inserted into the exact block form of the attention Jacobian, isolating the term $\max_i \|M(P^h_{i,:})\|_2$ that JaSMin is designed to minimize.

What would settle it

Sample probability vectors densely over the simplex in $\mathbb{R}^3$ and $\mathbb{R}^4$, compute the exact largest singular value of $\mathrm{diag}(x)-xx^{\top}$, and compare it with $g_1(x)$; a single vector with $\sigma_1>g_1(x)$ or with $\sigma_1>1/2$ would refute Corollary 1. A more targeted check is to diagonalize the matrix at each sample and inspect the eigenvector for the largest eigenvalue: if its coordinate signs do not follow the sorted pattern claimed in Theorem 4 at a point where the interlacing inequality also fails, the proof's pivot step is what breaks.

Watch

Extended reading notes

Core claim

On the paper's own terms: the local Lipschitz constant of a self-attention head is controlled not just by the spectral norms of the weight matrices but by the shape of the attention distribution, through the largest singular value of each softmax Jacobian block. The paper proves that for any probability vector $x$, the singular values of $\mathrm{diag}(x)-xx^{\top}$ interlace with the ordinal statistics $x_{(k)}(1-x_{(k)}+x_{(k+1)})$, so the largest singular value is at most $g_1(x) \leq 1/2$, with equality attained at $x=(1/2,1/2,0,\ldots,0)$. This feeds into Theorem 3, a head-level Jacobian bound that is strictly sharper than the Specformer bound and, in a slightly modified form, improves the smooth-attention bound by a constant factor. The same analysis shows that attention maps close to uniform and close to categorical both make the local self-attention map nearly constant, while intermediate distributions are the sensitive ones. On the practical side, the paper introduces JaSMin, which penalizes $g_1$ or the ratio $g_1/g_k$, and reports that ViT-B models trained with it show lower measured local Lipschitz constants and higher accuracy under FGSM, PGD, and AutoAttack than the baseline or Specformer.

Load-bearing premise

The load-bearing premise is the proof's assumption in Theorem 4 that each nonzero eigenvector of $\mathrm{diag}(x)-xx^{\top}$ has the specified alternating sign pattern over sorted coordinates with a nonzero coordinate to divide by, so that the interlacing chain leading to the $1/2$ bound goes through.

Editorial extensions

If this is right

  • Attention rows that are nearly uniform or nearly one-hot are locally smoother, while intermediate mixtures are the most sensitive to input perturbations.
  • The softmax Jacobian norm bound improves from the common 1-Lipschitz assumption to $1/2$, and the new attention-level bound is strictly tighter than the Specformer bound.
  • Minimizing $g_1$ pushes attention to be either more uniform or more categorical, whereas minimizing $g_1/g_k$ forces uniformity over at least the top $k$ tokens, giving a dial to trade off robustness against oversmoothing.
  • ViT-B trained with JaSMin shows reduced local Lipschitz constants across layers and higher accuracy under FGSM, PGD, and AutoAttack compared with baseline and Specformer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same softmax spectral bound could be used as a cheap, attack-free robustness monitor for any attention-based model: tracking $\max_i g_1(P^h_{i,:})$ during training would reveal when heads enter the sensitive intermediate regime.
  • The two-regime picture also suggests a reading of attention sinks: near-categorical rows have small local Lipschitz constants but concentrate gradients, so the paper's framework predicts they stabilize forward sensitivity while straining training stability, a trade-off that could be tested by correlating sink heads with measured Jacobian norms.
  • One testable extension would be to combine JaSMin with spectral regularization on $W^V$, $W^Q$, and $W^K$; since the bound factors into these weight norms and the attention-shape term, the product could be minimized more aggressively than either penalty alone.
  • The $1/2$ bound on the softmax Jacobian is a general smoothness fact that likely transfers to other softmax-based modules, such as mixture-of-experts routers, where the same categorical-versus-uniform trade-off should reappear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies local Lipschitz constants of dot-product self-attention. Theorem 3 gives a bound on the Jacobian norm of a self-attention head in terms of the attention matrix P and the per-row softmax Jacobians M(P_i,:). Theorem 4 states an interlacing chain for the singular values of M(x) = diag(x) - xx^T in terms of ordinal statistics g_k(x) = x_(k)(1 - x_(k) + x_(k+1)), yielding Corollary 1 that the softmax Jacobian spectral norm is at most g_1(sm(z)) <= 1/2. Based on this bound, the authors introduce a regularizer called JaSMin that penalizes g_1(P_i,:) or the ratio g_1/g_k, and report experiments on ViT-B with CIFAR-10/100 showing improved robustness under FGSM, PGD, and AutoAttack relative to Specformer, together with reduced empirical Jacobian norms.

Significance. If Theorem 4 is correct, the paper offers a genuinely distribution-aware local Lipschitz bound for self-attention and a sharp uniform half-upper-bound on the softmax Jacobian, which would improve on the commonly used 1-Lipschitz softmax bound. The empirical evidence that JaSMin increases robustness and lowers measured spectral norms is encouraging, and the authors report extensive hyperparameter sweeps and a power-iteration check of the whole-model Jacobian. However, the central theoretical lemma has a proof gap in a degenerate case, and the asymptotic comparison with the bound of Castin et al. requires a modified version of Theorem 3; these issues are fixable but need to be addressed before the main claims are fully established.

major comments (3)
  1. [Section 3, proof of Theorem 4 (around Eq. (6))] The proof of Theorem 4 is incomplete in a degenerate case. The authors derive alpha_k = x_k(1 - <x,y^(k)> / y^(k)_k) and then divide by y^(k)_k, and they also assert that y^(k) has its first k coordinates nonnegative and the remaining coordinates nonpositive. Neither property is established when x - alpha_k e has zero entries, i.e. when alpha_k coincides with one or more components of x (as happens with tied probabilities). In that situation the vector (x - alpha_k e) ⊙ y^(k) = <x,y^(k)> x does not force a definite sign pattern for y^(k), and an eigenvector in the eigenspace can be supported only on the tied coordinates, potentially with y^(k)_k = 0. The paper does not rule out this configuration, and the interlacing chain x_(1) >= g_1 >= sigma_1 >= x_(2) >= ... together with Corollary 1 and the JaSMin regularizer rely on it. The proof should either justify the existence of an eigenvector with y^(k)_k != 0 and the required sign pattern (for example, by a continuity or perturbation argument) or provide an alternative proof of the chain.
  2. [Section 3 and Appendix C] The comparison with the bound of Castin, Ablin, and Peyre is overstated in the main text. As written, Theorem 3 contains the factor ||X||_2^2, which is at most N R^2 on the ball B^N_R(0), so the unmodified bound grows linearly in the sequence length N, whereas Theorem 2 of [4] is O(sqrt(N)). Appendix C explicitly acknowledges this and obtains an improved asymptotic rate only after replacing the estimate of X^T M(P_i,:) with a modified block-norm argument, yielding ||W^V||_2 (sqrt(N) + 2 sqrt(N) R^2 ||A_h||_2). The abstract's claim of being 'more accurate than prior art' and the main-text sentence 'our estimate is strictly sharper than in Theorem 1' are therefore not accurate as a blanket statement. The authors should rephrase the comparison so that the claimed improvement over [4] refers either to the modified bound or to the empirical tightness observed on their models, not to Theorem 3 as stated.
  3. [Section 3, statement of Theorem 4] The statement of Theorem 4 uses g_n(x), but Definition 1 defines g_k only for k = 1,...,n-1. If the intended convention is x_(n+1) = 0, it should be stated explicitly. In addition, the theorem states a chain for 'singular values' sigma_i but proves assertions about eigenvalues alpha_i of the symmetric matrix A; since A is positive semidefinite this identification is valid, but it should be stated to avoid a gap in the reader's verification.
minor comments (5)
  1. [Section 3, Corollary 1] Corollary 1 refers to 'Definition 5' but the object g_1 is defined in Definition 1; the cross-reference should be corrected.
  2. [Table 3] In the row 'Specformer(0,0,1e-4)' for CIFAR-100, the PGD2 entry reads '2797', which appears to be a typo for '27.97' or a similar value; please check.
  3. [Section 4, Eq. (8) and (9)] The regularizers use a maximum over heads and layers and a logarithm of g_1, which is not differentiable at points where the top-two order statistics are tied or where the argmax of the maximum changes; a brief statement about subgradients or a smoothing is needed for reproducibility.
  4. [Appendix E, Figure 6] The text in Appendix E says that the checkpoint used for Figure 6 was trained with JaSMin, but the figure caption does not say this. Please state in the caption that all bounds in Figure 6 are evaluated on the JaSMin-trained model, so readers do not interpret the panel as a model-independent comparison of bound tightness.
  5. [Section 5] The power-iteration procedure used for the empirical spectral norms is not described (number of iterations, convergence tolerance, or initialization); a short description would increase reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Lipschitz bounds are derived from a known Jacobian formula and an independent interlacing argument, and the JaSMin experiments are verified against power-iteration spectral norms and external attack metrics; the flagged Theorem 4 proof gap is a rigor issue, not a circularity.

full rationale

The central derivation chain is self-contained. Theorem 3 bounds the self-attention Jacobian by decomposing the known Jacobian formula (Eq. (12)) from prior work and applying submultiplicativity; the result is an upper bound with no fitted constants. Theorem 4 is a separate spectral interlacing argument for the softmax Jacobian M(P) = diag(P) - PP^T, and Corollary 1 substitutes its g1 bound into the Theorem 3 expression. None of these steps defines the predicted quantity in terms of itself, and no parameter inside the bounds is fitted to the data being predicted. The JaSMin regularizer does minimize the same g1 upper bound that the paper analyzes, but the empirical claims are not evaluated solely by that bound: the paper reports power-iteration estimates of the exact spectral norm of the model Jacobian (Figure 4 and the “Exact” panel of Figure 6) and external robustness metrics (FGSM, PGD, AutoAttack). The ratio g1/||M||2 in Figure 3 is greater than 1 by construction, and the text explicitly says this is “as anticipated,” so it is not presented as an independent prediction. References to the Jacobian formula, the interlacing theorem, and baseline bounds are external or standard results, not an author self-citation chain. The only notable issue is in the proof of Theorem 4 around Eq. (6): the authors divide by the k-th coordinate of the eigenvector y^(k) and assert a sign pattern without handling the degenerate case y^(k)_k = 0 or proving that a sign-pattern eigenvector with nonzero k-th coordinate always exists. This is a potential gap in rigor for the proof as written, but it is a correctness risk rather than a circularity, because the claimed bound is still a substantive mathematical statement whose derivation does not reduce to its own assumptions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The theoretical bound introduces no fitted constants; the only hand-chosen parameters are regularization hyperparameters lambda and k, plus a small epsilon for numerical stability. The proof imports the self-attention Jacobian formula and the classical interlacing theorem, and Theorem 4 adds a nontrivial refinement. No new physical or model entities are postulated.

free parameters (3)
  • JaSMin regularization coefficient lambda = 1e-2, 1e-3, 1e-4 in hyperparameter sweeps
    Chosen by hand across a small grid; selecting the best robustness value from the same tables used for the comparison can inflate reported improvements.
  • JaSMin top-k parameter k = 0, 10, 30
    Trade-off parameter for how many top attention probabilities are forced toward uniformity; selected by hyperparameter sweep.
  • Numerical epsilon in JaSMin denominator = 1e-6
    Added to avoid division by zero near categorical attention distributions; appears in Appendix D and is an ad hoc implementation choice.
assumptions (4)
  • standard math Interlacing theorem for symmetric rank-1 corrections (Theorem 5 in the paper)
    Used to relate eigenvalues of diag(x) to eigenvalues of diag(x)-xx^T in Theorem 4; taken from [15,26] without proof.
  • domain assumption Exact block-form Jacobian of dot-product self-attention (Eq. (12) in Appendix A, from Kim et al. [18])
    The proof of Theorem 3 starts from this closed form; the paper does not re-derive it, so a mistake there would invalidate the bound.
  • standard math The matrix diag(x)-xx^T is positive semidefinite for x on the simplex, so singular values coincide with eigenvalues
    Implied in Theorem 4's use of interlacing on eigenvalues; follows from E[Y^2]-(E[Y])^2 >= 0 for a random variable Y taking values x_i with probabilities x_i.
  • domain assumption Row-wise softmax applied to the score matrix X A X^T defines the attention map P as in Eq. (2)
    All theoretical results concern this standard dot-product self-attention formulation; other attention variants are out of scope.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers." pith.science (2026). https://pith.science/paper/KGZ7HE3O

@misc{pith2026250707814,
  author       = {Pith},
  title        = {Pith review of: Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGZ7HE3O}},
  note         = {Machine review of arXiv:2507.07814}
}
read the original abstract

We present a novel local Lipschitz bound for self-attention blocks of transformers. This bound is based on a refined closed-form expression for the spectral norm of the softmax function. The resulting bound is not only more accurate than in the prior art, but also unveils the dependence of the Lipschitz constant on attention score maps. Based on the new findings, we suggest an explanation of the way distributions inside the attention map affect the robustness from the Lipschitz constant perspective. We also introduce a new lightweight regularization term called JaSMin (Jacobian Softmax norm Minimization), which boosts the transformer's robustness and decreases local Lipschitz constants of the whole network.

Figures

Figures reproduced from arXiv: 2507.07814 by the authors.

Figure 1
Figure 1. g1(x) from Def. 5 within its domain of definition (x(1) ≥ x(2)) and (x(1) + x(2) ≤ 1). The maximum g1 = 1/2 is attained at (1/2, 1/2) and the mini￾mums with g1 = 0 are at the corners (1, 0), (0, 0). Thus, in the context of Theorem 3, we obtain an upper bound: ∥M(P h i,:)∥2 ⩽ g1(P h i,:), (7) which can be efficiently calculated and differentiated. What is more, we can identify regions, where this norm is small. As we… view at source ↗
Figure 2
Figure 2. Attention probabilities for ViT-L1 model for certain heads. The title of each bound describes true average value of ∥M(P h i,:)∥2 together with its upper bound g1(P h i,:). Each attention map is averaged across 1000 ImageNet samples. The black square denotes an absolute position of the patch for which attention maps are printed. 1.00 1.05 1.10 Layer 3, Head 4 0 50 100 150 200 1.00 1.02 1.04 Layer 4, Head 5 0 100 200… view at source ↗
Figure 3
Figure 3. Histograms of g1(P h i,:)/∥M(P h i,:)∥2 – upper bound, divided by the exact value. As anticipated, all values are greater than 1, while remaining close to 1. Heads, query pixels and samples are kept as in the caption of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of distributions of the spectral norm of the whole model’s Jacobian (trained on CIFAR100). The spectral norm is computed via power it￾eration at each point in the evaluation dataset. For comparison we use the original ViT training setup with Specformer and J…
Figure 5
Figure 5. Figure 5: Comparison of regularization effects for ViT-B models trained on CIFAR100. The name of [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparison of bounds for the local Lipschitz constant of a single self-attention head. All [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Mechanistic Analysis of Looped Reasoning Language Models

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Looped LLMs converge to distinct cyclic fixed points per layer, repeating feedforward-style inference stages across recurrences.

  2. Principles of Lipschitz continuity in neural networks

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A thesis deriving an SDE for how a network's spectral-norm Lipschitz bound changes under SGD, proving a non-negative noise-driven drift term, plus closed-form singular-value Hessians and a Shapley-based spectral robus...

  3. Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    SCD replaces raw KV cache transmission with compact semantic codes via reuse and patching to achieve up to 2.65x TTFT speedup while staying within 5% F1 of oracle quality.

  4. Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussion

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    A data-parameter correspondence unifies data-centric and parameter-centric LLM optimizations as dual geometric operations on the statistical manifold via Fisher-Rao metric and Legendre duality.

Reference graph

Works this paper leans on

35 extracted references · 14 canonical work pages · cited by 4 Pith papers

  1. [4]

    How smooth is attention? arXiv preprint arXiv:2312.14820, 2023

    Valérie Castin, Pierre Ablin, and Gabriel Peyré. How smooth is attention? arXiv preprint arXiv:2312.14820, 2023

    arXiv 2023

  2. [1]

    Transformers and large language models for chemistry and drug discovery

    Andres M Bran and Philippe Schwaller. Transformers and large language models for chemistry and drug discovery. In Drug Development Supported by Informatics, pages 143–163. Springer, 2024. 9

    work page 2024

  3. [2]

    Neobert: A next-generation bert

    Lola Le Breton, Quentin Fournier, Mariam El Mezouar, and Sarath Chandar. Neobert: A next-generation bert. arXiv preprint arXiv:2502.19587, 2025

    arXiv 2025

  4. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [5]

    Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

    Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020

    2020

  6. [6]

    Lipschitz normalization for self-attention layers with application to graph neural networks

    George Dasoulas, Kevin Scaman, and Aladin Virmaux. Lipschitz normalization for self-attention layers with application to graph neural networks. In International Conference on Machine Learning, pages 2456–2466. PMLR, 2021

    work page 2021

  7. [7]

    How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?

    Xinhsuai Dong, Luu Anh Tuan, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre-trained language models be fine-tuned towards adversarial robustness? arXiv preprint arXiv: 2112.11668, 2021

    work page Pith review arXiv 2021

  8. [8]

    Attention is not all you need: pure ttention loses rank doubly exponentially with depth

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure ttention loses rank doubly exponentially with depth. In International conference on machine learning, pages 2793–2803. PMLR, 2021

    work page 2021

Show all 35 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  2. [10]

    EDIT: Enhancing vision transformers by mitigating attention sink through an encoder-decoder architecture

    Wenfeng Feng and Guoying Sun. EDIT: Enhancing vision transformers by mitigating attention sink through an encoder-decoder architecture. arXiv preprint arXiv: 2504.06738, 2025

    2025

  3. [11]

    On the properties of the softmax function with application in game theory and reinforcement learning

    Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805, 2017

    2017 arXiv

  4. [12]

    Regularisation of neural networks by enforcing lipschitz continuity

    Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110:393–416, 2021

    2021

  5. [13]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781, 2024

    2024 arXiv

  6. [14]

    Transformer training instability of softmax and lipschitz- kernel attentions, 2024

    JongHyun Hong and Sungyoon Lee. Transformer training instability of softmax and lipschitz- kernel attentions, 2024. URL https://openreview.net/forum?id=q541p2YLt2

    2024

  7. [15]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985

    1985

  8. [16]

    Specformer: Guarding vision transformer robustness via maximum singular value penalization

    Xixu Hu, Runkai Zheng, Jindong Wang, Cheuk Hang Leung, Qi Wu, and Xing Xie. Specformer: Guarding vision transformer robustness via maximum singular value penalization. In European Conference on Computer Vision, pages 345–362. Springer, 2024

    2024

  9. [17]

    MInference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. MInference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024

    2024 arXiv

  10. [18]

    The lipschitz constant of self-attention

    Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562–5571. PMLR, 2021

    2021

  11. [19]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  12. [20]

    On controllable sparse alternatives to softmax

    Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh Khapra, Karthik Sankaranarayanan, and Harish G Ramaswamy. On controllable sparse alternatives to softmax. Advances in neural information processing systems, 31, 2018

    2018

  13. [21]

    SGDR: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  14. [22]

    Understanding zero-shot adversarial robustness for large-scale models

    Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl V ondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022

    2022 arXiv

  15. [23]

    Lipsformer: Introducing lipschitz continuity to vision transformers

    Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, and Lei Zhang. Lipsformer: Introducing lipschitz continuity to vision transformers. arXiv preprint arXiv:2304.09856, 2023

    2023 arXiv

  16. [24]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  17. [25]

    Revisiting over-smoothing in bert from the perspective of graph

    Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.08625, 2022

    2022 arXiv

  18. [26]

    Springer Science & Business Media, 1997

    Evgeni˘ı Evgen‘evich Tyrtyshnikov.A brief introduction to numerical analysis. Springer Science & Business Media, 1997

    1997

  19. [27]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  20. [28]

    Adversarial demonstration attacks on large language models

    Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao. Adversarial demonstration attacks on large language models. arXiv preprint arXiv:2305.14950, 2023

    2023 arXiv

  21. [29]

    On the role of attention masks and layernorm in transformers

    Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. arXiv preprint arXiv:2405.18781, 2024

    2024 arXiv

  22. [30]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  23. [31]

    Learning physical simulation with message passing transformer

    Zeyi Xu and Yifei Li. Learning physical simulation with message passing transformer. arXiv preprint arXiv: 2406.06060, 2024

    2024 arXiv

  24. [32]

    Differential transformer

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05258, 2024

    2024 arXiv

  25. [33]

    CutMix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019

    2019

  26. [34]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017

    2017 arXiv

  27. [35]

    Exact” are averaged across the CIFAR-100 validation set. “Exact

    Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International conference on machine learning, pages 27378–27394. PMLR, 2022. 11 A Proof of Theorem 3 Proof of theore...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.