Self-attention's local Lipschitz constant can be bounded using the attention probability distribution, and the softmax Jacobian spectral norm is shown to be at most 1/2, leading to a new robustness regularizer.
Transformers and large language models for chemistry and drug discovery
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
Self-attention's local Lipschitz constant can be bounded using the attention probability distribution, and the softmax Jacobian spectral norm is shown to be at most 1/2, leading to a new robustness regularizer.