A spectral-norm ratio rule on the optimizer shrinks per-layer learning rates when weight updates are too large, letting Transformers train without warmup.
Attention is not all you need: Pure attention loses rank doubly exponentially with depth
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Taming Transformer Without Using Learning Rate Warmup
A spectral-norm ratio rule on the optimizer shrinks per-layer learning rates when weight updates are too large, letting Transformers train without warmup.