A two-stage softplus-based attention mechanism with re-weighting (LSSAR) is reported to keep validation loss nearly flat when a 124M-parameter GPT-2 is tested at up to 16x its 1024-token training length.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models
A two-stage softplus-based attention mechanism with re-weighting (LSSAR) is reported to keep validation loss nearly flat when a 124M-parameter GPT-2 is tested at up to 16x its 1024-token training length.