Delta-rule linear attention faithfully approximates softmax attention through key-dependent rank-1 projections, enabling efficient post-hoc linearization of LLMs up to 32B parameters.
Eleutherai/lm-evaluation- harness: v0
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
The Key to Going Linear: Analysis-Driven Transformer Linearization
Delta-rule linear attention faithfully approximates softmax attention through key-dependent rank-1 projections, enabling efficient post-hoc linearization of LLMs up to 32B parameters.