On 64k-token natural language training, power attention with degree 2 achieves lower loss per FLOP than both softmax attention and existing linear attention, and its GPU kernels run faster than Flash Attention at long context.
Compute-optimal Context Size , b
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Scaling Context Requires Rethinking Attention
On 64k-token natural language training, power attention with degree 2 achieves lower loss per FLOP than both softmax attention and existing linear attention, and its GPU kernels run faster than Flash Attention at long context.