Freezing query and key attention weights still lets transformers form induction heads and stay close to standard performance on language modeling, while random static attention (MixiT) fails on in-context tasks but succeeds on memorization and algorithmic tasks.
Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang.On exact computation with an infinitely wide neural net
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
Freezing query and key attention weights still lets transformers form induction heads and stay close to standard performance on language modeling, while random static attention (MixiT) fails on in-context tasks but succeeds on memorization and algorithmic tasks.