Lorsa replaces an MHSA layer with thousands of sparsely activated rank-1 attention heads and shows these heads recover known behaviors like induction heads plus new arithmetic and thematic units.
Circuits updates - march 2024
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
Lorsa replaces an MHSA layer with thousands of sparsely activated rank-1 attention heads and shows these heads recover known behaviors like induction heads plus new arithmetic and thematic units.