The paper reproduces the grokking phenomenon in a Transformer on modular arithmetic, finds no grokking in MLPs and LSTMs under the tested settings, and attributes the difference to missing weight decay.
Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Exploring Grokking: Experimental and Mechanistic Investigations
The paper reproduces the grokking phenomenon in a Transformer on modular arithmetic, finds no grokking in MLPs and LSTMs under the tested settings, and attributes the difference to missing weight decay.