Muon-trained transformers grok modular addition and then lose generalization because the hidden representation and the output readout drift apart; freezing the readout and embeddings after grokking removes the collapse.
Muon: An optimizer for hidden layers in neural networks, 2024.https://github.com/ KellerJordan/Muon
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.AI 1years
2026 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Muon-trained transformers grok modular addition and then lose generalization because the hidden representation and the output readout drift apart; freezing the readout and embeddings after grokking removes the collapse.