REVIEW 10 cited by
The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The grokking phenomenon as reported by Power et al. ( arXiv:2201.02177 ) refers to a regime where a long period of overfitting is followed by a seemingly sudden transition to perfect generalization. In this paper, we attempt to reveal the underpinnings of Grokking via a series of empirical studies. Specifically, we uncover an optimization anomaly plaguing adaptive optimizers at extremely late stages of training, referred to as the Slingshot Mechanism. A prominent artifact of the Slingshot Mechanism can be measured by the cyclic phase transitions between stable and unstable training regimes, and can be easily monitored by the cyclic behavior of the norm of the last layers weights. We empirically observe that without explicit regularization, Grokking as reported in ( arXiv:2201.02177 ) almost exclusively happens at the onset of Slingshots, and is absent without it. While common and easily reproduced in more general settings, the Slingshot Mechanism does not follow from any known optimization theories that we are aware of, and can be easily overlooked without an in depth examination. Our work points to a surprising and useful inductive bias of adaptive gradient optimizers at late stages of training, calling for a revised theoretical analysis of their origin.
Forward citations
Cited by 10 Pith papers
-
Learning words in groups: fusion algebras, tensor ranks and grokking
Group word operations can be learned by small two-layer networks because the associated word tensor has low rank, decomposable through the fusion algebra of the group's self-conjugate representations.
-
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Muon-trained transformers grok modular addition and then lose generalization because the hidden representation and the output readout drift apart; freezing the readout and embeddings after grokking removes the collapse.
-
Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat
An optimizer that boosts weight decay when attention-logit variance spikes speeded grokking on a+b mod 97, but the effect is small, single-task, and not yet shown to come from the signal itself.
-
The Active Ingredient in Muon's Grokking
Orthogonalization—not spectral scaling—is the active ingredient in Muon's faster grokking, and reducing Newton-Schulz iterations trades first-crossing speed for solution stability.
-
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
In a fully tractable 12K Llama-style model, grokking is a conditional fragile phase transition gated by coverage (tracking modulus more than structure), weight decay, and floating-point reduction order, so evidence mu...
-
Beamforming Feedback as a Novel Attack Surface for Wi-Fi Physical-Layer Security
BFIAttack reconstructs legitimate CSI from public beamforming feedback, reportedly succeeding over 93% (single-antenna) and about 73% (multi-antenna) against Wi-Fi physical-layer security.
-
NeuralGrok: Accelerate Grokking by Neural Gradient Transformation
NeuralGrok trains a softmax MLP to transform gradients via bilevel optimization, accelerating grokking on arithmetic tasks and correlating with reduced gradient entropy.
-
Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model
GrokTransfer transfers an embedding learned by a small 'weaker' model to a larger model, eliminating the grokking delay so the target model generalizes almost immediately.
-
Grokking vs. Learning: Same Features, Different Encodings
Grokked and steadily trained models learn the same features, but steady training can produce much more compressible models in a parameter regime that grokking does not reach.
-
Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers
Adding synthetic knowledge-graph facts to raise the ratio of inferred to atomic facts produces a grokking-like jump in OOD accuracy on comparison-style 2WikiMultiHopQA questions, but not on composition questions.
Discussion (0). Continue with ORCID to comment.