REVIEW 6 cited by
Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent work by Power et al. (2022) highlighted a surprising "grokking" phenomenon in learning arithmetic tasks: a neural net first "memorizes" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy. This paper studies the grokking phenomenon in theoretical setups and shows that it can be induced by a dichotomy of early and late phase implicit biases. Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy.
Forward citations
Cited by 6 Pith papers
-
Learning words in groups: fusion algebras, tensor ranks and grokking
Group word operations can be learned by small two-layer networks because the associated word tensor has low rank, decomposable through the fusion algebra of the group's self-conjugate representations.
-
Decomposing Prediction Mechanisms for In-Context Recall
In a toy in-context recall task, label-based task initiation and observation-based continuation are distinct mechanisms with separate emergence times, and the same first-token versus second-token gap appears in an OLM...
-
Grokking Beyond the Euclidean Norm of Model Parameters
Grokking is induced by any small nonzero regularizer whose favored solutions generalize, with a delay that scales like one over the learning rate times the regularization strength.
-
Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks
Trained MLPs and transformers solving modular addition can be unified under an approximate Chinese Remainder Theorem, and deep or embedding-based networks learn only O(log n) frequency features.
-
Mechanistic Insights into Grokking from the Embedding Layer
Trainable embeddings in a simple MLP cause delayed generalization (grokking) on modular arithmetic, and a higher embedding learning rate plus balanced sampling accelerates it.
-
Feature learning is decoupled from generalization in high capacity neural networks
Current feature learning measures quantify the magnitude of representation change, which the authors argue is decoupled from the generalization benefit that neural networks show over their neural tangent kernel.
Discussion (0). Continue with ORCID to comment.