REVIEW 8 cited by
Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
read the original abstract
This paper studies emergent phenomena in neural networks by focusing on grokking where models suddenly generalize after delayed memorization. To understand this phase transition, we utilize higher-order mutual information to analyze the collective behavior (synergy) and shared properties (redundancy) between neurons during training. We identify distinct phases before grokking allowing us to anticipate when it occurs. We attribute grokking to an emergent phase transition caused by the synergistic interactions between neurons as a whole. We show that weight decay and weight initialization can enhance the emergent phase.
Forward citations
Cited by 8 Pith papers
-
Topological Signatures of Grokking
Persistent homology detects a sharp increase in maximum and total H1 persistence during grokking on modular arithmetic, offering a topological diagnostic that links representation geometry to generalization.
-
Emergent Generalization by Representation Learning in Artificial Neural Networks
An explicit low-dimensional bottleneck is necessary for OOD generalisation in reservoir networks, and the non-monotonic rise of causal emergence in the latent code predicts generalisation both in silico and in mouse CA1.
-
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Deep linear network theory derives logarithmic decay for cross-entropy loss under gap-growth conditions versus polynomial closure for Schatten-regularized structural energy under late-time KL tails, separating fitting...
-
A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learning
Epistemic uncertainty collapses sharply at grokking in in-context learning transformers, serving as a diagnostic of delayed generalization and linked via a Bayesian linear model to a shared spectral mechanism.
-
Scalar Representations of Neural Network Training Dynamics
Scalar embeddings of neural network training trajectories treated as temporal networks preserve main dynamical features including Lyapunov exponents, enable definition of a characteristic decorrelation time, and show ...
-
Emergence via Phase Transitions: Mechanism Landscapes and Universal Convergence Across Complex Systems
HEF frames emergence as a phase transition in mechanism landscapes with proofs of convergence under structural assumptions, empirically tested via grokking in transformers showing weight-norm peaks and accuracy collap...
-
Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
Grokking emerges near the model size where memorization timescale T_mem(P) intersects generalization timescale T_gen(P) on modular arithmetic.
-
Phase Transitions in Driven Informational Systems: A Two-Field Perspective on Learning Theory and Non-Equilibrium Chemistry
Proposes a two-gradient-field model with candidate order parameters alpha_dagger and kappa_c to unify phase transitions across learning theory and non-equilibrium chemistry.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.