Exploring Grokking: Experimental and Mechanistic Investigations

Hu Qiye; Yu RuoXi; Zhou Hao

REVIEW 4 major objections 4 minor 15 references

Exploring Grokking: Experimental and Mechanistic Investigations

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that grokking in modular addition is conditional: the Transformer shows the delayed validation-accuracy jump at intermediate training-data fractions, while MLP and LSTM do not under the tested hyperparameters, and AdamW…

desk verdict A careful student replication of Power et al.'s grokking result, whose only new-looking claim (MLP/LSTM do not grok) is confounded by the fact that the Transformer got weight decay and the others did not. read the letter →

arxiv 2412.10898 v1 pith:SPBT7LNK submitted 2024-12-14 cs.LG stat.ML

Hu Qiye , Zhou Hao , Yu RuoXi This is my paper · ORCID

classification cs.LGstat.ML

keywords grokkingmodularadditiongeneralizationTransformerMLPLSTMAdamWweightdecay

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a broad set of experiments on when grokking appears in modular addition, the task where a network first memorizes its training inputs and later, after much more training, suddenly generalizes. The central finding is that this delayed jump in validation accuracy is strongest at intermediate training-data fractions: with modulus 97, fractions near 45–60% produce the classic long gap between perfect training accuracy and near-chance validation accuracy before generalization. The authors also find that, under the hyperparameter settings they chose, the Transformer exhibits the jump while the MLP and LSTM models do not, and that the AdamW form of weight decay gives the largest generalization improvement within a fixed budget. These results map the conditions that make grokking visible and show how sensitive the phenomenon is to architecture and optimizer choice. The paper is an experimental study that does not claim a new mechanism, and it explicitly notes that other results have produced grokking in MLPs with different initialization and weight-decay settings.

What carries the argument

The load-bearing object is the delayed generalization transition itself: the interval during which training loss is already near zero but validation accuracy remains at chance, ending in a sharp rise. The paper generates this object with the modular addition task, a training-data fraction $\alpha$ that controls how many of the $p^2$ pairs the model sees, and three model families. The Transformer is a two-layer decoder-only model with four attention heads and ReLU feed-forward blocks; the MLP and LSTM are matched to comparable size but trained with their own optimizers. The experimental argument compares the shape of the training and validation accuracy curves across $\alpha$, across architectures, and across optimizer variants, treating the gap between the two curves as the signature of grokking.

What would settle it

Train the same MLP on the same modular addition task with large initialization and small weight decay; if its validation accuracy shows the delayed sharp jump after training accuracy reaches 100%, the negative MLP result is a hyperparameter effect rather than an architectural one. This experiment would settle the paper's central model comparison.

Watch

Extended reading notes

Core claim

The central claim is that grokking is a real but configuration-dependent transition in the modular addition problem $(x,y)\mapsto (x+y)\bmod 97$ viewed as a classification task over the $97^2$ possible input pairs. In the Transformer, training accuracy reaches 100% within a few hundred steps while validation accuracy stays near zero for thousands of steps, then jumps sharply to perfect generalization; this behavior is most pronounced when roughly 30–60% of the input pairs are used for training. At very low fractions the model never generalizes, and at high fractions validation accuracy tracks training almost immediately. The same delayed jump is not seen in the MLP or LSTM under the hyperparameters tested, where training and validation accuracy rise together. In the optimizer comparison, AdamW with weight decay 1 produces the strongest improvement in validation accuracy within the fixed training budget, while full-batch Adam without weight decay still generalizes at higher data fractions, and poor learning-rate or minibatch choices sharply limit generalization.

Load-bearing premise

The comparison assumes that the different optimizer and initialization settings chosen for each model are the fair common ground, so the conclusion that MLPs and LSTMs do not grok is a conclusion about those settings rather than about the architectures themselves.

Editorial extensions

If this is right

For modular addition with $p=97$, the classic grokking gap appears near training-data fractions of 45–60%, with generalization arriving only after roughly 6,000 additional steps beyond perfect training accuracy.
Under the hyperparameters chosen in the paper, the Transformer, MLP, and LSTM separate into two behaviors: the Transformer shows the delayed validation jump, while MLP and LSTM show accuracy curves that rise together.
AdamW with weight decay 1 is the strongest single intervention for improving generalization within the training budget, but weight decay is not strictly required at higher data fractions, where full-batch Adam also generalizes.
The encoding of the arithmetic problem changes the difficulty: the more complex dictionary-style encoding from the original setup requires a larger training-data fraction before the Transformer can generalize than the direct classification encoding.
Grokking results are sensitive to optimizer configuration: suboptimal learning rates or minibatch choices can suppress the generalization transition within the fixed budget, so negative grokking results need to be tied to the exact training settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the absence of grokking in the MLP and LSTM is likely an artifact of the chosen optimization regime rather than an architectural fact, because the paper itself cites a setting with large initialization and small weight decay that produces grokking in MLPs.
Editorial inference: a direct test of that possibility would be to train the same MLP on the same modular addition task with large initialization and small weight decay; a delayed validation jump there would identify the optimizer configuration as the sufficient condition for grokking.
Editorial inference: the strong effect of weight decay suggests a follow-up sweep of weight-decay values at a fixed data fraction, which could reveal a sharp regularization threshold analogous to the sharp threshold in $\alpha$.
Editorial inference: if grokking is primarily an optimization-trajectory effect, then the structured-representation and implicit-bias explanations are not competing but complementary, and progress measures built on representation geometry should be tested in the large-initialization regime.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this reads like a solid undergrad project, not a research paper. The authors reproduce the data-fraction dependence for grokking on modular addition with a Transformer, and they run a wide-ish optimization sweep. That part is fine and honestly reported. What is not fine is the model comparison. The Transformer is trained with AdamW and weight decay 1; the MLP and LSTM are trained with plain Adam and no weight decay. The paper itself concedes in the conclusion that the absence of grokking in MLP/LSTM "might be attributed to not employing weight decay techniques." That concession is fatal to the Section 4.2 claim that these models do not exhibit grokking in this context. The stress-test note is right: the setup predicts exactly what they observed, and the cited Omnigrok/Lyu work gives a regime (large init, small weight decay) they never tested. So the negative result is a statement about hyperparameters, not architectures.

What is good: the exploration of data fractions at least qualitatively tracks Power et al.; the optimizer comparison in Fig 5 (even without seeds) shows the expected large benefit of weight decay; the mechanism discussion is a fair, brief summary of Liu et al. and Lyu et al. The writing is clear and the acknowledgments of their own limitations are candid. But the experimental gaps are real: no error bars, no multiple seeds, some internal inconsistencies (they say the MLP uses parameters identical to the Transformer's feed-forward network, but the Transformer's FFN is a one-layer MLP of width 512 while the MLP is described as 2 layers with hidden 512), and the claim of reproducing the 'alpha around 50%' result is weakened by their scan skipping 50% (45% and 60% shown). These are the kind of gaps you'd expect in a course report.

I would not send this to a serious referee. It reproduces known results, and the only attempted novel contribution is methodologically confounded. The right venue for this, if any, is a course archive or a blog post. It does show the authors engaged seriously with the literature and with the replication, so as a pedagogical artifact it has some value.

Referee Report

4 major / 4 minor

Summary. The paper reports experiments on grokking for the modular addition task with p=97, varying the training data fraction, the model architecture (Transformer, MLP, LSTM), and the optimizer/regularization scheme. It claims that the Transformer exhibits grokking most clearly at intermediate data fractions around 50%, that MLP and LSTM models do not exhibit grokking in this context, and that AdamW weight decay improves generalization the most among the interventions tested. It also summarizes two existing mechanistic accounts of grokking: structured representations in a 'Goldilocks zone' and implicit biases from large initialization and small weight decay.

Significance. If the empirical claims were fully supported, the paper would provide a useful replication and extension of Power et al.'s grokking experiments and would highlight interactions among architecture, data fraction, and optimizer. The paper deserves credit for reproducing the original OpenAI setup, providing a code link for the MLP experiments, and explicitly acknowledging in its conclusion that the absence of grokking in MLP/LSTM may be due to the lack of weight decay. However, the central comparative claim about architecture is currently under-supported because the models are compared under different optimizers and initializations, and no seed statistics are reported. The mechanism section is a literature review rather than a new mechanistic investigation, so the paper's value is primarily as a conditional empirical observation.

major comments (4)

[Section 4.2 / Section 6] The architecture comparison is confounded by optimizer and initialization. The Transformer is trained with AdamW and weight decay 1 (Section 3), while the MLP and LSTM are trained with plain Adam and no weight decay. The paper itself cites [6] and [8] showing that large initialization with small weight decay can induce grokking in MLPs, and Section 6 concedes that the absence 'might be attributed to not employing weight decay techniques.' Figures 3 and 4 therefore do not establish that MLP and LSTM fail to grok as a property of architecture; they establish a property of the combined hyperparameter configuration. Please rerun the MLP and LSTM in the large-initialization/small-weight-decay regime, or explicitly limit the claim to the tested configuration and remove the implication that architecture is the determining factor.
[Section 4.1, Figures 1-2, Figure 7] The data-fraction scan omits the claimed 50% point. The text states that grokking is 'most pronounced when alpha is around 50%,' but the experiments use α = 5%, 30%, 45%, 60%, 75%, and 90%; α = 50% is never run. The observed gap at 45% and 60% is consistent with a peak near 50%, but it does not directly measure that point. Either add the α = 50% experiment or rephrase the claim as an interpolation from neighboring fractions.
[Section 4.1, Figure 7] No error bars, confidence intervals, or multiple-seed trials are reported for any accuracy curve or for the steps-until-generalization values in Figure 7. Grokking transition times are known to be sensitive to seed and initialization, and a single trajectory is insufficient to support quantitative statements such as 'it takes around 6000 epoch for generalization' (Section 4.1). Please report statistics over at least a few seeds, or explicitly label these as single illustrative runs and avoid quantitative claims based on a single trajectory.
[Section 4.3, Figure 5] The claim that 'weight decay, i.e. AdamW, improves generalization the most' is not supported by a defined comparison. Figure 5 presents eight panels with different optimizers, learning rates, and batch sizes, but the text gives no metric (e.g., final validation accuracy, area under the curve, or time to a threshold) by which AdamW is judged best, and all panels appear to be single runs. In addition, the caption labels panel (a) as '3x baseline LR' while the text bullet list starts with 3e-4, which is 0.3x baseline; please align the caption and text and specify the comparison protocol.

minor comments (4)

[Section 2.1] The definition of α would be clearer if it explicitly stated that the total data size is p^2 for the modular addition task, since the text currently refers to p^2 only indirectly.
[Section 3 / Section 4.2] The LSTM is described as having 'hidden size of 20' in Section 3 but 'number of hidden layers to 20' in Section 4.2; please make the terminology and notation consistent.
[Section 4.3] There is a typo: 'updats' should be 'updates'.
[Section 5] The mechanism section is a literature review rather than a new mechanistic investigation; this is acceptable as background, but the framing should state this explicitly so readers do not expect a novel mechanism.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all central claims are direct empirical observations; the MLP/LSTM comparison has a potential confound but it is explicitly acknowledged and is a validity issue, not a circular derivation.

full rationale

The paper contains no derivation that could be circular. Its central claims are empirical observations: (i) Transformer grokking appears for intermediate training-data fractions (Section 4.1), (ii) under the paper's chosen setups MLP and LSTM do not show a delayed sharp validation jump (Section 4.2), and (iii) AdamW with weight decay 1 gives the largest generalization improvement within a fixed budget (Section 4.3). None of these is obtained by fitting a parameter to a target quantity and then predicting that same quantity; no self-citation supplies an unverified premise. The Section 4.2 model comparison is not a control across optimizer and initialization: the Transformer uses AdamW and full batch, while MLP and LSTM use plain Adam (Section 3). This is a potential confound or external-validity limitation, not circularity, and the paper itself flags it in Section 6: 'we understand this might be attributed to not employing weight decay techniques.' The statements are also explicitly conditional ('the MLP model does not exhibit the grokking phenomenon in this context'). The mechanism section is a literature review of prior work, not an argument that reduces to its own conclusion. Therefore no circular step is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new mathematical entities or fitted parameters; the free parameters listed are experimental design choices on which the qualitative conclusions depend. The central claims are empirical, not derived, so the axiom ledger mainly records domain and methodological assumptions.

free parameters (5)

Training data fractions alpha = {0.05, 0.15, 0.30, 0.45, 0.60, 0.75, 0.90}
Chosen values for the data-fraction scan; the 'around 50 percent' conclusion is inferred from 45 and 60 percent rather than measured at 50 percent.
Transformer hidden width d_model = 128
Architecture size chosen for the simplified Transformer; the grokking curves depend on this capacity.
Weight decay coefficient = 1
AdamW weight decay 1 is the key regularizer that the optimizer comparison concludes improves generalization the most.
LSTM hidden size = 20
Chosen so that training and validation accuracy reach a relatively high level after 100,000 epochs; the no-grokking conclusion depends on this capacity.
Optimization budget = 1800 gradient updates for optimizer comparison; up to 100,000 epochs for LSTM
Fixed budgets determine whether delayed generalization appears; the paper does not test longer horizons for MLP or LSTM.

assumptions (5)

standard math Cross-entropy loss (Eq. 1) is the correct objective and training uses standard backpropagation.
The reported accuracies are evaluated on this loss; no alternative loss is considered.
domain assumption Results on modular addition with p=97 are representative of grokking behavior generally.
The central claims about data fraction, architecture, and optimizer are drawn from a single prime modulus; the paper does not vary p.
domain assumption Validation accuracy on randomly held-out pairs measures generalization without leakage.
The encoding methods and data splitting are assumed to prevent the model from seeing the answer indirectly; the paper does not analyze possible leakage.
domain assumption Full-batch training produces smoother dynamics and reduces slingshots.
Section 3 states this as a rationale for the training choice, but the paper does not verify the slingshot connection.
domain assumption The reviewed mechanism papers (Liu et al., Lyu et al., Nanda et al., Varma et al.) are correct in their conclusions.
Section 5 is a literature summary; the paper contributes no independent verification of these mechanisms.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring Grokking: Experimental and Mechanistic Investigations." pith.science (2026). https://pith.science/paper/SPBT7LNK

@misc{pith2026241210898,
  author       = {Pith},
  title        = {Pith review of: Exploring Grokking: Experimental and Mechanistic Investigations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPBT7LNK}},
  note         = {Machine review of arXiv:2412.10898}
}

read the original abstract

The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero training error and near-random test error. Subsequent prolonged training leads to a sharp transition from no generalization to perfect generalization. Our study comprises extensive experiments and an exploration of the research behind the mechanism of grokking. Through experiments, we gained insights into its behavior concerning the training data fraction, the model, and the optimization. The mechanism of grokking has been a subject of various viewpoints proposed by researchers, and we introduce some of these perspectives.

Figures

Figures reproduced from arXiv: 2412.10898 by the authors.

**Figure 2.** Comparison of accuracy at different training data fractions. The blue lines for training and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Comparison of accuracy at different training data fractions with MLP model. The blue lines [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Comparison of accuracy at different training data fractions with LSTM model. The blue [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Different optimization algorithms lead to different amount of generalization within an [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Comparison of loss at different training data fractions. The blue lines for training and the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Steps until generalization at various training data fractions. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Comparison of loss at different training data fractions with MLP model. The blue lines for [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages

[6]

Omnigrok: Grokking beyond algorithmic data

Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117, 2022

arXiv 2022
[8]

Dichotomy of early and late phase implicit biases can provably induce grokking

Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S Du, Jason D Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking. arXiv preprint arXiv:2311.18817, 2023

arXiv 2023
[1]

Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang

Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit, 2023

work page 2023
[2]

Simplicity bias in trans- formers and their ability to learn sparse boolean functions, 2023

Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in trans- formers and their ability to learn sparse boolean functions, 2023

work page 2023
[3]

Can transformers learn the greatest common divisor?, 2023

François Charton. Can transformers learn the greatest common divisor?, 2023. 7

work page 2023
[4]

A toy model of universality: Reverse engineering how networks learn group operations, 2023

Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations, 2023

2023
[5]

Michaud, Max Tegmark, and Mike Williams

Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, 2022

work page 2022
[7]

Fixing weight decay regularization in adam

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017

arXiv 2017

Show all 15 references

[9]

Progress measures for grokking via mechanistic interpretability, 2023

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023

2023
[10]

Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens

Arvind Neelakantan, Luke Vilnis, Quoc V . Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks, 2015

2015
[11]

Grokking: Gen- eralization beyond overfitting on small algorithmic datasets

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

2022 arXiv
[12]

Mech- anism of feature learning in deep fully connected networks and kernel machines that recursively learn features, 2023

Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mech- anism of feature learning in deep fully connected networks and kernel machines that recursively learn features, 2023

2023
[13]

The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon, 2022

Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon, 2022

2022
[14]

Explaining grokking through circuit efficiency, 2023

Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency, 2023

2023
[15]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Appendix Losses and Steps Until Generalization We provide additi...

2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [6]

Omnigrok: Grokking beyond algorithmic data

Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117, 2022

arXiv 2022

[2] [8]

Dichotomy of early and late phase implicit biases can provably induce grokking

Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S Du, Jason D Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking. arXiv preprint arXiv:2311.18817, 2023

arXiv 2023

[3] [1]

Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang

Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit, 2023

work page 2023

[4] [2]

Simplicity bias in trans- formers and their ability to learn sparse boolean functions, 2023

Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in trans- formers and their ability to learn sparse boolean functions, 2023

work page 2023

[5] [3]

Can transformers learn the greatest common divisor?, 2023

François Charton. Can transformers learn the greatest common divisor?, 2023. 7

work page 2023

[6] [4]

A toy model of universality: Reverse engineering how networks learn group operations, 2023

Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations, 2023

2023

[7] [5]

Michaud, Max Tegmark, and Mike Williams

Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, 2022

work page 2022

[8] [7]

Fixing weight decay regularization in adam

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017

arXiv 2017