NeuralGrok: Accelerate Grokking by Neural Gradient Transformation

Jie Fu; Martin Jaggi; Simin Fan; Xinyu Zhou

REVIEW 5 major objections 5 minor 1 cited by

NeuralGrok: Accelerate Grokking by Neural Gradient Transformation

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NeuralGrok learns to reshape gradients and reaches 95% test accuracy on five modular arithmetic tasks faster than standard training or Grokfast.

desk verdict A genuinely new learned-gradient-transformation approach to accelerating grokking, honestly reported, but the speedup needs seed-level replication and a cleaner isolation of the learned component. read the letter →

arxiv 2504.17243 v2 pith:HMTFFKXL submitted 2025-04-24 cs.LG cs.AI

Xinyu Zhou , Simin Fan , Martin Jaggi , Jie Fu This is my paper · ORCID

classification cs.LGcs.AI

keywords grokkinggradienttransformationbileveloptimizationmodulararithmeticneural-amplifiergeneralizationabsoluteentropytransformertraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes NeuralGrok, a method that trains a small auxiliary MLP — the neural-amplifier — to transform each gradient before it updates a transformer trained on modular arithmetic. The amplifier weights gradient entries with a softmax and rescales the gradient to constant norm, and it is tuned by a bilevel loop on a tiny validation slice of the training set. The paper shows that this accelerates grokking on five arithmetic tasks, with up to 4.67× fewer steps than Grokfast-MA on the hardest task, and that test accuracy stays stable after generalization, unlike standard training or weight-decay baselines. It also introduces the Absolute Gradient Entropy (AGE) metric, reporting that transformed gradients stay low-entropy while original gradients spike, linking the speedup to reduced model complexity.

What carries the argument

The neural-amplifier $G(\varphi)$ is an MLP that maps each gradient entry to a softmax weight and then rescales the whole gradient to constant norm $c$: $g' = c \cdot p \cdot g / \|p \cdot g\|_2$. It is trained by bilevel optimization: the inner loop updates the base transformer with transformed gradients, and the outer loop (every $T$ steps) updates the amplifier to minimize loss on $D_{\text{outer}}$, using a one-step lookahead through a copy of the base model. The transformation is a learned rotation plus fixed rescaling, which the paper interprets as suppressing task-specific noise in cyclic modular arithmetic. The diagnostic is Absolute Gradient Entropy, $H(G) = -\sum_i |g_i| \ln |g_i|$, which rises during the memorization phase and falls during generalization; the amplifier keeps transformed gradients at lower AGE than original gradients.

What would settle it

Train the same bilevel loop but replace $D_{\text{outer}}$ with a corrupted or shuffled version (e.g., random labels or only even inputs). If NeuralGrok still reaches 95% test accuracy at the same speed, the outer-loop signal is not driving generalization-oriented gradient shaping; alternatively, if the speedup vanishes under a different 2% slice of the same training set, the result hinges on the specific slice rather than the method.

Watch

Extended reading notes

Core claim

The central claim is that grokking's long memorization phase can be shortened by learning a task-specific gradient transformation rather than relying on hand-designed filters or regularization. In the inner loop, gradients $g$ from a training subset pass through the neural-amplifier, which produces a softmax distribution $p$ over gradient entries and forms $g' = c \cdot p \cdot g / \|p \cdot g\|_2$; the base transformer is updated with $g'$. In the outer loop, the amplifier is updated to minimize loss on $D_{\text{outer}}$, a 2% slice of the training data, evaluated through a one-step copy of the base model. On five modular arithmetic datasets — $(a+b) \bmod 97$, $(a-b) \bmod 97$, $(a\cdot b) \bmod 97$, $(a^2-b) \bmod 97$, and $(ac+bd-e) \bmod 7$ — this reaches 95% test accuracy in fewer steps than standard training and Grokfast-MA (e.g., 1896 vs 8853 steps on the hardest task), and test accuracy does not collapse after generalization. The paper also reports that standard gradient normalization alone, without the amplifier, stabilizes training better than weight decay on these tasks.

Load-bearing premise

The amplifier is trained on a 2% slice of the training set, and the method assumes that minimizing loss on that tiny, same-distribution slice after a single transformed gradient step is a reliable proxy for held-out test generalization.

Editorial extensions

If this is right

If correct, the neural-amplifier gives a drop-in gradient preprocessing step that shortens grokking on modular arithmetic transformers without changing the base architecture.
The bilevel formulation implies that a small validation slice can drive fast generalization, suggesting gradient shaping can substitute for careful weight-decay tuning.
Gradient normalization alone is reported to stabilize training and enable generalization on a task where standard training fails, so gradient magnitude control may be the operative regularizer in these settings.
The AGE metric provides a per-step complexity signal that rises in memorization and falls in generalization, offering a candidate progress measure for grokking.
Because learned transformations transfer poorly across even similar arithmetic tasks, the speedup is task-specific, which bounds direct reuse but points toward meta-learning shared transformations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A stress test of the outer-loop proxy would be to permute, shrink, or corrupt $D_{\text{outer}}$: if the speedup survives a corrupted validation signal, the amplifier is not actually tracking generalization.
The rotation-plus-rescaling view suggests a testable hypothesis: the amplifier learns to de-emphasize gradient entries tied to answer tokens or high-frequency components; inspecting learned $p$ values per parameter group would reveal the pattern.
If the AGE drop is causal rather than correlational, directly penalizing gradient entropy during training should reproduce part of the speedup; the paper does not run that experiment.
The low cross-task transfer implies practical value depends on per-task meta-tuning, so a natural extension is a shared amplifier trained across tasks with a meta-objective.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this paper does something new—it trains a small MLP to transform gradients via a softmax weighting plus norm rescaling in a bilevel loop, explicitly to shorten grokking—and it reports speedups on five modular arithmetic tasks. The central claim is plausible, but the evidence lacks error bars over seeds, and the learned softmax is not isolated from plain gradient normalization as cleanly as the title implies.

What is actually new: the closest prior work, Grokfast, uses a fixed low-pass filter on gradients. Here the transformation is learned per task, which is a meaningful difference. The outer-loop optimization against a small validation subset is a reasonable application of learned-optimizer ideas to grokking. The paper also documents that gradient normalization alone stabilizes training on these tasks better than weight decay, and that the combination is best on the hard task—useful empirical information.

The paper does some things well: the ablation on the rescaling coefficient c is thorough, and the robustness of NeuralGrok to c is shown. The limitations section is candid: the authors explicitly note the narrow Douter validation objectives and the poor cross-task transfer, and they include transfer experiments rather than hiding them. The AGE metric is a natural extension of AWE to gradients; it tracks phase transitions and could be a handy monitoring tool, though it is descriptive, not causal.

Soft spots, in order of severity: first, no seeds or error bars anywhere. Grokking step counts are high-variance; a single run claiming 1896 vs 8853 could be within run-to-run spread. Second, the outer loop uses Douter of only 94–168 examples, and the paper never varies the Dinner:Douter split or replaces the outer-loop signal with a fixed or random transformation to show the learned component alone is responsible. The comparison to standard training with gradient normalization (Figure 3) does suggest the softmax adds speed beyond normalization, since NeuralGrok reaches 900 steps where plain norm takes 3450, but it's not the same training pipeline, so the isolation is imperfect. Third, the interpretability story rests on qualitative entropy dynamics, not a mechanism.

Who this is for: researchers working on grokking or small-transformer generalization will get genuine value from the method and the honest failure modes. Outside that subfield, the significance is modest.

Recommendation: this deserves peer review, not desk rejection. The revision should add multiple seeds, a fixed-transformation ablation, and a split-ratio sensitivity check. If those hold up, I'd cite it.

Referee Report

5 major / 5 minor

Summary. The paper proposes NeuralGrok, a bilevel optimization method that trains an auxiliary MLP (the neural-amplifier) to transform the gradients of a transformer base model during grokking. The transformation is a softmax-weighted elementwise scaling of each gradient entry followed by renormalization to a constant norm c. The method is evaluated on five modular arithmetic tasks: (a+b) mod 97, (a-b) mod 97, (a*b) mod 97, (a*a-b) mod 97, and (a*c+b*d-e) mod 7. The central claim is that NeuralGrok accelerates grokking, reaching 95% test accuracy in fewer optimization steps than standard training and Grokfast-MA, with the hardest task showing 1896 steps versus 8853 for Grokfast-MA. The paper also introduces an Absolute Gradient Entropy (AGE) metric and argues that NeuralGrok stabilizes training and reduces model complexity, in contrast to weight decay, which can cause instability.

Significance. If the central claim holds, the paper makes a useful empirical contribution: it extends the Grokfast line of work by showing that a learned, gradient-modulating module can shorten the memorization phase on arithmetic tasks, and it documents that simple gradient normalization enables generalization on a task where standard training fails. The authors provide code and compare against external baselines on a disjoint test set, which are strengths. However, the current evidence is preliminary: all headline numbers come from single runs, the learned softmax component is not ablated against fixed transformations under the same protocol, and the outer-loop validation signal uses a tiny 2% subset of the training set without robustness checks. The AGE analysis is also qualitative and limited to one task. With added seeds, ablations, and split-ratio analysis, the empirical claims would be substantially strengthened.

major comments (5)

[Section 3.1, Table 1] All results in Table 1 and Figure 1 come from a single run per setting. Grokking step-to-generalization is known to vary substantially across seeds, including in the original Grokfast ablations, so the reported speedups (e.g., 1896 vs 8853 on (ac+bd-e) mod 7) may reflect run-to-run variance rather than a reliable method effect. The authors should report results over at least 5-10 seeds with medians and spreads (e.g., IQR or min-max) for each method and task.
[Section 3.2, Algorithm 2] The experiments do not isolate the learned softmax component p from the fixed rescaling/normalization. The only ablation in Section 3.2 varies the rescaling coefficient c, and Figure 3 shows that standard training with plain gradient normalization already generalizes on the hard task, reaching 95% at roughly 29770 steps. To support the claim that the neural-amplifier 'learns an optimal gradient transformation,' the authors should compare NeuralGrok, under the same Dinner/Douter split and outer-loop budget, against (i) a fixed uniform p with the same norm rescaling, (ii) a fixed random p, and (iii) an amplifier trained with the same bilevel pipeline but with shuffled labels in Douter. Without such controls, the acceleration cannot be attributed to the learned modulation rather than to the normalization itself.
[Section 3 (data split), Section 5, Appendix E] The outer-loop signal uses Douter, only 2% of Dtrain: roughly 94 examples for the mod-97 tasks and about 168 examples for the five-argument task. The paper neither varies the Dinner:Douter ratio nor tests the reliability of the one-step lookahead proxy on Douter, and it does not check seed sensitivity of the split. The authors themselves note in Section 5 and Appendix E that the 'narrow validation objectives (Douter)' may force the amplifier into local task geometries, so the central mechanism—that minimizing loss on this tiny subset teaches a generalizable gradient transformation—remains unvalidated. Please add split-ratio ablations (e.g., 98:2, 95:5, 90:10) and a control in which Douter labels are shuffled, to test whether the outer-loop signal is necessary for the speedup.
[Section 3.1, Table 1] The reported 'minimal optimization steps' count only base-model inner-loop steps on Dinner; the additional amplifier updates in the outer loop and the extra hyperparameter tuning are not accounted for. Also, NeuralGrok trains the base model on 98% of Dtrain while baselines use 100% (with the 2% set aside as Douter), so the comparison mixes data-seen and compute-cost differences. The authors should report total wall-clock time or total optimizer steps including outer-loop updates, and state explicitly that base-model steps are the reported metric. This will clarify how much of the apparent acceleration is due to the transformation versus the comparison protocol.
[Section 4, Figures 5-6] The AGE analysis is qualitative: transition windows in Figure 5 are marked by hand, no quantitative definition or automated detector is given, and all entropy curves come from a single seed on one task (a+b mod 97). The statement that AGE 'consistently correlates' with phase transitions is therefore not supported by measurement. At minimum, define the window boundaries, show multiple seeds and tasks, and compare AGE against the existing AWE and weight-norm baselines to justify the claimed explanatory role.

minor comments (5)

[Section 2, Eq. (2)] Calling the elementwise multiplication p·g a 'rotation' is inaccurate, since p has nonnegative entries; it is a direction-changing positive rescaling, not a rotation. Please rephrase.
[Appendix D, title] The Appendix D heading says '(axc+bxd-e) mod 97' while the text and Figure 11 are about (ac+bd-e) mod 7; please fix the inconsistency.
[Table 1, caption] Please define the meaning of '-' in Table 1; presumably it indicates failure to reach 95% within the step budget, but this should be stated explicitly.
[Section 3.1, Section 5] There are several typos and wording issues: 'arguements' should be 'arguments', 'nuance reasoning mechanisms' should be 'nuanced reasoning mechanisms', and 'Godilocks zone' should be 'Goldilocks zone'. Please proofread.
[Algorithm 2] In Algorithm 2, the copy M'(θ) is updated to θ' = θ - ηθ g'_θ, and the meta-gradient is written as ∇φ L(θ', Douter) = ∇φ L(θ - ηθ G(φ, gθ), Douter). This equality is only correct if the dependence of θ' on φ is made explicit; please write θ'(φ) to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central speedup is measured against a held-out test set and external baselines, and the bilevel Douter fit does not reduce to the reported test-set predictions.

full rationale

The paper's central claim is an empirical acceleration result: NeuralGrok reaches 95% test accuracy in fewer optimization steps than standard training and Grokfast-MA on five arithmetic tasks. The neural-amplifier G(phi) is optimized only on Douter, a 2% subset of Dtrain (Section 3: 'we further divide Dtrain into Dinner and Douter with a ratio of 49 : 1'), while the reported step-to-95% numbers and accuracy curves are evaluated on a disjoint held-out test set ('All methods are tested on the same test set Dtest'). The outer-loop objective in Algorithm 2 minimizes loss on Douter after one transformed gradient step, and no test-set information enters the amplifier's training. Equation (2) is a fixed functional form (softmax-weighted gradient normalization with constant c), not a renaming of the output; the learned part is trained only on training-subset validation signal. The comparison baselines are external (standard training and Lee et al.'s Grokfast), and the paper includes ablations of the rescaling coefficient c, showing that the learned amplifier adds acceleration beyond plain normalization. The self-citations (e.g., Fan et al. 2024) are background references and are not load-bearing for the main result. The authors' own limitation about 'narrow validation objectives (Douter)' and the absence of seed-based error bars are concerns about statistical robustness and external validity, not circularity. The AGE-based interpretation is post-hoc and correlational, but even if overinterpreted, it does not make the empirical speedup equal to its inputs by construction. No step in the derivation chain reduces a prediction to a fitted parameter or to a self-citation chain.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The method is an empirical algorithm, so the main ledger entries are hand-chosen hyperparameters (c, T, amplifier size, data split) and domain assumptions about the validation proxy and the entropy metrics. No new physical entity is introduced; the conceptual entity is the AGE measure, which is defined and used only within this paper.

free parameters (6)

gradient rescaling coefficient c = 1.0 (default; ablated 0.01-2.0)
Chosen by hand; rescales the transformed gradient to a constant norm in Eq. 2. Default 1.0; ablation shows robustness but some sensitivity at c=0.01 and 2.0.
inner-loop frequency T = 4 for two-argument tasks, 1 for the five-argument task
Chosen per task; controls how often the neural-amplifier is updated. Not justified beyond 'fast adaptation' for the hard task.
amplifier hidden dimension = 32
Set in Appendix B; no sensitivity study.
amplifier layers and alpha = 2 layers, alpha=16 in code
Architecture choices in Appendix B code; alpha is not mentioned in the main text and is not ablated.
weight decay for baselines = 1e-3
Chosen because 1e-2 prevents learning in standard training and Grokfast-MA; the paper uses this value for all methods.
data split ratio Dinner:Douter = 49:1
Douter is 2% of Dtrain; chosen to keep enough training data while providing a validation signal. No sensitivity analysis.

assumptions (4)

domain assumption Optimizing the amplifier on a small subset Douter of the training set (2% of Dtrain) is a valid proxy for improving held-out test generalization.
Central to the bilevel objective; no proof given. If Douter is not representative, the amplifier could overfit to memorization. Section 2 and Section 3 data split.
ad hoc to paper The gradient transformation g' = c * p * g / ||p*g||_2 preserves enough information to allow the base model to learn generalizable features.
The softmax-over-entries map is proposed ad hoc; no theorem guarantees it improves generalization.
ad hoc to paper Absolute Gradient Entropy (AGE) is a meaningful measure of model complexity that indicates phase transitions in grokking.
Introduced in Eq. 4; the paper only shows qualitative visual correlation, no quantitative test or external benchmark.
domain assumption A single SGD step on a copy of the base model in the outer loop approximates the effect of T inner-loop steps on the true model.
The Learn-Amplifier function uses a one-step copy update to differentiate through; the approximation error is not analyzed. Section 2, Algorithm 2.

invented entities (1)

Absolute Gradient Entropy (AGE)
purpose: Measure instantaneous model complexity from gradient vectors to explain grokking phase transitions.
Introduced in Eq. 4; the paper shows qualitative correlation with phase transitions but no external benchmark or falsifiable prediction tied to AGE beyond the same training runs used to define it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NeuralGrok: Accelerate Grokking by Neural Gradient Transformation." pith.science (2026). https://pith.science/paper/HMTFFKXL

@misc{pith2026250417243,
  author       = {Pith},
  title        = {Pith review of: NeuralGrok: Accelerate Grokking by Neural Gradient Transformation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMTFFKXL}},
  note         = {Machine review of arXiv:2504.17243}
}

read the original abstract

Grokking is proposed and widely studied as an intricate phenomenon in which generalization is achieved after a long-lasting period of overfitting. In this work, we propose NeuralGrok, a novel gradient-based approach that learns an optimal gradient transformation to accelerate the generalization of transformers in arithmetic tasks. Specifically, NeuralGrok trains an auxiliary module (e.g., an MLP block) in conjunction with the base model. This module dynamically modulates the influence of individual gradient components based on their contribution to generalization, guided by a bilevel optimization algorithm. Our extensive experiments demonstrate that NeuralGrok significantly accelerates generalization, particularly in challenging arithmetic tasks. We also show that NeuralGrok promotes a more stable training paradigm, constantly reducing the model's complexity, while traditional regularization methods, such as weight decay, can introduce substantial instability and impede generalization. We further investigate the intrinsic model complexity leveraging a novel Absolute Gradient Entropy (AGE) metric, which explains that NeuralGrok effectively facilitates generalization by reducing the model complexity. We offer valuable insights on the grokking phenomenon of Transformer models, which encourages a deeper understanding of the fundamental principles governing generalization ability.

Figures

Figures reproduced from arXiv: 2504.17243 by the authors.

**Figure 1.** Train and Test accuracies on arithmetic tasks. NEURALGROK consistently accelerates generalization under the grokking phenomenon, especially on the challenging task. Setup. We construct five arithmetic tasks with various difficulty levels, including four tasks between two arguments: (a+b) mod 97, (a-b) mod 97, (a×b) mod 97, (a×a-b) mod 97, and one challenging task with five arguments: (a×c+b×d-e) mod 7. For the … view at source ↗

**Figure 2.** Standard training with various gradient rescaling coefficient c on (a+b) mod 97 task. With c=0.5, 1.0, 2.0, the training is effectively stabilized compared to Figure 1a with unchanged magnitude. magnitude. It indicates that gradient rescaling can be a more effective regularization than conventionally used weight-decay on arithmetic task learning. 10 1 10 2 10 3 10 4 Steps 0.5 1.0 3450 (a) (a+b) mod 97. 10 1 10 2 10 … view at source ↗

**Figure 3.** Standard training with standard gradient normalization c=1.0. The gradient normalization enables the generalization on the challenging task (a×c+b×d-e) mod 7, which is failed in Figure 1a, when gradient normalization is not applied. NEURALGROK is robust with various rescaling coefficients. While gradient rescaling acts as a crucial factor when applying standard training on transformer models, NEURALGROK exhibits a r… view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: NEURALGROK with various gradient rescaling coefficient c on the (a+b) mod 97 task. The model can achieve a perfect test accuracy in similar speed (1.3ksteps) with c ranging from 0.2 to 1.0. While applying a larger gradient magnitude c=2.0 could lead to a delayed genera…

**Figure 5.** Figure 5: Model complexity measured in entropy on task (a+b) mod 97. The transition windows for Memorization and Generalization phases are marked in red and green. Absolute Gradient Entropy as an effective indicator of phase transitions. As shown in [PITH_FULL_IMAGE:figures/ful…

**Figure 6.** Figure 6: Absolute Gradient Entropy before and after gradient transformation by the neural-amplifier. Is Weight-decay always a good regularization? While previous studies claim that weight-decay is the crucial factor to enable generalization under the grokking phenomenon, we…

**Figure 7.** Figure 7: Code for NEURALGROK C Supplement Results on Baselines C.1 Standard Training with Various weight-decay We try two different values (i.e., 1e −2 , 1e −3 ) of weight-decay to observe the learning pattern of the model in the standard training. However, we find that when we…

**Figure 8.** Figure 8: Standard training with weight-decay 1e −2 . The model fails to memorize and generalize in all four experiments. C.2 GROKFAST-MA with Various weight-decay We also compare different values (i.e., 1e −2 , 1e −3 ) of weight-decay influence on GROKFASTMA. The results are s…

**Figure 9.** Figure 9: GROKFAST-MA training using different weight-decay values. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: GROKFAST-EMA results on all tasks. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: Standard training on ac + bd − e (mod 7) with various combination of regularization techniques. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: Transfer learning experiments from other tasks to [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Structure-Specific Representational Priors Causally Control the Grokking Delay
cs.LG 2026-07 conditional novelty 6.5 of 10

The grokking delay is causally the time to form the right feature-level representational structure, not a fixed optimization constant or a pure weight-norm effect.

Reference graph

Works this paper leans on

24 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

Deep learning using rectified linear units (relu), 2019

Abien Fred Agarap. Deep learning using rectified linear units (relu), 2019. URL https://arxiv.org/abs/1803.08375

arXiv 2019
[2]

Unifying grokking and double descent, 2023

Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent, 2023. URL https://arxiv.org/abs/2303.06173

arXiv 2023
[3]

The complexity dynamics of grokking, 2024

Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The complexity dynamics of grokking, 2024. URL https://arxiv.org/abs/2412.09810

arXiv 2024
[4]

Deep grokking: Would deep neural networks generalize better?, 2024

Simin Fan, Razvan Pascanu, and Martin Jaggi. Deep grokking: Would deep neural networks generalize better?, 2024. URL https://arxiv.org/abs/2405.19454

arXiv 2024
[5]

Progress measures for grokking on real-world tasks, 2024

Satvik Golechha. Progress measures for grokking on real-world tasks, 2024. URL https://arxiv.org/abs/2405.12755

arXiv 2024
[6]

Train faster, generalize better: Stability of stochastic gradient descent, 2016

Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent, 2016. URL https://arxiv.org/abs/1509.01240

arXiv 2016
[7]

Deep networks always grok and here is why, 2024

Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why, 2024. URL https://arxiv.org/abs/2402.15555

arXiv 2024
[8]

Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training

Rie Johnson and Tong Zhang. Inconsistency, instability, and generalization gap of deep neural network training, 2023. URL https://arxiv.org/abs/2306.00169

work page Pith review arXiv 2023

Show all 24 references

[9]

A simple weight decay can improve generalization

Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991

1991
[10]

Gershman, and Cengiz Pehlevan

Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics, 2024. URL https://arxiv.org/abs/2310.06110

2024 arXiv
[11]

Grokfast: Accelerated grokking by amplifying slow gradients, 2024

Jaerin Lee, Bong Gyun Kang, Kihoon Kim, and Kyoung Mu Lee. Grokfast: Accelerated grokking by amplifying slow gradients, 2024. URL https://arxiv.org/abs/2405.20233

2024 arXiv
[12]

On generalization error bounds of noisy gradient methods for non-convex learning, 2020

Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning, 2020. URL https://arxiv.org/abs/1902.00621

2020 arXiv
[13]

Michaud, Max Tegmark, and Mike Williams

Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, 2022. URL https://arxiv.org/abs/2205.10343

2022 arXiv
[14]

Michaud, and Max Tegmark

Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data, 2023. URL https://arxiv.org/abs/2210.01117

2023 arXiv
[15]

Do machine learning models memorize or generalize?, 2023

Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg, and Lucas Dixon. Do machine learning models memorize or generalize?, 2023. URL https://pair.withgoogle.com/explorables/grokking/

2023
[16]

Grokking: Generalization beyond overfitting on small algorithmic datasets

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

2022 arXiv
[17]

Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the edge of numerical stability, 2025. URL https://arxiv.org/abs/2501.04697

2025 arXiv
[18]

The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon, 2022

Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon, 2022. URL https://arxiv.org/abs/2206.04817

2022 arXiv
[19]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

2023 arXiv
[20]

On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective, 2024

Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective, 2024. URL https://arxiv.org/abs/2011.11152

2024 arXiv
[21]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[22]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[23]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[24]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Deep learning using rectified linear units (relu), 2019

Abien Fred Agarap. Deep learning using rectified linear units (relu), 2019. URL https://arxiv.org/abs/1803.08375

arXiv 2019

[2] [2]

Unifying grokking and double descent, 2023

Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent, 2023. URL https://arxiv.org/abs/2303.06173

arXiv 2023

[3] [3]

The complexity dynamics of grokking, 2024

Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The complexity dynamics of grokking, 2024. URL https://arxiv.org/abs/2412.09810

arXiv 2024

[4] [4]

Deep grokking: Would deep neural networks generalize better?, 2024

Simin Fan, Razvan Pascanu, and Martin Jaggi. Deep grokking: Would deep neural networks generalize better?, 2024. URL https://arxiv.org/abs/2405.19454

arXiv 2024

[5] [5]

Progress measures for grokking on real-world tasks, 2024

Satvik Golechha. Progress measures for grokking on real-world tasks, 2024. URL https://arxiv.org/abs/2405.12755

arXiv 2024

[6] [6]

Train faster, generalize better: Stability of stochastic gradient descent, 2016

Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent, 2016. URL https://arxiv.org/abs/1509.01240

arXiv 2016

[7] [7]

Deep networks always grok and here is why, 2024

Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why, 2024. URL https://arxiv.org/abs/2402.15555

arXiv 2024

[8] [8]

Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training

Rie Johnson and Tong Zhang. Inconsistency, instability, and generalization gap of deep neural network training, 2023. URL https://arxiv.org/abs/2306.00169

work page Pith review arXiv 2023