Grokking and Generalization Collapse: Insights from \texttt{HTSR} theory

Charles H. Martin; Hari K. Prakash

REVIEW 4 major objections 4 minor 16 references

Grokking and Generalization Collapse: Insights from \texttt{HTSR} theory

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read After very long training without weight decay, a grokked network's test accuracy collapses to near chance while training accuracy stays perfect, and the spectral exponent alpha falling below 2 heralds the collapse.

desk verdict One genuine new observation — late-training collapse after grokking under zero weight decay — but it rests on a single seed-0 run, so the phenomenon is promising, not yet established. read the letter →

arxiv 2506.04434 v1 pith:OEMPTWTW submitted 2025-06-04 cs.LG cs.AIstat.ML

Hari K. Prakash , Charles H. Martin This is my paper

classification cs.LGcs.AIstat.ML

keywords grokkinganti-grokkingheavy-tailedself-regularizationspectralexponentcorrelationtrapsgeneralizationcollapserandommatrixtheoryweightdecay

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that grokking, the delayed jump in test accuracy long after training accuracy saturates, has a late-time endpoint that earlier studies missed: if the network trains far past the grokking transition with zero weight decay, test accuracy collapses back to near chance while training accuracy stays perfect. The paper calls this third phase anti-grokking and claims it is invariably heralded by the heavy-tailed self-regularization (HTSR) layer quality metric $\alpha$, the heavy-tailed power-law exponent of each layer weight spectrum, falling below $2$, together with the appearance of correlation traps, outlier eigenvalues in the elementwise-randomized weight matrix. A careful reader would care because $\alpha$ is computed from weights alone, so the claim offers a way to sense overfitting and impending generalization collapse without test-set access, and because it says that very long training runs are themselves a generalization risk even after the model has grokked.

What carries the argument

The central object is $\alpha$, the exponent of the heavy-tailed power-law fit to the empirical spectral density of each layer's Gram matrix $X = \tfrac{1}{N}W^{\top}W$, so that $\rho_{\mathrm{emp}}(\lambda) \sim \lambda^{-\alpha}$ over a chosen tail region. The argument sets the random-matrix Marchenko--Pastur density as the null model for uncorrelated weights and treats deviations from it as learned structure; $\alpha \approx 2$ is the claimed universal layer-convergence target, with $2 \lesssim \alpha \lesssim 5$ indicating well-conditioned layers, $\alpha \gtrsim 5$ indicating underfit or random-like layers, and $\alpha < 2$ indicating very heavy tails and overfitting. The companion mechanism is the correlation trap: an eigenvalue of the elementwise-randomized weight matrix lying far outside the best-fit Marchenko--Pastur edge $\lambda_+$, interpreted as an anomalous rank-one-or-higher perturbation that makes the layer's weight distribution atypical. The paper uses these tools to read off which phase the network is in at each stage of training.

What would settle it

Retrain the same MLP on the same 1,000-sample MNIST subset with zero weight decay and the same hyperparameters across ten random seeds out to $10^7$ steps, logging test accuracy and layer $\alpha$; if even one seed keeps test accuracy high past $10^7$ without an $\alpha<2$ episode, or shows a collapse with $\alpha$ never below $2$, the 'invariable herald' claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that training a depth-3, width-200 ReLU MLP on 1,000 MNIST samples with mean-squared error and no weight decay for $10^7$ steps exposes a third phase beyond pre-grokking and grokking: test accuracy falls to roughly $0.5$ while training accuracy remains perfect and the $\ell^2$ weight norm keeps growing. The paper argues that pre-grokking is a mixed-layer state, where some layers approach $\alpha \approx 4$ while others stay near $\alpha \approx 5$, that grokking peaks when all important layers approach $\alpha \approx 2$, and that anti-grokking is marked by one or more layers crossing below $\alpha = 2$, with correlation traps visible in the spectrum of the randomized weights. It also claims that only the HTSR $\alpha$ separates all three phases: activation sparsity, absolute weight entropy, approximate local circuit complexity, and the $\ell^2$ norm track the first two phases but fail to distinguish grokking from anti-grokking. In the weight-decay control run the $\alpha$ curve plateaus near $2$ and test accuracy plateaus instead of collapsing, which the paper reads as corroboration that $\alpha$ marks the phase.

Load-bearing premise

The headline claim is carried by one long run: a single random seed, one 3-layer MLP, and one 1,000-sample MNIST subset, with no seed-averaged curves, so the assertion that $\alpha<2$ and correlation traps invariably herald collapse has only one trajectory behind it.

Editorial extensions

If this is right

The paper's claim implies that, without weight decay, training far past grokking does not stay at peak test accuracy; the model can silently enter anti-grokking, so training duration itself becomes a generalization risk.
Tracking $\alpha$ per layer can warn before test accuracy falls, because the crossing of $\alpha<2$ precedes or coincides with collapse and the metric needs only the weights, not labels or test data.
Explanations of grokking built on the $\ell^2$ norm, activation sparsity, entropy, or circuit complexity miss the late failure mode; the $\alpha=2$ cutoff is the benchmarked signal that separates healthy grokking from collapse.
In the weight-decay control, $\alpha$ plateaus near $2$ and test accuracy stays on a plateau rather than collapsing, suggesting that weight decay keeps layers out of the $\alpha<2$ regime and may protect against anti-grokking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's evidence for the word 'invariably' is a single seed-0 trajectory; seed-averaged runs with varied optimizers and learning rates would determine whether anti-grokking is a robust phase or an artifact of one initialization.
Because alpha is a function of weights alone, the same diagnostic could be applied to transformers and convolutional networks under long training, although the paper studies only one MLP on MNIST and leaves that extension implicit.
The correlation-trap description, a large mean shift that makes the weight distribution atypical, resonates with rank-collapse and benign-overfitting ideas in other settings, but connecting those ideas is the reader's inference, not the paper's.
If the alpha-about-2 target is as universal as claimed, an alpha-guided regularizer or early-stopping rule is a natural testable application; the paper lists such strategies as future work without demonstrating them.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

John — quick take. The noteworthy thing here is the empirical observation that a 3-layer MLP trained for 10^7 steps on 1k MNIST without weight decay can grok and then later collapse to near chance while train accuracy stays perfect. I don't think that specific late-time collapse has been reported in the grokking literature, and the comparison against Golechha's progress measures is useful. If the effect is real, it gives practitioners a test-set-free overfitting signal.

What the paper does well: it tracks several metrics on the same runs, it runs a WD>0 control, and it is candid about its own limitations. Appendix E says explicitly that the alpha-to-performance mapping is not bidirectional and that generalization across architectures remains open. The use of WeightWatcher is at least reproducible in the sense that the tool is public, and the KS/MP null tests on randomized weights are a concrete, falsifiable check for correlation traps. The distinction between un-grokking (dataset shrunk, with weight decay) and this late collapse (same dataset, no weight decay) is real and clearly stated.

Where I'd push back. The main claim rests on a single seed-0 run. Table 3 fixes Random Seed 0, Figure 1 shows one trajectory, and no seed-averaged test curves for the collapse are given. The word 'invariably' in the abstract is not supported by the evidence. Table 1 mentions 'various seeds' for the alpha values, but those runs are not described and no error bars appear on the central collapse curve. So anti-grokking is currently one long trajectory; it could be a real phase, or it could be an atypical basin or optimizer drift after 10^7 steps. The alpha<2 threshold is also inherited from the authors' own HTSR/SETOL line, and the paper doesn't show it was calibrated independently of the run it is applied to. In Figure 4, alpha dips below 2 around the same time as the collapse, not clearly before it, so 'heralds' is doing more work than the data support. I also see a minor inconsistency: Table 2 reports correlation traps in the WD>0 setting, but Appendix C says WD>0 reaches a stable plateau with no collapse. If traps can appear without collapse, then the 'definitive signal' language is too strong.

None of this kills the paper. It is a solid observational contribution with an honest limitations section. For a serious referee, I'd want multi-seed replication, released code and checkpoints, and a quantitative statement about lead time: does alpha<2 precede the test drop by a consistent number of steps, and how robust is that across seeds? Send it out; treat the central phenomenon as promising but not established.

Referee Report

4 major / 4 minor

Summary. The paper studies grokking dynamics in a 3-layer MLP (width 200) trained on a 1,000-sample MNIST subset for up to 10^7 steps with zero weight decay, and reports a novel third phase called "anti-grokking": after an initial grokking phase, test accuracy collapses to chance while training accuracy remains perfect. The authors claim that their heavy-tailed self-regularization (HTSR) layer quality metric alpha alone delineates all three phases (pre-grokking, grokking, anti-grokking), that alpha falling below 2 "invariably" heralds the collapse, and that this collapse is accompanied by "correlation traps"—outlier eigenvalues of randomized weight matrices detected by the WeightWatcher tool and validated by Kolmogorov-Smirnov tests. They further argue that competing metrics (activation sparsity, absolute weight entropy, approximate local circuit complexity, l2 norm) fail to distinguish grokking from anti-grokking.

Significance. If the central claims hold, the paper would contribute a striking new empirical phenomenon—late-training generalization collapse without weight decay—and a practical spectral progress measure that can detect it without test-set access. The use of an established open-source tool, the explicit comparison against several prior progress measures, and the attempt to validate correlation traps with KS tests are strengths. However, the significance is currently contingent on the robustness of a single long trajectory, and the paper's strongest language ("invariably," "definitive signal") goes beyond the evidence provided. The claimed novelty of the anti-grokking phase deserves serious scrutiny, but the manuscript needs substantial additional experimental evidence and more careful hedging before the main claims can be accepted.

major comments (4)

[Appendix A, Table 3; §4.1, Fig. 1] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.
[§4.1, Fig. 4; §3.1] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.
[§4.2, Table 2; Appendix D; Appendix E] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.
[§3.1, §4.1, Refs. [8,9,10,11]] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.

minor comments (4)

[Appendix D, Table 4] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.
[§4.1 and Figure 5 caption] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.
[Throughout] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.
[Equation (9)] This comment is identical to the one above but appears twice in the array; the JSON schema does not require unique comments, but the author should treat this as a single point. This duplicate is a formatting artifact; please ignore the second occurrence.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity: alpha is measured from weights independently of test accuracy, and the self-cited HTSR threshold functions as external theory rather than an input that constructs the result.

full rationale

The three phases (pre-grokking, grokking, anti-grokking) are defined by the independently observed training and test accuracy curves (Figure 1), not by alpha. Alpha is computed from layer weight spectra via PL fits and randomized-matrix MP comparisons; it is not fitted to the test accuracy labels, and no equation in the paper defines test collapse in terms of alpha or vice versa. The claim that alpha < 2 heralds collapse imports the alpha = 2 cutoff and correlation-trap terminology from prior HTSR/SETOL work by the same authors (refs. [9,10,11]), but those works are external evidence rather than a reduction of this paper's observations to its own inputs, and the paper adds its own KS-test and visual spectral validation of the traps. The main weaknesses are statistical and evidentiary: the headline anti-grokking finding rests on a single seed-0 WD=0 trajectory, the word 'invariably' is not supported by seed-averaged curves, Table 2 labels the WD>0 endpoint 'Anti-grokking (Collapse)' even though Appendix C reports a plateau rather than a collapse, and Appendix E itself concedes that alpha below 2 does not strictly imply suboptimal generalization. These are correctness and robustness concerns, not circularity-by-construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The paper's central claims rest on HTSR theory and the alpha threshold, both largely self-authored, plus the representativeness of a single long training run. No new free parameters are fit in this paper; the alpha cutoff and phase labels are assumed from prior work.

assumptions (4)

domain assumption ESD of trained layer weights follows a power-law tail rho ~ lambda^{-alpha}, and alpha ranges correspond to training phases with alpha=2 optimal and alpha<2 overfit.
Central interpretive lens from HTSR papers [10,11] and SETOL [9] by the same authors; not derived in this paper.
standard math Elementwise randomization W to Wrand produces i.i.d. weights whose ESD should fit the Marchenko-Pastur distribution, and deviations indicate correlation traps.
Eqs. 3-6; MP is the standard null model for i.i.d. entries and is used to detect outliers.
domain assumption Power-law exponent alpha estimated via Clauset MLE with automated lambda_min selection is reliable.
Section 3.1; sensitivity of alpha to lambda_min is acknowledged but assumed handled by the WeightWatcher tool.
domain assumption The 10^7-step WD=0 training dynamics are a normal optimizer trajectory, not a numerical artifact.
The main result depends on this single trajectory; no ablations of optimizer, learning rate, or seed are shown.

invented entities (2)

correlation trap independent evidence
purpose: Outlier singular values in randomized weight matrices Wrand, claimed to signal overfitting and anti-grokking.
Defined operationally as lambda_trap much greater than the MP bulk edge and detected by KS tests, but the causal link to overfitting is hypothesized, not proven.
anti-grokking independent evidence
purpose: Late-stage generalization collapse after grokking with perfect training accuracy, proposed as a distinct third phase.
Observable in the test accuracy curve, but its distinctness from generic overfitting or double descent is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Grokking and Generalization Collapse: Insights from \texttt{HTSR} theory." pith.science (2026). https://pith.science/paper/OEMPTWTW

@misc{pith2026250604434,
  author       = {Pith},
  title        = {Pith review of: Grokking and Generalization Collapse: Insights from \textttHTSR theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OEMPTWTW}},
  note         = {Machine review of arXiv:2506.04434}
}

abstract

We study the well-known grokking phenomena in neural networks (NNs) using a 3-layer MLP trained on 1 k-sample subset of MNIST, with and without weight decay, and discover a novel third phase -- \emph{anti-grokking} -- that occurs very late in training and resembles but is distinct from the familiar \emph{pre-grokking} phases: test accuracy collapses while training accuracy stays perfect. This late-stage collapse is distinct, from the known pre-grokking and grokking phases, and is not detected by other proposed grokking progress measures. Leveraging Heavy-Tailed Self-Regularization HTSR through the open-source WeightWatcher tool, we show that the HTSR layer quality metric $\alpha$ alone delineates all three phases, whereas the best competing metrics detect only the first two. The \emph{anti-grokking} is revealed by training for $10^7$ and is invariably heralded by $\alpha < 2$ and the appearance of \emph{Correlation Traps} -- outlier singular values in the randomized layer weight matrices that make the layer weight matrix atypical and signal overfitting of the training set. Such traps are verified by visual inspection of the layer-wise empirical spectral densities, and by using Kolmogorov--Smirnov tests on randomized spectra. Comparative metrics, including activation sparsity, absolute weight entropy, circuit complexity, and $l^2$ weight norms track pre-grokking and grokking but fail to distinguish grokking from anti-grokking. This discovery provides a way to measure overfitting and generalization collapse without direct access to the test data. These results strengthen the claim that the \emph{HTSR} $\alpha$ provides universal layer-convergence target at $\alpha \approx 2$ and underscore the value of using the HTSR alpha $(\alpha)$ metric as a measure of generalization.

Figures

Figures reproduced from arXiv: 2506.04434 by the authors.

**Figure 1.** The three phases of grokking. Training curves for a depth-3, width-200 MLP on MNIST. The initial pre-grokking phase (grey): training accuracy (red line) surges at 102 steps, saturating between 104−105 steps, while test accuracy (purple line) remains low; the grokking phase (yellow): with test accuracy rapidly increasing after ∼ 105 steps, and reaching a maximum at 106 steps; and the newly revealed late-stage anti-gr… view at source ↗

**Figure 2.** (Left) shows an example of a PL fit on a log-log scale for a representative layer after training. The plot displays the ESD for a typical NN layer (a histogram or kernel density estimate of eigenvalues), the automatically chosen λmin (xmin, vertical line, red), the λmax (xmax, vertical line, orange), and the best fit for the PL tail (dashed line, red) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Examples of Correlation Traps. ESDs of (Wrand) (light purple) of Layer 2 for the randomized weight matrix Wrand for different models, compared to an MP fit (red). Correlation traps λtrap are depicted as small spikes to the right of the MP fit. (x-axis is log scale) Left: Right Before Collapse (i.e. at more than ∼ 106 steps) (σmp ≈ 0.9879). The KS test (P-value ≈ 4×10−13) indicates a strong deviation from the MP mode… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: HTSR results vs. optimization steps. Top: Average α across layers. Middle: α for the first fully connected layer (FC1). Bottom: α for the second fully connected layer (FC2). Note the significant dip below the critical threshold α = 2, especially in FC2, coinciding with…

**Figure 5.** Figure 5: Alternative progress measures (Golechha [2]) vs. optimization steps. Top: Activation [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: HTSR α exponent evolution for the MLP trained with WD=0.01 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Progress measures (Activation Sparsity, Weight Entropy, Circuit Complexity) and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

[1]

Aaron Clauset, Cosma Rohilla Shalizi, and Mark E.J. Newman. Power-law distributions in empirical data. SIAM Review, 51(4):661–703, 2009

work page 2009
[2]

Progress measures for grokking on real-world tasks, 2024

Satvik Golechha. Progress measures for grokking on real-world tasks, 2024

work page 2024
[3]

The impact of ac- tivation sparsity on overfitting in convolutional neural networks

Karim Huesmann, Luis Garcia Rodriguez, Lars Linsen, and Benjamin Risse. The impact of ac- tivation sparsity on overfitting in convolutional neural networks. InPattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part III, volume 12663 of Lecture Notes in Computer Science , pages 130–145. Spring...

work page 2021
[4]

Deep networks always grok and here is why, 2024

Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why, 2024

work page 2024
[5]

Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar

Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phe- nomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations (ICLR) , 2023. arXiv:2210.06313

arXiv 2023
[6]

Nolte, Eric J

Ziming Liu, Ouail Kitouni, Niklas S. Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Surbhi Koyejo, Sham Kakade (formerly Mohamed), Aarti Agarwal, Danielle Belgrave, Kyunghyun Cho, and Alice Oh, editors, Advances in Neural Information Processing Systems , volume 35,...

work page 2022
[7]

Marchenko and Leonid Andreevich Pastur

Vladimir A. Marchenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheskii Sbornik, 72(114)(4):507–536, 1967

work page 1967
[8]

Charles H. Martin. WeightWatcher: Analyze Deep Learning Models without Training or Data. https://github.com/CalculatedContent/WeightWatcher, 2018-2024. Ver- sion 0.7.5.5 used in this study. Accessed May 12, 2025

work page 2018

Show all 16 references

[9]

Martin, Christopher Hinrichs, and Michael W

Charles H. Martin, Christopher Hinrichs, and Michael W. Mahoney. SETOL: A Semi- Empirical Theory of (Deep) Learning.https://github.com/CalculatedContent/ setol_paper/blob/main/setol_draft.pdf, 2025. Preprint. 9

2025
[10]

Martin and Michael W

Charles H. Martin and Michael W. Mahoney. Implicit self-regularization in deep neural net- works: Evidence from heavy-tailed spectral analysis, 2021

2021
[11]

Martin and Michael W

Charles H. Martin and Michael W. Mahoney. Predicting trends in the quality of state-of-the-art deep learning models, 2021

2021
[12]

A tale of two circuits: grokking as competition of sparse and dense subnetworks, 2023

William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: grokking as competition of sparse and dense subnetworks, 2023

2023
[13]

Progress measures for grokking via mechanistic interpretability, 2023

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023

2023
[14]

Theoretical explanation of activation sparsity through flat minima and adversarial robustness, 2023

Ze Peng, Lei Qi, Yinghuan Shi, and Yang Gao. Theoretical explanation of activation sparsity through flat minima and adversarial robustness, 2023

2023
[15]

Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

2022
[16]

Explaining grokking through circuit efficiency, 2023

Vikrant Varma, Rohin Shah, Zachary Kenton, J ´anos Kram´ar, and Ramana Kumar. Explaining grokking through circuit efficiency, 2023. 10 Appendices A Experimental Setup We train a Multi-Layer Perceptron (MLP) on a subset of the MNIST dataset using the hyperparame- ters detailed ...

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Aaron Clauset, Cosma Rohilla Shalizi, and Mark E.J. Newman. Power-law distributions in empirical data. SIAM Review, 51(4):661–703, 2009

work page 2009

[2] [2]

Progress measures for grokking on real-world tasks, 2024

Satvik Golechha. Progress measures for grokking on real-world tasks, 2024

work page 2024

[3] [3]

The impact of ac- tivation sparsity on overfitting in convolutional neural networks

Karim Huesmann, Luis Garcia Rodriguez, Lars Linsen, and Benjamin Risse. The impact of ac- tivation sparsity on overfitting in convolutional neural networks. InPattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part III, volume 12663 of Lecture Notes in Computer Science , pages 130–145. Spring...

work page 2021

[4] [4]

Deep networks always grok and here is why, 2024

Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why, 2024

work page 2024

[5] [5]

Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar

Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phe- nomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations (ICLR) , 2023. arXiv:2210.06313

arXiv 2023

[6] [6]

Nolte, Eric J

Ziming Liu, Ouail Kitouni, Niklas S. Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Surbhi Koyejo, Sham Kakade (formerly Mohamed), Aarti Agarwal, Danielle Belgrave, Kyunghyun Cho, and Alice Oh, editors, Advances in Neural Information Processing Systems , volume 35,...

work page 2022

[7] [7]

Marchenko and Leonid Andreevich Pastur

Vladimir A. Marchenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheskii Sbornik, 72(114)(4):507–536, 1967

work page 1967

[8] [8]

Charles H. Martin. WeightWatcher: Analyze Deep Learning Models without Training or Data. https://github.com/CalculatedContent/WeightWatcher, 2018-2024. Ver- sion 0.7.5.5 used in this study. Accessed May 12, 2025

work page 2018