Pith. sign in

REVIEW 2 major objections 6 minor 25 references

Reading embedding effective rank at the grokking transition overstates the converged circuit by several fold, because compression keeps going long after accuracy jumps.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 00:45 UTC pith:CBPNUM3C

load-bearing objection Solid measurement paper: at-grok rank is a real transient, compression lags by ~T_grok, LayerNorm sets the lag size, and they ship a usable audit with adversarial tests and self-corrections. the 2 major comments →

arxiv 2607.06639 v1 pith:CBPNUM3C submitted 2026-07-07 cs.LG cs.AI

At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics

Truong Xuan Khanh This is my paper
classification cs.LG cs.AI
keywords grokkingeffective rankrepresentation metricsmeasurement validitycompression lagLayerNormmodular arithmeticnorm budget
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a common shortcut in grokking research—reading a representation metric at the moment test accuracy rises and treating it as a property of the generalizing circuit—is unsafe. On modular arithmetic, embedding effective rank is still high and nearly flat at that moment; long training then drives it down to a much lower floor, so the transition-time snapshot overstates converged complexity by about 3–5× on an MLP and 1.3–1.5× on a transformer, and on the MLP it even erases which runs compress at all. Compression does not coincide with generalization: it lags by an amount on the order of the time-to-grok (at least 10,000 steps). A one-variable ablation shows that normalization sets how large that lag is: adding LayerNorm alone moves the fraction of compression already done by the grok step from 0.87 to 0.25. The authors package the discipline as a tested measurement-validity audit that separates onset from compression, gates incomplete runs, checks that the reference floor itself has plateaued, and refuses ordering claims the data do not support.

Core claim

On modular arithmetic, a network’s embedding keeps compressing for tens of thousands of steps after it has already generalized. A value of embedding effective rank read at the grokking transition is therefore a transient: it overstates the converged floor by 3–5× on an MLP (and hides which cells compress) and by 1.3–1.5× on a transformer trained to true convergence. Compression lags the accuracy transition by an amount of order T_grok rather than coinciding with it, and the size of that lag is controlled by the normalization scheme—specifically, LayerNorm defers most of the compression past the grok step.

What carries the argument

The compression-clock audit: two clocks T_grok (first step test accuracy crosses the grokking threshold) and T_compress (first post-onset step at which the representation metric settles near its plateaued floor), plus a boundary gate, a floor-plateau check, a censoring flag, and the summary quantity frac-pre (fraction of total rank drop already completed by the grok step). Together they decide when a transition-time reading can be trusted.

Load-bearing premise

That the spectral effective rank of the embedding (or related spectral measures) is a faithful proxy for the representational compression that matters, so dating when that rank reaches its floor is the same as dating when the generalizing circuit has settled.

What would settle it

Train the same modular models past the accuracy jump, log embedding effective rank (or participation ratio / stable rank) to a true plateau, and check whether the at-grok value still sits several-fold above the floor and whether T_compress still lags T_grok by order T_grok; if the two clocks coincide and the at-grok value already equals the floor, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper argues that, on modular arithmetic, embedding effective rank read at the grokking transition is a transient: it overstates the long-train floor by ~3–5× on a modular MLP (and erases which norm-budget cells compress) and by ~1.3–1.5× on a transformer trained to convergence. Separately, rank compression lags the accuracy transition by an amount of order T_grok (≥10^4 steps) rather than coinciding with it. A one-variable free-decay ablation (MLP vs LayerNorm-free transformer vs the same transformer with LayerNorm) shows that LayerNorm moves frac-pre (fraction of rank compression already done by T_grok) from 0.87 to 0.25 and enlarges the lag; a pre-registered RMSNorm-on-embedding control rejects per-token scale invariance as the mechanism. The authors package an operational audit (T_grok vs T_compress, boundary gate, censoring flag, floor-plateau check, order-statistic-backed verdicts) with an adversarial suite and a third-party adapter, and report a secondary MLP-only “depth law” as a negative generality result (fails on a transformer; sign flips under free weight decay).

Significance. If the empirical claims hold—and the multi-task, multi-protocol, multi-architecture evidence plus self-correction and pre-registration make that plausible—this is a useful measurement-validity contribution to the grokking literature. The field increasingly reads representation structure at transition-time checkpoints; documenting a large, architecture-modulated transient and lag for a standard spectral metric, and shipping a tested procedure that refuses undefined orderings, is more valuable than another unguarded complexity curve. Explicit strengths: (i) separation of a measurement hazard from a timing claim with a named referent (Yunis et al.); (ii) pre-registered negative control on scale invariance; (iii) self-correction of an earlier small-grid “one-clock” reading; (iv) adversarial suite that caught a false-confidence regression; (v) released analyzer, adapter, sample data, and figure scripts. The secondary depth-law is correctly demoted rather than oversold.

major comments (2)
  1. [Abstract / Sec. 3 / Sec. 8] Abstract and §1 frame the contribution as an audit for “grokking representation metrics,” but the operational clock (Sec. 3: T_compress = first post-onset step within ε of a final-plateau floor) assumes a quantity that falls and settles. The paper itself documents that other representation quantities move the opposite way across the same transition (Fourier circuit-synchronization leads; H1 persistence rises; §2, §8, [19,7]). Metric-agnostic checks in §4 cover only participation ratio and stable rank—same spectral family. For the toolkit claim to match the title, the manuscript should state as a hard precondition (in the abstract takeaways and in the analyzer contract) that the audited metric must be monotone-compressing toward a floor, and that rising clocks require a dual definition; otherwise third-party use on PH/Fourier will silently mis-date.
  2. [Abstract / Sec. 6 / Sec. 7] The abstract and contribution bullets report the LayerNorm frac-pre shift 0.87→0.25 and the associated lag enlargement as a main positive finding, but §6 explicitly scopes that harness as low-powered (few seeds, 6×10^4-step budget) and notes that the dramatic ~3.2× LayerNorm transient is not the well-powered number (Sec. 7 gives 1.3–1.5× on the embedding). The direction is corroborated by Sec. 7’s lag/T_grok≈0.63, which is enough for a qualitative mechanism claim, but the headline coefficients in the abstract should carry the same power caveat already present in the body, or be replaced by the better-powered transformer lag figure, so the central positive result is not overstated relative to the evidence tier the authors themselves assign.
minor comments (6)
  1. [Table 1 / Sec. 2] Table 1’s “coinc.” for Yunis et al. is fair as a reading of their claim, but a short footnote clarifying that their simultaneous low-rank discovery is across weight matrices (not specifically the embedding) would reduce the risk of overstating the correction.
  2. [Sec. 3 / Appendix B] Appendix B Table 3: the lag/T_grok column uses ratio-of-medians while Table 2 uses median-of-ratios; the footnote explains the ~0.04 discrepancy, but putting both definitions once in Sec. 3 would help readers who only skim the main text.
  3. [Sec. 4 / Sec. 6 / Fig. 4] Figure 4C’s second late collapse is important for the floor-plateau check; a one-sentence pointer in the Sec. 4 “denominator can also mislead” paragraph to the exact floor-plateau criterion used by the analyzer would make the guard reproducible without reading the code.
  4. [Sec. 7 / Fig. 6] Sec. 7: “only 16 of 21 clamp cells and 14 of 18 free-decay cells reach 0.90” is appropriately flagged; consider stating in the figure captions for Fig. 6 how many seeds underlie each median so the generalization check’s power is visible at a glance.
  5. [Title / Figs. 1–3] Typos/style: title missing space after the colon (“Converged:A”); occasional “½” in figure axis labels appears to be a ρ rendering artifact in the manuscript text dump—verify in the camera-ready figures.
  6. [References] References include several 2026 arXiv items and the authors’ own related preprints [24,25]; ensure citation dates and versions are stable at camera-ready, and that [25] is cited in the main text if it is load-bearing for the spectral-entropy framing.

Circularity Check

0 steps flagged

No significant circularity: operational clocks measure empirical lag and transient; self-citations are complementary, not load-bearing.

full rationale

The paper’s central claims are empirical measurement facts about embedding (and related spectral) effective rank under long training: the at-grok value overstates the converged floor, compression lags T_grok by order T_grok, and LayerNorm moves frac-pre. T_grok (first step with median test accuracy ≥0.9) and T_compress (first post-onset step within ε of the plateaued floor) are independent operational definitions; the lag is not forced by construction—the adversarial suite includes compression-before-grok, censoring, high-floor boundary, and rebound cases, and the analyzer can decline ordering or return partially-separated/large-lag verdicts. Frac-pre is a descriptive ratio of observed ranks, not a fitted parameter renamed as a prediction. The secondary MLP depth law is self-falsified by pre-registered architecture and protocol tests rather than protected. Self-citations to the authors’ norm-separation delay law [24] are explicitly scoped as complementary (when grokking happens vs. whether a transition-time metric has converged) and as a future scaling check, not as premises that force the present lag or transient. No uniqueness theorem, ansatz smuggled via self-citation, or self-definitional reduction of a claimed first-principles result is present. The work is self-contained against its own multi-architecture/protocol runs, boundary/floor-plateau gates, and released adversarial suite.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The paper is empirical and measurement-focused; its load-bearing content is operational definitions plus experimental protocols rather than new physical entities. Free parameters are the audit thresholds (ε, gate, floor window, accuracy threshold). Domain assumptions are standard grokking setup choices (modular arithmetic, AdamW, norm clamp as control knob) and the choice of variance-normalized spectral-entropy effective rank as the primary metric. No new particles, forces, or conserved quantities are postulated.

free parameters (5)
  • compression tolerance ε = 0.10 (default)
    Fraction of the at-grok-to-floor drop that defines T_compress; default 0.10. Lag/T_grok ranges [0.65,1.46] as ε varies 0.05–0.20; the ≥10^4-step claim is invariant but the exact ratio is not.
  • boundary drop-threshold (gate thr) = 0.25
    Drop/at-grok ratio below which a cell is excluded as non-compressing; default 0.25. Sensitivity shows the gate always excludes only ρ=1.00 on addition.
  • floor-averaging window (floor frac) = 0.10
    Final fraction of training used to estimate the reference floor; default 0.10. Required because the floor itself can undergo a second late collapse.
  • grokking accuracy threshold = 0.90
    Median test accuracy that defines T_grok; default 0.90. Sensitivity table shows verdict family stable under 0.95/0.99.
  • norm-budget clamp values ρ = ρ ∈ [1.00,1.40] (MLP); [0.9,1.4] (transformer)
    Discrete grid of global-norm multipliers relative to free-dynamics grok norm; chosen by hand to span fast-to-slow grokking cells.
axioms (4)
  • domain assumption Variance-normalized spectral-entropy effective rank of a weight matrix is a valid proxy for representational complexity/compression in grokking circuits.
    Inherited from DeMoss et al. and Roy–Vetterli; used as the primary clocked quantity throughout. Metric-agnostic checks cover only participation ratio and stable rank.
  • domain assumption The global parameter-norm clamp ρ∥W∥_c is a valid control knob that isolates post-grok representation dynamics without changing the generalizing solution class.
    Standard in the weight-norm line (Liu et al.); free-decay replications are used to show the transient is not clamp-specific.
  • ad hoc to paper T_grok = first step with median test accuracy ≥ threshold; T_compress = first post-onset step within ε of the final-plateau floor.
    Operational definitions introduced in Sec. 3; the audit’s contribution is the discipline around them (censoring, boundary gate, floor-plateau check).
  • domain assumption Modular arithmetic (p=59) with two-layer MLP / one-layer attention is a representative setting for studying grokking representation dynamics.
    Standard grokking benchmark; parity is added as a non-arithmetic control, but larger non-modular systems are left open.
invented entities (2)
  • frac-pre (fraction of embedding-rank compression completed by the grok step) independent evidence
    purpose: Scalar that orders architectures by how much compression precedes vs. follows generalization; used to identify LayerNorm as the lag-size modulator.
    Defined as (r_init − r_grok)/(r_init − r_floor). Operational, not a new physical entity; independent evidence is the one-variable ablation and the agreeing transformer lag direction.
  • compression-clock audit (T_grok / T_compress with boundary gate, censoring flag, floor-plateau check) independent evidence
    purpose: Tested procedure that decides whether a transition-time representation metric has converged.
    The paper’s main methodological product; adversarial suite provides falsifiable handles on each verdict reason.

pith-pipeline@v1.1.0-grok45 · 27792 in / 3847 out tokens · 38663 ms · 2026-07-11T00:45:39.992008+00:00 · methodology

0 comments
read the original abstract

On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized. Reading effective rank at the grokking transition overstates the converged value by 3-5x on an MLP, and by 1.3-1.5x on a transformer trained to convergence; on the MLP it also erases which cells compress at all. Compression lags the accuracy transition by an amount on the order of the time-to-grok, at least 10,000 steps, rather than coinciding with it. A one-variable ablation shows what sets the lag size: adding LayerNorm to an otherwise identical transformer moves the fraction of compression done by the grok step from 0.87 to 0.25, and a pre-registered control rules out scale invariance as the mechanism. We package this as an audit that separates onset from compression, flags censoring, excludes boundary cells that never fully generalize, and checks that the reference floor has plateaued, with an adversarial suite that caught a false-confidence bug in our own branch. A secondary, MLP-specific depth law linking norm budget to converged floor fails a generality test on a transformer and flips sign under free weight decay. Code and the toolkit are released.

Figures

Figures reproduced from arXiv: 2607.06639 by Truong Xuan Khanh.

Figure 1
Figure 1. Figure 1: Compression-lag dashboard on the modular-addition MLP ( [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Reading effective rank at grokking inverts the picture. Embedding effective rank read at grokking [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The same audit on the modular-multiplication MLP ( [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: One harness, three architectures, one variable at a time. Embedding squared-normalized effective [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The pre-registered scale-invariance control (thresholds frozen before the run). Adding RMS [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The at-grok transient generalizes to the transformer. Effective rank (Roy–Vetterli) read at grokking [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The norm-budget depth law is MLP-specific (negative result). Converged effective-rank floor vs. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · 21 internal anchors

  1. [1]

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

    A. Power, Y. Burda, H. Edwards, I. Babuschkin, V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv:2201.02177, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  2. [2]

    Z. Liu, E. J. Michaud, M. Tegmark. Omnigrok: Grokking beyond algorithmic data.ICLR, 2023. arXiv:2210.01117

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    Explaining grokking through circuit efficiency

    V. Varma, R. Shah, Z. Kenton, J. Kram´ ar, R. Kumar. Explaining grokking through circuit efficiency. arXiv:2309.02390, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Grokking as the Transition from Lazy to Rich Training Dynamics

    T. Kumar, B. Bordelon, S. J. Gershman, C. Pehlevan. Grokking as the transition from lazy to rich training dynamics.ICLR, 2024.arXiv:2310.06110

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Nanda, L

    N. Nanda, L. Chan, T. Lieberum, J. Smith, J. Steinhardt. Progress measures for grokking via mechanistic interpretability.ICLR, 2023

    work page 2023

  6. [6]

    The Complexity Dynamics of Grokking

    B. DeMoss, S. Sapora, J. Foerster, N. Hawes, I. Posner. The complexity dynamics of grokking. arXiv:2412.09810;Physica D, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Y. Tang, Q. Wang, I. Garc´ ıa-Redondo, A. Monod. Topological signatures of grokking.arXiv:2605.06352, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [8]

    Approaching Deep Learning through the Spectral Dynamics of Weights

    D. Yunis, K. K. Patel, S. Wheeler, P. Savarese, G. Vardi, K. Livescu, M. Maire, M. R. Walter. Approach- ing deep learning through the spectral dynamics of weights.arXiv:2408.11804, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Mechanistic Insights into Grokking from the Embedding Layer

    H. AlQuabeh, V. Bojkovi´ c, M. Nwadike, K. Inui. Mechanistic insights into grokking from the embedding layer.arXiv:2505.15624, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Measuring Sharpness in Grokking

    J. Miller, P. Gleeson, C. O’Neill, T. Bui, N. Levi. Measuring sharpness in grokking.arXiv:2402.08946, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Z. Wang, Y. Ying, T. Kanamori. Distributional spectral diagnostics for localizing grokking transitions. arXiv:2605.08237, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [12]

    L. Verma. Weight decay regimes in grokking transformers: cheap online diagnostics.arXiv:2605.20441, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  13. [13]

    Y. Xu. Early-warning signals of grokking via loss-landscape geometry.arXiv:2602.16967, 2026. 21

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [14]

    Y. Xu. The geometry of multi-task grokking: transverse instability, superposition, and weight decay phase structure.arXiv:2602.18523, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  15. [15]

    P. Wang. Grokking as dimensional phase transition in neural networks.arXiv:2604.04655, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  16. [16]

    H. K. Prakash, C. H. Martin. Late-stage generalization collapse in grokking: detecting anti-grokking with WeightWatcher.arXiv:2602.02859, 2026

    work page arXiv 2026

  17. [17]

    O. Roy, M. Vetterli. The effective rank: a measure of effective dimensionality.EUSIPCO, 2007

    work page 2007

  18. [18]

    B. C. A. Brown, J. Juravsky, A. L. Caterini, G. Loaiza-Ganem. Relating regularization and gener- alization through the intrinsic dimension of activations.NeurIPS 2022 Workshop on Optimization for Machine Learning (OPT2022), 2022.arXiv:2211.13239

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  19. [19]

    Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking

    A. Sivasankar. Circuit synchronization precedes generalization: a causal precursor to grokking. arXiv:2606.12966, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [20]

    S. Golwala. ILDR: geometric early detection of grokking.arXiv:2604.20923, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    S. B. Manir, A. P. Rupa. A systematic empirical study of grokking: depth, architecture, activation, and regularization.arXiv:2603.25009, 2026

    work page arXiv 2026

  22. [22]

    M. Li. Low-rank decay for grokking in scale-invariant transformers: a spectral-geometric view. arXiv:2606.04405, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  23. [23]

    I. T. Ersoy, K. Wiesner. Noise-driven escape from metastable phases explains grokking in deep neural networks.HiLD Workshop, 2026.arXiv:2606.17120

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  24. [24]

    X. K. Truong, Q. H. Truong, D. T. Luu, T. D. Phan. The norm-separation delay law of grokking: a first-principles theory of delayed generalization.arXiv:2603.13331, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin

    X. K. Truong, Q. H. Truong, D. T. Luu, T. D. Phan. Spectral entropy collapse as an empirical signature of delayed generalisation in grokking.arXiv:2604.13123, 2026. 22 A The norm-budget depth law (MLP-specific, self-falsified) This appendix collects the secondary observation that the norm budget sets the depth of compres- sion on the modular MLP, and the ...

    work page internal anchor Pith review Pith/arXiv arXiv 2026