Pith. sign in

REVIEW 2 major objections 5 minor 17 references

The grokking delay is causally the time it takes to form the right task features, not a fixed optimization constant.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:59 UTC pith:RXBWQD4P

load-bearing objection Clean causal intervention on grokking: content-matched SupCon priors give a true/sibling/random gradation, norm-matched controls rule out pure norm mediation, and a predicted clamp turns the bimodal effect into a reliable accelerator—scoped to modular addition. the 2 major comments →

arxiv 2607.04333 v2 pith:RXBWQD4P submitted 2026-07-05 cs.LG

Structure-Specific Representational Priors Causally Control the Grokking Delay

Gunner Levi Howe This is my paper
classification cs.LG
keywords grokkingrepresentational priorsmodular additionsupervised contrastive learningweight-norm delayfeature formationtransformersgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Grokking is the long lag between a network memorizing its training set and actually generalizing. This paper shows that lag is not an unavoidable quirk of training dynamics: it is the time needed to build the right internal structure for the task. By forcing a small transformer on modular addition to organize its hidden states according to different partitions—true addition, a coherent sibling (subtraction), or pure randomness—the authors make generalization appear, survive, or vanish while holding the loss form, strength, and geometry fixed. Only the true structure also speeds the transition, and once a side-effect on weight norm is removed the speedup becomes reliable and large. A sympathetic reader cares because this turns an observational story about representation formation into a causal, bidirectional control knob over when generalization arrives.

Core claim

With identical loss form, strength, class sizes, and geometry, whether a one-layer transformer ever generalizes on modular addition tracks the structural content of an injected representational prior: true equivalence classes yield generalization in 22/30 runs, a coherent-but-wrong sibling that reuses the same periodic features yields 14/15, and a size-matched random partition yields 0/20. A weight-norm-matched plain cross-entropy control also yields 0/15, so the norm trajectory is not the mediator. Only the true structure additionally accelerates the transition (up to 2.75×, and median 8.6× once the norm is clamped). Structure formation in probes precedes and predicts every accuracy jump. T

What carries the argument

A supervised-contrastive auxiliary loss on normalized projections of the residual stream, whose only free parameter is the definition of positive sets. That definition is set to true modular-addition classes, modular-subtraction classes, or a fixed random partition, so that differences in outcome isolate structural content. A subsequent weight-norm clamp turns the same prior into a reliable accelerator by removing the inflation side-effect that otherwise races structure formation.

Load-bearing premise

The causal story is established on one modular-arithmetic task and one tiny transformer without LayerNorm, where the generalizing Fourier circuit is already known ground truth; transfer beyond that regime is untested.

What would settle it

Train the same model with a coherent, learnable positive structure built from a different feature family (for example magnitude bands of the inputs that periodic embeddings cannot express); if that structure still permits reliable generalization like the sibling subtraction prior, the feature-level account is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The grokking delay can be steered in both directions by representational content alone: true structure shortens it, random structure abolishes it.
  • The weight-norm delay law is conditional on structure-agnostic training; a strong structural objective can generalize at norms where plain cross-entropy saturates and dies.
  • Clamping the weight norm while seeding the true structure converts a bimodal intervention into a standalone accelerator with median 8.6× (up to 22×) speedup that grows monotonically as the norm target is lowered.
  • Reliability of generalization is decided by feature-family coherence; speed requires matching the task’s actual equivalence structure.
  • Structure-agnostic accelerators (gradient filtering) and structure-specific priors do not beneficially compose at the strengths tested.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the feature-level account is correct, any algorithmic task whose generalizing solution lives in a known low-dimensional feature family should admit an analogous positive-set prior that collapses its own grokking delay.
  • A self-supervised version that harvests positives from data invariances rather than labels would remove the need to know the equivalence structure in advance and is the natural route to larger models.
  • The race between structure seeding and norm-driven saturation suggests that other auxiliary losses that inflate weight norms may hide similar stall modes that become visible only under matched-seed paired comparisons.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper tests whether the grokking delay is causally the time to form task-structured representations. In a one-layer transformer on modular addition (p=97), a supervised-contrastive auxiliary loss injects positives encoding (i) true structure ((a+b) mod p), (ii) coherent-but-wrong sibling ((a-b) mod p), or (iii) a size-matched random partition, holding loss form, strength, class sizes, and geometry fixed. Generalization tracks content: true 22/30, sibling 14/15, random 0/20 (Fisher p=1.3e-7); a weight-norm-matched CE control that replays the inflated-norm trajectory never generalizes (0/15). Representation probes (Fourier concentration, class cosine gap, CKA) rise before accuracy jumps in all runs. Only true structure accelerates (up to 2.75x), but dose-dependently and bimodally. A race account (structure seeding vs norm-driven saturation) predicts that clamping/replaying/annealing the norm should preserve speedup and remove stalls; three mitigations confirm this, with standalone norm-clamp giving median 8.6x (up to 22x) and monotone speedup as the norm is held lower. The authors conclude the delay is the time to form the right structure, decided at the level of features rather than labels.

Significance. If the result holds within its scope, this is a clean causal intervention on a previously observational claim about grokking. The design is a genuine strength: content-matched SupCon conditions, a learnable sibling control, a weight-norm-matched counterfactual, representation timing on all 95 runs, and a falsifiable race prediction tested with three independent mitigations that convert a bimodal intervention into a reliable accelerator. That package advances the field beyond structure-agnostic levers (Grokfast, norm clamping, radial penalties) by showing structural content itself is a bidirectional control on whether and when generalization occurs. The work is carefully scoped to modular addition with a known Fourier circuit; transfer is untested, which the authors state. The prediction-and-confirmation loop and the explicit bounding of the weight-norm delay law are particularly valuable contributions.

major comments (2)
  1. [Abstract; Discussion §6; Limitations] Abstract final sentence and Discussion §6 (“Right structure is decided at the feature level”): the load-bearing slogan that the operative variable is features rather than labels rests on a single sibling ((a-b) mod p) that, by the authors’ own account, still requires the same periodic token features as the task. Limitations correctly flags the discriminating experiment (a coherent, learnable partition from a different feature family, e.g. magnitude bands of a) as unrun. Without it, the data equally support the weaker reading that any algebraically coherent, size-matched, Fourier-expressible partition preserves the path to generalization while only memorization-only partitions abolish it. The true/sibling/random gradation and norm-matched control remain strong for structure-specificity; the feature-level refinement should be hedged in the abstract and Discussion to match the evidence, or
  2. [Abstract; §5; Table 2] Table 2 and §5: stall elimination is significant only when pooled across three mitigations and two λ values (0/40 vs 6/20, p=7.7e-4), not per method (each 10/10 vs 8/10, Fisher p=0.47). The authors correctly rest the accelerator claim on the monotone speedup with held norm rather than on per-method stall counts. That is methodologically sound, but the abstract’s phrasing (“the residual stalls also vanish, though significant only pooled…”) still leads with stall vanishing. Tighten the abstract and §5 so the primary claim is the monotone delay-law control, with pooled stall counts as secondary corroboration only.
minor comments (5)
  1. [Table 1; §4.1] Table 1: “Median paired ratio” for SupCon-true λ=1.0 is reported as 0.80 while the text emphasizes up to 2.75x acceleration and a heavy stalled tail; a short note that the median is pulled by censored runs (scored at budget) would prevent misreading the acceleration claim.
  2. [§4.5; Figure 2] Figure 2 caption and §4.5: the operationalization of “structure precedes generalization” (cosine gap >0.05 or Fourier top-8 >0.45 before 95% test acc; no Fourier >0.8 without generalizing) is only in a footnote. Promote the thresholds into the main text or figure caption so the 95/95 claim is checkable without hunting.
  3. [§3.1] §3.1: omitting LayerNorm is justified for the norm-matched control, but a one-sentence note on whether the qualitative true/sibling/random pattern is expected to survive with LayerNorm (or a pointer to related work) would help readers who train with standard LayerNorm stacks.
  4. [References] Related Work: NeuralGrok is cited as arXiv:2504.17243 without author list in the bibliography entry; complete the citation for consistency with other 2025–2026 preprints.
  5. [Discussion; Limitations] Practical note / Limitations: wall-clock overhead (~1.5x per epoch for full-batch SupCon) is discussed honestly; stating the epoch-ratio threshold for net wall-clock win (~0.67) once in the abstract or intro would set expectations earlier.

Circularity Check

0 steps flagged

No circularity: interventional empirical design with content-matched controls and a tested, falsifiable race prediction; nothing reduces by construction to its inputs.

full rationale

The paper is not a derivation that claims first-principles prediction from fitted constants or self-defined quantities. Its central causal claim is established by three content-matched SupCon conditions (true / sibling / random) that share identical loss form, strength, class-size distribution and geometry, plus a weight-norm-matched CE control that replays the exact norm trajectory; outcomes (22/30, 14/15, 0/20, 0/15) therefore isolate structural content rather than being forced by the auxiliary loss or the norm. Representation probes are measured independently and shown to precede generalization in all 95 runs. The subsequent “confirm the mechanism by prediction” step (Section 5) is ordinary hypothesis testing: the race account observed in the bimodal true-structure runs predicts that suppressing norm inflation will eliminate stalls while preserving speedup; three independent mitigations (anneal, norm-replay, constant clamp) are then run and produce the predicted monotone speedup and pooled stall reduction. No parameter is fitted to a subset and then re-labeled a prediction; no uniqueness theorem or ansatz is imported from the same author; the sole author cites only external work. The Limitations section itself flags the missing different-feature coherent prior, confirming that the feature-level slogan is an inference, not a circular redefinition. The entire chain is self-contained against external benchmarks (baseline, Grokfast, norm-matched) and does not reduce by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard modular-addition grokking setup choices, the interpretability ground truth that the generalizing solution is a Fourier multiplication circuit, and the modeling choice that SupCon positive-set content is a pure structure-injection device when geometry and strength are matched. Hyperparameters (λ, τ, clamp target) are free knobs that change quantitative speedups but not the qualitative true/sibling/random gradation. No new physical entity is postulated; the 'race' is a mechanistic account tested by prediction.

free parameters (4)
  • SupCon strength λ
    Hand-chosen grid {0.1, 0.3, 1.0}; acceleration and stall rates are dose-dependent on this knob, so quantitative speedups depend on it even though the qualitative structure gradation appears across λ.
  • SupCon temperature τ
    Fixed at 0.1 following standard SupCon practice; shapes the geometry of the auxiliary loss and is not ablated.
  • Norm-clamp target ||W||=45
    Chosen constant that holds the norm near the baseline equilibrium; median 8.6× speedup is reported at this specific target and is shown to grow as the held norm is lowered.
  • Weight decay / AdamW hyperparameters
    Standard grokking configuration (wd=1.0, η=1e-3, full batch) under which the delay robustly appears; the phenomenon and controls are conditioned on this training regime.
axioms (4)
  • domain assumption The generalizing solution for modular addition in this architecture is a Fourier multiplication circuit whose formation can be tracked by embedding Fourier concentration and related probes.
    Imported from Nanda et al. and related work; used throughout §3.4 and §4.5 to interpret representation timing as structure formation.
  • domain assumption Omitting LayerNorm keeps weight scale coupled to network function so that norm-matched controls remain interpretable.
    Stated in §3.1 citing Truong et al.; without it the weight-norm-matched control would be uninterpretable by the paper's own argument.
  • ad hoc to paper When loss form, strength, class-size distribution, and normalized-projection geometry are matched, differences between true/sibling/random SupCon conditions are attributable to structural content of the positive sets.
    Core identification assumption of the experimental design (§3.2–3.3); sibling and shuffled controls are built to support it.
  • domain assumption Modular addition with p=97, 30% train split, and a one-layer transformer is a faithful model system for studying the grokking delay.
    Canonical setup from Power/Nanda line of work; all causal claims are demonstrated only in this regime (Limitations).
invented entities (2)
  • Race between structure seeding (Channel 1) and norm-driven saturation (Channel 2) independent evidence
    purpose: Explains bimodal acceleration vs stall under true-structure SupCon and predicts that removing norm inflation yields reliable speedup.
    Introduced as a trajectory-level account in §4.4–5; partially independently tested by three norm-side-effect mitigations that behave as predicted, so not pure post-hoc labeling.
  • Structure-agnostic anti-saturation channel of the contrastive auxiliary loss no independent evidence
    purpose: Explains why both true and shuffled SupCon avoid logit-scale collapse that kills the norm-matched CE control.
    Inferred from confidence/logit-scale trajectories (§4.3); useful for mechanism separation but not independently measured outside this paper's runs.

pith-pipeline@v1.1.0-grok45 · 17783 in / 3556 out tokens · 41002 ms · 2026-07-11T19:59:43.537993+00:00 · methodology

0 comments
read the original abstract

Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ($(a+b) \bmod p$), (ii) a coherent-but-wrong sibling ($(a-b) \bmod p$), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher $p=1.3\times10^{-7}$). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to $2.75\times$), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median $8.6\times$ speedup (up to $22\times$ on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled across methods ($0/40$ vs $6/20$, $p=7.7\times10^{-4}$), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.

Figures

Figures reproduced from arXiv: 2607.04333 by Gunner Levi Howe.

Figure 1
Figure 1. Figure 1: Survival curves: fraction of seeds that have not yet reached 95% test accuracy, per [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representation-timing probes (medians across seeds, primary [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mediating variables (medians across seeds, primary [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Survival curves (fraction not yet generalized) for the mitigation and composition methods at [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Test-accuracy trajectories at the primary strength [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Epochs to generalization and delay (tgen − tfit) per condition at λ=1.0 (points: seeds; bars: medians; censored runs plotted at the 50,000-epoch budget). 10 1 10 0 Contrastive strength 10 4 Epochs to generalization True structure Wrong structure Shuffled structure Baseline [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Dose–response: epochs to generalization versus auxiliary-loss strength [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 15 linked inside Pith

  1. [1]

    Grokking: Gen- eralization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022. 12

    Pith/arXiv arXiv 2022

  2. [2]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. InInternational Conference on Learning Representations, 2023. arXiv:2301.05217

    Pith/arXiv arXiv 2023

  3. [3]

    Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    Andrey Gromov. Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Grokfast: Accelerated grokking by amplifying slow gradients

    Jaerin Lee, Bong Gyun Kang, Kihoon Kim, and Kyoung Mu Lee. Grokfast: Accelerated grokking by amplifying slow gradients. InAdvances in Neural Information Processing Systems,

  5. [5]

    Neuralgrok: Accelerate grokking by neural gradient transformation.arXiv preprint arXiv:2504.17243, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    The weight norm sets the grokking timescale: A causal delay law.arXiv preprint arXiv:2606.13753, 2026

    Xuan Khanh Truong, Hoang Viet Doan, Duc Trung Luu, and Thanh Duc Phan. The weight norm sets the grokking timescale: A causal delay law.arXiv preprint arXiv:2606.13753, 2026

    arXiv 2026

  7. [7]

    What does the weight norm control in grokking? logit-scale mediation under cross-entropy.arXiv preprint arXiv:2606.18465, 2026

    Xuan Khanh Truong. What does the weight norm control in grokking? logit-scale mediation under cross-entropy.arXiv preprint arXiv:2606.18465, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Radial suppression accelerates algorithmic generalization: A geometric analysis of delayed generalization.arXiv preprint arXiv:2606.32000, 2026

    Srijan Tiwari, Aditya Chauhan, and Manjot Singh. Radial suppression accelerates algorithmic generalization: A geometric analysis of delayed generalization.arXiv preprint arXiv:2606.32000, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Two speeds of learning: A representation-readout decomposition of grokking and double descent

    Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu, Yao-Yuan Yang, and SueYeon Chung. Two speeds of learning: A representation-readout decomposition of grokking and double descent. arXiv preprint arXiv:2605.27078, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Circuit synchronization precedes generalization: A causal precursor to grokking.arXiv preprint arXiv:2606.12966, 2026

    Achyuthan Sivasankar. Circuit synchronization precedes generalization: A causal precursor to grokking.arXiv preprint arXiv:2606.12966, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. InAdvances in Neural Information Processing Systems, volume 33, 2020. arXiv:2004.11362

    Pith/arXiv arXiv 2020

  12. [12]

    Towards understanding grokking: An effective theory of representation learning

    Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. InAdvances in Neural Information Processing Systems, 2022. arXiv:2205.10343

    Pith/arXiv arXiv 2022

  13. [13]

    Omnigrok: Grokking beyond algorithmic data

    Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. InInternational Conference on Learning Representations, 2023. arXiv:2210.01117

    Pith/arXiv arXiv 2023

  14. [14]

    Explaining grokking through circuit efficiency.arXiv preprint arXiv:2309.02390, 2023

    Vikrant Varma, Rohin Shah, Zachary Kenton, J´ anos Kram´ ar, and Ramana Kumar. Explaining grokking through circuit efficiency.arXiv preprint arXiv:2309.02390, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Beyond neural collapse: Task-intrinsic geometry governs neural representations in modular arithmetic.arXiv preprint arXiv:2606.08985, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Model capacity determines grokking through competing memorisation and generalisation speeds.arXiv preprint arXiv:2605.09724, 2026

    Yiding Song and Hanming Ye. Model capacity determines grokking through competing memorisation and generalisation speeds.arXiv preprint arXiv:2605.09724, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Similarity of neural network representations revisited.International Conference on Machine Learning, 2019

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited.International Conference on Machine Learning, 2019. arXiv:1905.00414. 13 A Hyperparameters and reproducibility Task (a+b) mod 97, tokens [a, b,=], 30% of 97 2 pairs train Model 1-layer transformer,d model=128, 4 heads,d mlp=512 (ReLU...

    Pith/arXiv arXiv 2019