Pith. sign in

REVIEW 2 major objections 2 minor 2 cited by

Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent

T0 review · 2 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fluctuations in the relative speeds of representation learning and readout calibration explain grokking and epoch-wise double descent.

desk verdict The representation-readout split organizes grokking and double descent under relative speeds but the metrics may not isolate the processes cleanly enough to support the attribution. read the letter →

arxiv 2605.27078 v2 pith:IXQU5PMA submitted 2026-05-26 cs.LG cs.AI

Chi-Ning Chou , Oscar Uzdelewicz , Neng-Chun Chiu , Yao-Yuan Yang , SueYeon Chung This is my paper · ORCID
classification cs.LGcs.AI
keywords grokkingdoubledescentrepresentationlearningreadoutcalibrationgeneralizationdynamicsneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper decomposes neural network training into representation learning inside the encoder and readout calibration in the final classifier. It argues that both processes run throughout training and that changes in which one moves faster produce the delayed test improvement of grokking and the non-monotonic test curves of double descent. In typical grokking the readout becomes overly tuned to training examples while representations continue to improve slowly. The same decomposition shows that some reported cases of delayed or non-monotonic generalization are artifacts of non-standard training choices that degrade representations or misalign the readout. The framework is presented as a general diagnostic tool that applies across tasks and architectures.

What carries the argument

The representation-readout decomposition, which tracks representation quality with representational geometry and neural tangent kernels while tracking readout quality with linear probing.

What would settle it

Track representation quality and readout accuracy on held-out data throughout training on a grokking task; the central claim would be falsified if the two quantities do not show independent speed changes that align with the observed delay in test performance.

Watch

Extended reading notes

Core claim

Both representation learning and readout calibration are active throughout training, with the fluctuations of their relative speed giving rise to seemingly anomalous generalization dynamics. Applying the representation-readout decomposition to grokking across a wide range of tasks and architectures, we find that the readout is train-biased before grokking onset, and representation learning is gradual but not absent, contrary to the lazy-to-rich account. The framework further provides diagnostic signatures distinguishing spurious from genuine generalization: in a previously reported MNIST grokking example and an epoch-wise double descent example, apparent delayed or non-monotone generalizatio

Load-bearing premise

That representational geometry, neural tangent kernels, and linear probing cleanly separate representation learning from readout calibration without the measurements themselves being altered by the ongoing training dynamics.

Editorial extensions

If this is right

  • The readout is train-biased before grokking onset.
  • Representation learning is gradual but not absent during the period before grokking.
  • Non-standard training recipes can induce representation degradation or readout misalignment that produces spurious delayed generalization.
  • Genuine generalization is diagnosed when representation and readout metrics improve together rather than diverge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Interventions that selectively speed up representation learning while slowing readout calibration could shorten the grokking delay.
  • The same decomposition may clarify other training phenomena in which train and test curves diverge, such as certain forms of overfitting or forgetting.
  • Simple synthetic models could be constructed in which the two speeds are controlled independently to reproduce grokking without large networks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript claims that grokking and epoch-wise double descent arise from fluctuations in the relative speeds of two processes: representation learning in the encoder and readout calibration in the final classifier. Using representational geometry, neural tangent kernels, and linear probing across a range of tasks and architectures, it shows both processes remain active throughout training; the readout is train-biased before grokking onset while representation learning proceeds gradually (contrary to lazy-to-rich accounts). The framework also supplies diagnostic signatures that attribute some previously reported instances of these phenomena to representation degradation and readout misalignment induced by non-standard training recipes.

Significance. If the decomposition is shown to cleanly isolate the two processes, the work supplies a task-agnostic, top-down framework for diagnosing generalization dynamics that moves beyond task-specific accounts. The identification of spurious effects traceable to training choices offers a practical diagnostic contribution, and the breadth of empirical coverage across architectures strengthens the case for generality. These elements would be useful for interpretability research aimed at revealing underlying algorithms.

major comments (2)
  1. [Abstract and measurement sections] Abstract and measurement sections: The attribution of grokking and double descent to relative speeds rests on the claim that representational geometry, NTK spectra, and linear probing cleanly separate representation learning from readout calibration. The manuscript itself reports that non-standard recipes induce representation degradation and readout misalignment, which implies the same metrics can be sensitive to training dynamics; explicit controls or invariance tests are required to establish that the separation is not distorted by the overlapping timescales under study.
  2. [Grokking results] Grokking results: The statement that representation learning is gradual but not absent (contrary to lazy-to-rich) is load-bearing for the framework. Quantitative comparison of the rate of representational change versus readout change, together with ablation of the measurement tools, is needed to confirm the observed gradual shifts are not artifacts of metric contamination.
minor comments (2)
  1. [Methods] Clarify the precise definitions and hyper-parameters used for the representational geometry and NTK computations so that the reported signatures can be reproduced exactly.
  2. [Figures] Include error bars or statistical significance markers on all plots that display relative speeds or metric trajectories.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address the two major comments below and agree that additional controls and quantitative analyses will strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract and measurement sections] Abstract and measurement sections: The attribution of grokking and double descent to relative speeds rests on the claim that representational geometry, NTK spectra, and linear probing cleanly separate representation learning from readout calibration. The manuscript itself reports that non-standard recipes induce representation degradation and readout misalignment, which implies the same metrics can be sensitive to training dynamics; explicit controls or invariance tests are required to establish that the separation is not distorted by the overlapping timescales under study.

    Authors: We agree that the potential sensitivity of the metrics warrants explicit validation. The paper already leverages this sensitivity as a diagnostic for spurious grokking and double descent under non-standard recipes. In the revision we will add invariance tests that apply the same metrics under controlled variations of training hyperparameters (while keeping the core recipe standard) to confirm that the relative-speed interpretation and separation remain stable when timescales overlap. revision: yes

  2. Referee: [Grokking results] Grokking results: The statement that representation learning is gradual but not absent (contrary to lazy-to-rich) is load-bearing for the framework. Quantitative comparison of the rate of representational change versus readout change, together with ablation of the measurement tools, is needed to confirm the observed gradual shifts are not artifacts of metric contamination.

    Authors: The current results already demonstrate continued representational change via multiple independent metrics (geometry, NTK, probing) that persist after readout saturation. To directly address possible contamination, the revised manuscript will include (i) explicit quantitative rate comparisons (e.g., epoch-wise slopes or integrated change) between representation and readout quantities and (ii) ablation experiments that systematically disable or replace individual measurement tools to verify that the gradual-representation conclusion is robust. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical application of existing metrics

full rationale

The paper presents an empirical analysis that applies established tools (representational geometry, NTK spectra, linear probing) to training trajectories across tasks. No derivation chain reduces reported signatures to quantities defined by parameters fitted within the paper, nor does any central claim rest on a self-citation whose content is itself unverified or defined by the target result. The framework is self-contained against external benchmarks and falsifiable via the same measurement families on held-out runs; the reader's assessment of score 2.0 is consistent with minor non-load-bearing citations at most.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The framework rests on the domain assumption that the two processes can be separated and tracked with the listed tools; no free parameters or new invented entities are introduced in the abstract.

assumptions (1)
  • domain assumption Representation learning in the encoder and readout calibration in the final classifier can be meaningfully isolated and measured independently throughout training using representational geometry, neural tangent kernels, and linear probing.
    This separation is the load-bearing premise that allows the relative-speed explanation of grokking and double descent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent." pith.science (2026). https://pith.science/paper/IXQU5PMA

@misc{pith2026260527078,
  author       = {Pith},
  title        = {Pith review of: Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IXQU5PMA}},
  note         = {Machine review of arXiv:2605.27078}
}
read the original abstract

Training loss and accuracy are the standard signals used to monitor generalization during deep neural network training. Two well-documented phenomena complicate this picture: in grokking, train loss falls rapidly while test performance improves abruptly only after a long delay; in epoch-wise double descent, train loss decreases monotonically while test loss or error rises and falls. Existing accounts are often task-specific, and a task-agnostic analysis framework for diagnosing and explaining these phenomena across realistic tasks and architectures is missing. We address this challenge by analyzing two competing processes that underlie learning dynamics: representation learning in the encoder and readout calibration in the final classifier. Using tools from representational geometry, neural tangent kernels, and linear probing, we show that both processes are active throughout training, with the fluctuations of their relative speed giving rise to seemingly anomalous generalization dynamics. Applying the representation-readout decomposition to grokking across a wide range of tasks and architectures, we find that the readout is train-biased before grokking onset, and representation learning is gradual but not absent, contrary to the lazy-to-rich account. The framework further provides diagnostic signatures distinguishing spurious from genuine generalization: in a previously reported MNIST grokking example and an epoch-wise double descent example, apparent delayed or non-monotone generalization is shown to arise from representation degradation and readout misalignment induced by non-standard training recipes. Together, these results establish the representation-readout decomposition as a top-down framework for understanding learning dynamics and revealing underlying algorithms for interpretability research.

Figures

Figures reproduced from arXiv: 2605.27078 by the authors.

Figure 1
Figure 1. Overview of our framework. a. A neural network can be conceptually decomposed into two parts: a feature learning part (orange) that encourages generalization, and a readout calibration part (yellow) that is prone to overfitting (see Section 2 for details). b. We propose that grokking (top) and epoch￾wise double descent (bottom) can be explained by the decoupling between the dynamics of representation learning and re… view at source ↗
Figure 2
Figure 2. Task-agnostic diagnostic measures for the representation-readout decomposition. a, Summary of the four measures, their interpretations, and their diagnostic usage. b, The critical dimension Ncrit is defined as the median threshold of the random-projection separability probability p(Nproj); a smaller Ncrit indicates more robustly untangled manifolds. c, Four task-relevant geometric measures characterize the underlyin… view at source ↗
Figure 3
Figure 3. Representation learning is slow but not absent before grokking. a, Using critical dimension Ncrit to compare the candidate mechanism of lazy-to-rich transition (bottom left) and slower￾representation-learning (bottom right). b, A grokking example. Top: model’s accuracy curves. Bottom: Ncrit decreases throughout the training. c, Results across tasks and architectures (averaged over 3 seeds, all results in Appendix D.… view at source ↗
Figures from the paper (32 more)
Figure 4
Figure 4. Figure 4: Readout learning direction is train-biased before grokking. a, NTK alignment difference (train minus test) throughout training for permutation composition on MLP (seed=0). The gap peaks near training saturation and decays to zero as grokking completes. The inset shows …
Figure 5
Figure 5. Figure 5: Different geometric mechanisms of repre￾sentation learning in MLPs and Transformers. A common mechanistic interpretability ap￾proach is to determine whether a network solves a task via a specific algorithm by identifying un￾derlying algorithmic components (Nanda et al.…
Figure 6
Figure 6. Figure 6: Diagnosing spurious learning in an MNIST grokking example. See [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Diagnosing spurious learning in a double descent example. See [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Three regimes of the two-ellipsoid toy. Both classes share the same covariance Σ = [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Random 1-D projections of the two-ellipsoid toy (one column per projection direction, [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Polar plot of the Bayes-optimal 1-D projection accuracy Acc( [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Summary sweep over σ⊥. Left: best-case separability (optimal LDA and LinearSVC) is flat. Middle: mean random-projection accuracy decreases monotonically. Right: Critical dimension increases monotonically, tracking the degradation in average-case separability. A.2.3 Cr…
Figure 12
Figure 12. Figure 12: Critical dimension tracks the degree of feature learning. 2-layer non-linear networks [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Intuition for GLUE geometric measures. Each measure is validated on synthetic Gaussian manifolds (N = 1000, P = 2, M = 200) from (Chou et al., 2025a). Each column sweeps one ground-truth parameter while holding the others fixed. Top: estimated GLUE measure vs. ground-…
Figure 14
Figure 14. Figure 14: A toy 2D example of high NTK alignment [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: A toy 2D example of low NTK alignment. The class means now nearly coincide (separated by only 0.2 along x1), while the variance along x2 is very large (= 10) and shared by both classes. Every pairwise inner product Kij = ϕ ⊤ i ϕj is dominated by the x2 component, whic…
Figure 16
Figure 16. Figure 16: Grokking examples of ModAdd MLP [PITH_FULL_IMAGE:figures/full_fig_p039_16.png]
Figure 17
Figure 17. Figure 17: Grokking examples of ModAdd Transformer. [PITH_FULL_IMAGE:figures/full_fig_p039_17.png]
Figure 18
Figure 18. Figure 18: Grokking examples of PermComp MLP [PITH_FULL_IMAGE:figures/full_fig_p040_18.png]
Figure 19
Figure 19. Figure 19: Grokking examples of PermComp Transformer. [PITH_FULL_IMAGE:figures/full_fig_p040_19.png]
Figure 20
Figure 20. Figure 20: Grokking examples of SparseParity MLP [PITH_FULL_IMAGE:figures/full_fig_p041_20.png]
Figure 21
Figure 21. Figure 21: Grokking examples of SparseParity Transformer. [PITH_FULL_IMAGE:figures/full_fig_p041_21.png]
Figure 22
Figure 22. Figure 22: Non-grokking examples of ModAdd MLP [PITH_FULL_IMAGE:figures/full_fig_p042_22.png]
Figure 23
Figure 23. Figure 23: Non-grokking examples of ModAdd Transformer. [PITH_FULL_IMAGE:figures/full_fig_p042_23.png]
Figure 24
Figure 24. Figure 24: Non-grokking examples of PermComp MLP [PITH_FULL_IMAGE:figures/full_fig_p043_24.png]
Figure 25
Figure 25. Figure 25: Non-grokking examples of PermComp Transformer. [PITH_FULL_IMAGE:figures/full_fig_p043_25.png]
Figure 26
Figure 26. Figure 26: Non-grokking examples of SparseParity MLP. [PITH_FULL_IMAGE:figures/full_fig_p044_26.png]
Figure 27
Figure 27. Figure 27: Non-grokking examples of SparseParity Transformer. [PITH_FULL_IMAGE:figures/full_fig_p044_27.png]
Figure 28
Figure 28. Figure 28: Epoch-wise double descent through label noises, 0% label noise. [PITH_FULL_IMAGE:figures/full_fig_p045_28.png]
Figure 29
Figure 29. Figure 29: Epoch-wise double descent through label noises, 10% label noise. [PITH_FULL_IMAGE:figures/full_fig_p045_29.png]
Figure 30
Figure 30. Figure 30: Epoch-wise double descent through label noises, 20% label noise. [PITH_FULL_IMAGE:figures/full_fig_p046_30.png]
Figure 31
Figure 31. Figure 31: Pairwise critical dimension (Ncrit) matrices for the ModAdd task. Top: MLP. Bottom: Transformer. Each panel shows a 113 × 113 matrix (class a vs. class b). Rows correspond to grokking (top) and non-grokking (bottom) conditions; columns correspond to four representativ…
Figure 32
Figure 32. Figure 32: Pairwise critical dimension matrices for the PermComp task. Top: MLP. Bottom: [PITH_FULL_IMAGE:figures/full_fig_p049_32.png]
Figure 33
Figure 33. Figure 33: Transfer linear probe accuracy. Top: ModAdd subtraction task (( [PITH_FULL_IMAGE:figures/full_fig_p050_33.png]
Figure 34
Figure 34. Figure 34: Consistency between train and test curves across different values of the laziness param [PITH_FULL_IMAGE:figures/full_fig_p051_34.png]
Figure 35
Figure 35. Figure 35: Consistency between the test and train curve across tasks, architectures, and training [PITH_FULL_IMAGE:figures/full_fig_p053_35.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Structure-Specific Representational Priors Causally Control the Grokking Delay

    cs.LG 2026-07 conditional novelty 6.5 of 10

    The grokking delay is causally the time to form the right feature-level representational structure, not a fixed optimization constant or a pure weight-norm effect.

  2. Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Muon-trained transformers grok modular addition and then lose generalization because the hidden representation and the output readout drift apart; freezing the readout and embeddings after grokking removes the collapse.

Reference graph

Works this paper leans on

4 extracted references · cited by 2 Pith papers

  1. [1]

    Data augmentation consists of random crops (padding 4) and random horizontal flips; no augmentation is applied at test time

    Training runs for 4,000 epochs. Data augmentation consists of random crops (padding 4) and random horizontal flips; no augmentation is applied at test time. Checkpoints are saved at 200 log-uniformly spaced epochs. Table 5: Hyperparameters for the double descent task. ResNet18 Optimizer Adam (β1, β2) (0.9,0.999) Learning rate 1×10 −4 (fixed) Weight decay ...

    2023

  2. [2]

    Clauw et al

    proposed weight sparsity as a progress measure, observing that the sparsification of weights contributing to the network output co-occurs with the onset of grokking. Clauw et al. (Clauw et al.,

  3. [3]

    Our framework differs from these approaches primarily in interpretability

    applied O-Information—an entropy-based measure of higher-order statistical dependencies— to model activations, identifying training phases that correspond to the qualitative behavior of the train and test loss curves during grokking. Our framework differs from these approaches primarily in interpretability. While the measures described above are effective...

    2023

  4. [4]

    and GLUE theory (see Appendix C.5), offering a robust lens that captures aspects of the transition to generalization that weight-based analyses may overlook. C.5 Previous applications of GLUE framework The Geometry Linked to Untangling Efficiency (GLUE) theory (Chou et al., 2025a), which serves as an extension of earlier work that developed Manifold Capac...

    2018

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.