A Two-Phase Perspective on Deep Learning Dynamics

Animik Ghosh; Robert de Mello Koch

REVIEW 4 major objections 5 minor 23 references

A Two-Phase Perspective on Deep Learning Dynamics

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that grokking, double descent, and the information bottleneck are the same two-phase learning process, and that mutual information I(T;X) is the progress measure that tracks it.

desk verdict A plausible two-phase synthesis of grokking, double descent, and information bottleneck, but the central alignment claim rests on under-specified single-run mutual information estimates. read the letter →

arxiv 2504.12700 v1 pith:GX6ME2OB submitted 2025-04-17 hep-th cond-mat.dis-nncs.LG

Robert de Mello Koch , Animik Ghosh This is my paper · ORCID

classification hep-thcond-mat.dis-nncs.LG

keywords grokkingdoubledescentinformationbottlenecktwo-phaselearningmutualcoarsegrainingrenormalizationgroupdeepdynamics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that learning in deep neural networks is not a single smooth process but unfolds in two phases: a rapid curve-fitting phase, in which the network memorizes training data and drives training error to zero, followed by a slower compression or coarse-graining phase in which the network discards input details irrelevant to the task and thereby acquires generalization. It claims that three well-known phenomena—grokking, double descent, and the information bottleneck—are not separate curiosities but the same two-phase dynamics seen from different angles. The paper supports this by showing in two transformer settings (modular division and multiplication in S5) that the timescales align: the moment test error starts to drop matches the onset of compression in the information plane. If true, a single progress measure—the mutual information I(T;X) between a hidden layer and the input—tracks learning, and the slow second phase is not a fundamental prerequisite but an incidental side effect that tailored algorithms could shorten.

What carries the argument

The central object is the information plane trajectory of a hidden layer, with coordinates (I(T;X), I(T;Y)) — mutual information between the layer's representation and the input, and between the representation and the labels. The paper also uses the circuit framework for ReLU/GeLU networks: linear-region partitions of input space, local complexity, and the Linear Mapping Number as complexity measures. The argument's load-bearing move is the empirical alignment of three timescales: end of fitting, onset of second descent, and onset of compression.

What would settle it

Train the same two transformer architectures on a task where grokking occurs, but compute mutual information with a known-accurate estimator (or with analytically tractable low-dimensional representations); if the decrease in I(T;X) does not begin at the same step as the second descent of test error, the claimed alignment fails. Alternatively, if a training algorithm that directly minimizes I(T;X) does not shorten the delay to generalization, the claim that phase two is incidental and compressible is undercut.

Watch

Extended reading notes

Core claim

The paper's central claim is that learning in a deep neural network proceeds through two distinct phases. Phase one is curve fitting: the loss is actively minimized, training accuracy rises to perfect, and I(T;Y) grows as the network absorbs task-relevant information; this is the ascending limb before the interpolation threshold. Phase two is coarse graining: training loss stays at zero while the network reorganizes its linear regions, reduces local complexity and the linear mapping number, and gradually decreases I(T;X) while keeping I(T;Y) roughly constant. The paper's empirical claim is that the timescales of grokking, double descent, and the information bottleneck coincide: the point where training accuracy reaches perfect marks the end of fitting; the onset of the second descent in test error and the rightmost point of the information-plane trajectory both occur at the same training step; and the end of generalization corresponds to the information-plane fixed point. Grokking is therefore the visible face of this two-phase structure, and I(T;X) is proposed as a natural process measure of learning.

Load-bearing premise

The whole alignment rests on the reliability of the reported mutual information estimates I(T;X) and I(T;Y); if those numbers are biased by the estimator or binning, the coincident timescales could be an artifact.

Editorial extensions

If this is right

Grokking, double descent, and the information bottleneck are manifestations of the same two-phase learning dynamics; observations in one setting can be transferred to the others.
Mutual information I(T;X) serves as a progress measure whose decrease during the second phase signals movement toward generalization, and whose flattening marks the end of learning.
Because the second phase is not explicitly optimized by standard training algorithms, it can be unnecessarily prolonged; algorithms that deliberately promote compression could accelerate generalization.
The renormalization-group analogy suggests the second phase is a principled form of forgetting: irrelevant details, like irrelevant parameters in RG flow, are discarded, and this forgetting is critical for generalization.
The three geometric mechanisms—alignment, disappearance, and merging of zero-locus lines—provide a starting point for a dynamical theory of the compression phase.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the timescale alignment holds across more architectures and datasets, one could predict grokking's onset from information-plane measurements alone, without waiting for test accuracy to rise.
The claim that compression is incidental suggests a testable recipe: modifying the loss to penalize I(T;X) directly, or adding explicit compression pressure, should shorten phase two; the paper does not run this experiment.
The renormalization-group analogy implies the existence of marginal and relevant 'circuit parameters' that survive coarse graining; identifying them in trained networks could connect compression to generalization bounds.
The layer-depth insensitivity of phase-one end hints that the fitting-to-compression transition is governed by the global task structure rather than by individual layer architecture.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the empirical claim that the onset of generalization (grokking), the second descent in double descent, and the turn toward compression in the information plane all happen at the same training time, in two different transformer settings. The two-phase picture itself is not new — the paper's own citations already contain it — but the alignment of all three time scales in the same runs is something I have not seen stated before. If it holds up, it is a useful unification and makes I(T;X) a plausible monitoring tool for early stopping. The paper also writes clearly and the RG discussion is honest about being analogical, building on prior work by the same group.

That said, the evidence for the central claim is thinner than the paper admits. Figures 3 and 4 are single runs, read by eye. No seeds, no error bars, no checkpoint schedule, and — critically — no description of the mutual information estimator, let alone binning or sample counts. Mutual information between high-dimensional activations and inputs is notoriously fragile; a biased estimate can shift the apparent compression onset by orders of magnitude. Since the abstract and Section 4.1 lean on this alignment as the main support for the two-phase hypothesis, the missing estimator details are a load-bearing gap, not a minor omission. I also think the treatment of double descent as a training-time phenomenon (test error during a single run) needs explicit justification, since the standard double descent curve is about model complexity, not training time. Those are interpretive and empirical issues, not fatal ones.

I disagree with any suggestion that the paper is circular: the alignment observation is new data, not fitted parameters. And the self-citations on RG are fine, since they are prior work and the paper does not pretend otherwise.

Bottom line: the hypothesis is interesting and worth taking seriously, but the empirical support currently does not carry the weight. The weaknesses are addressable with more seeds, an explicit estimator, error bars, and released code. I would send this to peer review rather than desk reject, because a serious referee could push it toward becoming a much stronger paper. If the authors can show the alignment survives across seeds and a described MI estimator, this could be a genuinely useful contribution. As it stands, I would not cite it in my own work yet; I would want the reproducibility details first.

Referee Report

4 major / 5 minor

Summary. The paper proposes that learning in deep neural networks proceeds in two phases: a fast curve-fitting phase followed by a slower compression or coarse-graining phase. It argues that three phenomena—grokking, double descent, and the information bottleneck—share a common temporal structure, and it presents numerical experiments on modular division modulo 96 and the multiplication table of S5 using a two-layer transformer. The experiments plot training and test accuracy, test error, and information-plane trajectories, and the paper claims that the end of the fitting phase, the onset of the second descent, and the right-most point of the information-plane trajectory all align. It then discusses this two-phase picture in terms of circuit complexity, a renormalization-group analogy, and three geometric mechanisms for reducing the number of linear regions.

Significance. If the time-scale alignment claim is correct, the paper would provide a unifying perspective on grokking, double descent, and the information bottleneck, and it would identify I(T;X) as a useful progress measure for deep learning. A strength of the paper is that it makes a concrete, falsifiable empirical claim and does not fit parameters to confirm a pre-chosen trend. However, the significance is currently conditional: the central alignment claim rests on under-specified mutual-information estimates from single runs, and without the estimator details, multiple seeds, and quantitative time-scale definitions, the claim cannot be independently checked or built upon. The theoretical RG analogy is exploratory, but the empirical part is the load-bearing component.

major comments (4)

[Section 3.1 and Figures 3-5] The mutual information estimator used for I(T;X) and I(T;Y) is never specified. Please state the estimator, the binning or partitioning scheme, the number of samples used for estimation, the network layers over which T is defined, and the checkpoint schedule. Without these details, the information-plane trajectories in Figures 3-5 cannot be reproduced, and the claimed alignment of the three time scales is not independently checkable.
[Section 3.2] The identification of the relevant time scales is made by visual inspection of single runs, with qualitative language such as "midway between 10^3 and 10^4 steps" and "around 10^5 steps." Please provide a quantitative definition of each time scale (for example, a threshold or change-point criterion), error bars over multiple seeds, and the number of seeds used. The central claim is that three time scales align, which requires a quantitative comparison rather than visual coincidence.
[Section 4.2] The claim that I(T;X) serves as a progress measure of learning depends on the decrease of I(T;X) being a real dynamical feature rather than an artifact of a biased or variance-heavy mutual-information estimator. Because the estimator is not specified in Section 3.1, this part of the two-phase interpretation is currently unsupported; please provide estimator validation, for example by comparing against a known baseline or by reporting the estimates as a function of sample size.
[Section 2.3] The two-phase structure of the information-plane trajectory is introduced as established fact, citing Refs. [16,17], and is then used to label the observed trajectories in Figures 3-5. The paper should clearly separate this inherited assumption from the new empirical evidence, which concerns the alignment of time scales rather than the existence of the two phases. A precise operational definition of the "onset of compression" or "right-most point" would help make the test falsifiable.

minor comments (5)

[Section 4.2] The paper alternates between "progress measure" and "process measure"; please use one term consistently and define it.
[Section 3.1] The learning rate is written as "10 −3" with a stray space; please format as 10^-3.
[Section 3.1] "division modulo 96" should be phrased as "modular division with modulus 96" for clarity.
[Section 4.4] The reference to "Figure 2 of [6]" is not self-contained; either reproduce the figure or describe the relevant observations in the text.
[General] No code or data availability statement is included. Making the experimental code and checkpoints available would substantially improve reproducibility, especially given the missing estimator details.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the timescale alignment is a direct observational claim, and the self-cited RG references are peripheral.

full rationale

The paper's central claim is an empirical alignment of three measured timescales: the step at which training accuracy saturates, the onset of the second descent in the test-error curve, and the right-most point of the mutual-information trajectory. These are read off from Figures 3-5, not obtained by fitting parameters and then renaming them as predictions. The two-phase framing is inherited from the information-bottleneck literature, but the specific numerical agreement is not forced by that literature: the right-most point of the MI plane could in principle occur at a different training step than zero training error. The only self-citations are references [21-23] used for the speculative renormalization-group analogy in Section 4.3; that analogy is not load-bearing for the empirical alignment claim. No uniqueness theorem is invoked, and no fitted quantity is relabeled as a prediction. The main weakness is that the MI estimator, binning, and number of seeds are not specified, so the alignment is hard to reproduce independently; this is a correctness and reproducibility concern, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities and fits no numerical constants to data. Its central claims rest on four unproven assumptions: the reliability of unspecified MI estimates, the identification of training-time test error dynamics with double descent, the transfer of ReLU circuit concepts to GeLU networks, and the relevance of the RG analogy. These assumptions are load-bearing for the empirical alignment and for the proposed compression mechanisms.

assumptions (4)

domain assumption The mutual information between high-dimensional hidden activations and inputs can be estimated reliably from the finite training sample using an unstated estimator.
Figures 3 through 5 plot I(T;X) and I(T;Y) without specifying the estimation method; finite-sample MI estimates in continuous high dimensions are fragile, so the compression onset and fixed point readings depend on this assumption.
domain assumption A decrease in test error during a single training run after an interpolation hump is the same phenomenon as double descent, which is classically defined across model complexity.
Section 2.2 reinterprets the double descent 'time scale' as the peak of the U-curve during training and Section 3.2 reads the second descent from training-time test error; this identification is an interpretive leap.
domain assumption The GeLU-based transformer still fits the ReLU circuit framework (local complexity, linear mapping number) used to motivate compression.
Section 3.1 acknowledges GeLU has no sharp active/inactive boundary and asserts the circuit framework remains valid, but no evidence is given that local complexity or linear mapping number is meaningful for this architecture.
ad hoc to paper The renormalization group block-spin analogy is a useful model for the second learning phase.
Section 4.3 draws the RG analogy to justify calling the second phase 'coarse graining'; it is speculative, not tested, and relies on prior self-cited work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Two-Phase Perspective on Deep Learning Dynamics." pith.science (2026). https://pith.science/paper/GX6ME2OB

@misc{pith2026250412700,
  author       = {Pith},
  title        = {Pith review of: A Two-Phase Perspective on Deep Learning Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GX6ME2OB}},
  note         = {Machine review of arXiv:2504.12700}
}

read the original abstract

We propose that learning in deep neural networks proceeds in two phases: a rapid curve fitting phase followed by a slower compression or coarse graining phase. This view is supported by the shared temporal structure of three phenomena: grokking, double descent and the information bottleneck, all of which exhibit a delayed onset of generalization well after training error reaches zero. We empirically show that the associated timescales align in two rather different settings. Mutual information between hidden layers and input data emerges as a natural progress measure, complementing circuit-based metrics such as local complexity and the linear mapping number. We argue that the second phase is not actively optimized by standard training algorithms and may be unnecessarily prolonged. Drawing on an analogy with the renormalization group, we suggest that this compression phase reflects a principled form of forgetting, critical for generalization.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

[1]

Grokking: Generalization beyond overfitting on small algorithmic datasets,

Power, A., Burda, Y., Edwards, H., Babuschkin, I. and Misra, V., 2022. “Grokking: Generalization beyond overfitting on small algorithmic datasets,” arXiv preprint arXiv:2201.02177. – 14 –

arXiv 2022
[2]

Omnigrok: Grokking beyond algorithmic data,

Liu, Z., Michaud, E.J. and Tegmark, M., 2022. “Omnigrok: Grokking beyond algorithmic data,” arXiv preprint arXiv:2210.01117

arXiv 2022
[3]

Benign overfitting and grokking in relu networks for xor cluster data,

Xu, Z., Wang, Y., Frei, S., Vardi, G. and Hu, W., 2023. “Benign overfitting and grokking in relu networks for xor cluster data,” arXiv preprint arXiv:2310.02541

arXiv 2023
[4]

Deep networks always grok and here is why,

Humayun, A.I., Balestriero, R. and Baraniuk, R., 2024. “Deep networks always grok and here is why,” arXiv preprint arXiv:2402.15555

arXiv 2024
[5]

Explaining grokking through circuit efficiency,

Varma, V., Shah, R., Kenton, Z., Kram´ ar, J. and Kumar, R., 2023. “Explaining grokking through circuit efficiency,” arXiv preprint arXiv:2309.02390

arXiv 2023
[6]

Grokking and the Geometry of Circuit Formation,

Humayun, A.I., Balestriero, R. and Baraniuk, R., “Grokking and the Geometry of Circuit Formation,” In ICML 2024 Workshop on Mechanistic Interpretability

work page 2024
[7]

Why do you grok? a theoretical analysis of grokking modular addition,

Mohamadi, M.A., Li, Z., Wu, L. and Sutherland, D.J., 2024. “Why do you grok? a theoretical analysis of grokking modular addition,” arXiv preprint arXiv:2407.12332

arXiv 2024
[8]

Progress measures for grokking via mechanistic interpretability,

Nanda, N., Chan, L., Lieberum, T., Smith, J. and Steinhardt, J., 2023, “Progress measures for grokking via mechanistic interpretability,” arXiv preprint arXiv:2301.05217

arXiv 2023

Show all 23 references

[9]

Zoom in: An introduction to circuits,

Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020, “Zoom in: An introduction to circuits,” Distill, 5(3), pp.e00024-001

2020
[10]

Hidden progress in deep learning: Sgd learns parities near the computational limit,

Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E. and Zhang, C., 2022. “Hidden progress in deep learning: Sgd learns parities near the computational limit,” Advances in Neural Information Processing Systems, 35, pp.21750-21764

2022
[11]

Grokking as compression: A nonlinear complexity perspective,

Liu, Z., Zhong, Z. and Tegmark, M., 2023. “Grokking as compression: A nonlinear complexity perspective,” arXiv preprint arXiv:2310.05918

2023 arXiv
[12]

Deep double descent: Where bigger models and more data hurt,

Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B. and Sutskever, I., 2021, “Deep double descent: Where bigger models and more data hurt,” Journal of Statistical Mechanics: Theory and Experiment, 2021(12), p.124003

2021
[13]

Double trouble in double descent: Bias and variance (s) in the lazy regime,

d’Ascoli, S., Refinetti, M., Biroli, G. and Krzakala, F., 2020, November, “Double trouble in double descent: Bias and variance (s) in the lazy regime,” In International Conference on Machine Learning (pp. 2280-2290). PMLR

2020
[14]

A brief prehistory of double descent,

Loog, M., Viering, T., Mey, A., Krijthe, J.H. and Tax, D.M., 2020, “A brief prehistory of double descent,” Proceedings of the National Academy of Sciences, 117(20), pp.10625-10626

2020
[15]

The information bottleneck method,

Tishby, N., Pereira, F.C. and Bialek, W., 2000, “The information bottleneck method,” arXiv preprint physics/0004057

2000 arXiv
[16]

Deep learning and the information bottleneck principle,

Tishby, N. and Zaslavsky, N., 2015, April, “Deep learning and the information bottleneck principle,” In 2015 ieee information theory workshop (itw) (pp. 1-5). Ieee

2015
[17]

Opening the black box of deep neural networks via information,

Shwartz-Ziv, R. and Tishby, N., 2017, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.00810

2017 arXiv
[18]

The renormalization group: Critical phenomena and the Kondo problem,

Wilson, K.G., 1975, “The renormalization group: Critical phenomena and the Kondo problem,” Reviews of modern physics, 47(4), p.773

1975
[19]

Mehta, and D.J

P. Mehta, and D.J. Schwab, 2014. “An exact mapping between the variational renormalization group and deep learning. arXiv preprint arXiv:1410.3831

2014 arXiv
[20]

Mutual information, neural networks and the renormalization group

M. Koch-Janusz and Z. Ringel, “Mutual information, neural networks and the renormalization group.” Nature Physics 14, no. 6 (2018): 578-582. – 15 –

2018
[21]

Is deep learning a renormalization group flow?

E. de Mello Koch, R. de Mello Koch and L. Cheng, 2020, “Is deep learning a renormalization group flow?” IEEE Access, 8, pp.106487-106505

2020
[22]

Short-sighted deep learning,

E. de Mello Koch, A. de Mello Koch, N. Kastanos, and L. Cheng, 2020, “Short-sighted deep learning,” Physical Review E, 102(1), p.013307

2020
[23]

Why unsupervised deep networks generalize,

A. de Mello Koch, E. de Mello Koch and R. de Mello Koch, 2020, “Why unsupervised deep networks generalize,” arXiv preprint arXiv:2012.03531. – 16 –

2020 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Grokking: Generalization beyond overfitting on small algorithmic datasets,

Power, A., Burda, Y., Edwards, H., Babuschkin, I. and Misra, V., 2022. “Grokking: Generalization beyond overfitting on small algorithmic datasets,” arXiv preprint arXiv:2201.02177. – 14 –

arXiv 2022

[2] [2]

Omnigrok: Grokking beyond algorithmic data,

Liu, Z., Michaud, E.J. and Tegmark, M., 2022. “Omnigrok: Grokking beyond algorithmic data,” arXiv preprint arXiv:2210.01117

arXiv 2022

[3] [3]

Benign overfitting and grokking in relu networks for xor cluster data,

Xu, Z., Wang, Y., Frei, S., Vardi, G. and Hu, W., 2023. “Benign overfitting and grokking in relu networks for xor cluster data,” arXiv preprint arXiv:2310.02541

arXiv 2023

[4] [4]

Deep networks always grok and here is why,

Humayun, A.I., Balestriero, R. and Baraniuk, R., 2024. “Deep networks always grok and here is why,” arXiv preprint arXiv:2402.15555

arXiv 2024

[5] [5]

Explaining grokking through circuit efficiency,

Varma, V., Shah, R., Kenton, Z., Kram´ ar, J. and Kumar, R., 2023. “Explaining grokking through circuit efficiency,” arXiv preprint arXiv:2309.02390

arXiv 2023

[6] [6]

Grokking and the Geometry of Circuit Formation,

Humayun, A.I., Balestriero, R. and Baraniuk, R., “Grokking and the Geometry of Circuit Formation,” In ICML 2024 Workshop on Mechanistic Interpretability

work page 2024

[7] [7]

Why do you grok? a theoretical analysis of grokking modular addition,

Mohamadi, M.A., Li, Z., Wu, L. and Sutherland, D.J., 2024. “Why do you grok? a theoretical analysis of grokking modular addition,” arXiv preprint arXiv:2407.12332

arXiv 2024

[8] [8]

Progress measures for grokking via mechanistic interpretability,

Nanda, N., Chan, L., Lieberum, T., Smith, J. and Steinhardt, J., 2023, “Progress measures for grokking via mechanistic interpretability,” arXiv preprint arXiv:2301.05217

arXiv 2023