Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that L2 regularization drives deep networks through a series of first-order accuracy transitions, each marked by a sudden change in scalar (Ricci) curvature, and that grokking is the hysteresis of such a transition.

desk verdict Worth a read for the curvature diagnostic, but the grokking-hysteresis claim is overreach: the experiment shows initialization dependence at fixed beta, not a hysteresis loop. read the letter →

arxiv 2505.06597 v2 pith:PZ4LR554 submitted 2025-05-10 cs.LG cond-mat.dis-nncond-mat.stat-mechphysics.data-an

Ibrahim Talha Ersoy , Karoline Wiesner This is my paper · ORCID
classification cs.LGcond-mat.dis-nncond-mat.stat-mechphysics.data-an MSC 68T0753B2082B26
keywords L2regularizationphasetransitionsdeepneuralnetworksgrokkingRiccicurvatureerrorlandscapehysteresisonsetoflearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the accuracy of an L2-regularized deep network does not degrade smoothly as regularization strength grows: it jumps between distinct accuracy phases, and those jumps are first-order phase transitions. The authors trace the mechanism to the local geometry of the error landscape, specifically to sudden changes in scalar (Ricci) curvature at the model's location in parameter space. They argue that this geometric picture explains the onset-of-learning, predicts additional transitions when data complexity rises, and accounts for grokking as hysteresis: the model trapped in a lower-accuracy basin. If the picture holds, curvature measurements offer a probe of what a network has learned without invoking high-level notions like memorization or compression.

What carries the argument

The central object is the scalar (Ricci) curvature $R$ of the error surface, viewed as a $d$-dimensional submanifold embedded in the $(d+1)$-dimensional space of parameters plus error. For mean-squared error the paper derives $R = \frac{1}{\|\nabla F\|}(\operatorname{tr}(H)^2 - \operatorname{tr}(H^2)) + \frac{2}{\|\nabla F\|^2}\nabla l^T (H^2 - \operatorname{tr}(H)H)\nabla l$, involving only the error gradient $\nabla l$ and Hessian $H$. A change-point in $R$ as $\beta$ varies is taken as the geometric signature of crossing an accuracy-basin boundary; its stability in practice is why the authors use the Ricci scalar instead of the Gauss-Kronecker curvature.

What would settle it

Train the same architecture on a family of datasets whose complexity is controlled (for instance, Gaussian inputs with output dimension 1, 2, 4, 8) and sweep $\beta$ for each; if the number of curvature change-points and accuracy jumps does not increase with output dimension, or if an accuracy jump occurs with no simultaneous curvature change-point, then the geometric explanation fails.

Watch

Extended reading notes

Core claim

The paper claims that increasing L2 regularization strength $\beta$ pushes a trained network through a sequence of accuracy basins, each separated by a first-order phase transition. The jump occurs when the global minimum of the regularized loss shifts past the boundary of an error-landscape basin, forcing the model to a different region of parameter space; the transition is registered as a simultaneous jump in error, in distance from the origin, and in the scalar Ricci curvature computed at the model point. The same mechanism, the authors argue, explains grokking: a model initialized in a lower-accuracy basin can remain trapped there for many epochs because the geometry of the error surface presents a barrier, until it escapes—a hysteresis effect rather than a change in memorization or compression. They support this with experiments on Gaussian toy data and MNIST, and with a deliberately induced grokking-like delay in a small network.

Load-bearing premise

The claim that the number of transitions grows with data complexity rests on the assumption that more complex data creates more distinct accuracy basins in the error landscape, which the paper infers from a small set of experiments rather than derives.

Editorial extensions

If this is right

  • Sweeping $\beta$ and recording curvature change-points gives a practical way to map accuracy phases, including phases beyond the onset-of-learning.
  • Higher data complexity should produce more accuracy basins and therefore more transitions; the paper observes this direction from one-dimensional Gaussian output to two-dimensional output and to MNIST classification.
  • Grokking should appear naturally in sufficiently complex tasks even with random initialization, because the probability of landing in a low-accuracy basin increases.
  • The same geometric account should apply to regularizers other than L2: the KL-regularized VAE-like model shows transitions with the same signature, since the error-landscape geometry is shared.
  • Single-hidden-layer networks should show continuous (second-order-like) transitions rather than jumps, matching the paper's observation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if curvature change-points mark basin boundaries, one could test basin nesting by initializing models at different $\beta$-grown parameter positions and measuring escape times; the hysteresis picture predicts escape time should diverge near transition points.
  • The supplementary relation between the induced metric and the Fisher information metric means the curvature probe could connect accuracy phases to information-geometric accounts of learning, a link the paper does not develop.
  • Testable extension: on a dataset with tunable input or output dimension, count curvature change-points as a function of dimension; monotonic growth would support the paper's data-complexity claim, while saturation or non-monotonicity would bound it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript proposes that L2-regularized deep neural networks undergo first-order phase transitions between distinct accuracy regimes, that these transitions are detectable as change-points in the Ricci scalar curvature of the error surface, and that the number of such transitions increases with data complexity. The authors report numerical experiments on Gaussian regression tasks (1D and 2D outputs), MNIST classification, a one-hidden-layer variant, and a VAE-like model with KL regularization, and they offer an explanation of grokking as a hysteresis effect caused by a model being trapped in a lower-accuracy basin. The paper includes a first-principles derivation of a Ricci-curvature formula for error surfaces and shows qualitative agreement between jumps in the error, jumps in the distance from the origin, and discontinuities in the curvature.

Significance. The paper's main contribution would be a geometric, parameter-free diagnostic for accuracy phase transitions and a unified explanation of delayed convergence in deep networks. The explicit curvature formula, stated as derived from the Gauss equation, is a potentially useful contribution, and the observation that curvature change-points coincide with accuracy transitions is interesting. The claimed connection to grokking is ambitious and, if supported by proper hysteresis experiments, would be significant. However, the current evidence is largely single-run and correlational; the load-bearing hysteresis claim and the basin-count prediction are not yet established. The paper does not fit any free parameter to the observed transition locations, which is a strength, but the empirical base is too thin to support the breadth of the claims.

major comments (4)
  1. [Grokking section, Fig. 3] The central claim that 'grokking is nothing but a hysteresis effect' is not supported by the reported experiments. In Fig. 3 the regularization strength beta is fixed at a sub-critical value and only the initial condition is varied; the green and orange curves show slower escape from low-accuracy initializations. Hysteresis in a first-order transition requires sweeping the control parameter beta in both directions and demonstrating that the transition point depends on the sweep direction. Since no reverse sweep or annealing-down experiment is reported, the delayed convergence is initial-condition dependence, not a demonstrated hysteresis loop. The manuscript itself concedes that natural grokking is 'unlikely to occur' in this setup, so the connection to actual grokking remains hypothetical.
  2. [Eq. (3) and Supplementary Eq. (17)] The scalar-curvature formula appears to have incorrect denominators. Starting from the authors' own definitions g^{ij} = delta^{ij} - partial^i l partial^j l / ||nabla F||^2 and II_{ij} = H_{ij}/||nabla F||, the Gauss equation gives R = [tr(H)^2 - tr(H^2)]/||nabla F||^2 + 2/||nabla F||^4 nabla l^T (H^2 - tr(H)H) nabla l, not the printed expression with denominators ||nabla F|| and ||nabla F||^2. A direct expansion of the authors' Supplementary Eqs. (15) and (16) yields this corrected form. If the numerical experiments used the printed formula, the reported Ricci-scalar values in Figs. 2 and 4 and the derived change-points are affected; if the correct formula was used, the manuscript should be corrected to avoid propagating the error.
  3. [Supplementary 'Varying data-complexity' and MNIST section] The prediction that the number of transitions increases with data complexity is not isolated. The 1D-versus-2D Gaussian comparison changes the output dimension and hence the target distribution; the MNIST experiment changes the loss from MSE to cross-entropy, the input dimension, and the task, so any increase in transition count cannot be attributed to data complexity alone. The paper states this premise as hypothesis (iii) without a derivation or a controlled experiment, yet it is load-bearing for the prediction of new transition points. A minimal control, such as fixing the architecture and loss while varying only the complexity of the Gaussian target, would be needed.
  4. [Results and Supplementary 'Experimental Setup'] The main curves appear to be single runs without error bars, multiple random seeds, or a description of the change-point detection algorithm (the MNIST caption says only 'some change-point detection algorithm'). Because the central evidence is the existence and location of discontinuities in noisy-looking curves, the absence of uncertainty quantification and a reproducible detection procedure makes it difficult to judge whether the reported beta0 and beta1 values are robust. The authors should specify the detection algorithm, its thresholds, and the statistics over repeated runs.
minor comments (4)
  1. [Title and reference list] The title contains a typo ('Neur al'), and references [4] and [21] are duplicates of the same Ziyin and Ueda paper; these should be cleaned up.
  2. [Grokking section, first paragraph] The phrase 'β j β1' is incomplete; it should read 'β ≪ β1' or similar. The lowercase spelling of 'grokking' throughout is also inconsistent with standard usage.
  3. [Completeness of the experimental description] The paper states that the full Hessian matrices are saved for each trained model but does not report how the Hessian is computed (e.g., exact, Hutchinson, or finite differences), which is important for reproducibility of the curvature values.
  4. [Conclusion] The conclusion acknowledges that the global structure of the error landscape cannot be deduced from the reported experiments; this limitation should be reflected earlier in the paper when the basin-count hypothesis is introduced.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Ricci-curvature diagnostic is derived and measured without fitting transition locations; the main weaknesses (hysteresis claim, post-hoc 'prediction' of transitions) are evidential rather than definitional.

full rationale

The Ricci scalar is derived from first principles via the Gauss equation and is evaluated at trained models; no parameter is fitted to the observed transition locations, so the central geometric quantity is not constructed from the target predictions. The concurrence of curvature change-points with accuracy/error jumps at β0 and β1 is a consistency relation rather than an independent confirmation: since R is a deterministic function of the trained parameters and Hessian, any discontinuous jump in the annealing path θ(β) will, generically, produce a jump in R. Thus the curvature plots provide a diagnostic of the same event rather than an independent causal explanation. The 'prediction' of additional transitions with increasing data complexity is an empirical post-hoc generalization from 1D, 2D, and MNIST experiments (which also change loss and architecture), and the grokking/hysteresis claim is asserted on the basis of initialization-dependent convergence without a forward-reverse β sweep; these are correctness/evidential weaknesses, not circular reductions. No self-citation chain or imported uniqueness theorem is load-bearing. Therefore no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new particles or physical entities. Its main postulates are the existence of accuracy basins and the scaling of their number with data complexity, both of which are assumptions rather than derived results. The Ricci scalar formula is standard differential geometry (though sourced from Stephenson et al.). The free parameters are experimental hyperparameters and the unspecified change-point detector, which affect the reported number and location of transitions.

free parameters (3)
  • change-point detection algorithm and threshold
    The number and location of transition points beta_i depend on an unspecified 'change-point detection algorithm' used in the main MNIST figure and in the toy-model figures. This choice directly affects the claim of how many transitions exist.
  • network width (15 hidden neurons) = 15
    The hidden layer width is chosen by running trials; the paper states that above 15 there is no significant improvement, but the choice is not derived from theory and may influence the presence and location of transitions.
  • training sample size N=10^4 = 10^4
    The sample size is chosen by trying different orders of magnitude. It is a hyperparameter of the experiments, not derived from the theory.
assumptions (4)
  • standard math The error surface is a smooth d-dimensional submanifold of R^{d+1} so that curvature quantities can be computed.
    Invoked in the supplementary (Geometry of the Model Space) to define the first and second fundamental forms and derive the Ricci scalar. This is a standard assumption for the mathematical framework, but real loss landscapes may be non-smooth or degenerate.
  • domain assumption Training converges to a local or global minimum of the regularized loss for each beta value.
    The curvature and distance measurements are taken at the final trained model after a fixed number of epochs. In non-convex neural network training, convergence is not guaranteed, and the results may depend on the optimization trajectory.
  • domain assumption In annealing mode, the model follows the global minimum as beta increases, so the sequence of trained models traces a continuous path of loss minima.
    The experimental setup uses the previous model's parameters as initialization for the next beta. This assumes the optimizer tracks the minimum, which is plausible for small beta changes but not proven.
  • ad hoc to paper The error landscape contains distinct high-accuracy basins whose number increases with data complexity.
    This is the central explanatory hypothesis in the Results section ('the model traverses through a set of basins'). It is not derived from the data or theory; it is a postulate used to interpret the observed transitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks." pith.science (2026). https://pith.science/paper/PZ4LR554

@misc{pith2026250506597,
  author       = {Pith},
  title        = {Pith review of: Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZ4LR554}},
  note         = {Machine review of arXiv:2505.06597}
}
read the original abstract

Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase -- the so-called onset-of learning. We explain this transition via the scalar (Ricci) curvature of the error landscape. We predict new transition points as the data complexity is increased and, in accordance with the theory of phase transitions, the existence of hysteresis effects. We confirm both predictions numerically. Our results provide a natural explanation of the recently discovered phenomenon of '\emph{grokking}' as DNN models getting stuck in a local minimum of the error surface, corresponding to a lower accuracy phase. Our work paves the way for new probing methods of the intrinsic structure of DNNs in and beyond the L2 context.

Figures

Figures reproduced from arXiv: 2505.06597 by the authors.

Figure 1
Figure 1. FIG. 1. Sketch of a loss landscape with increasing L2 regularize [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2. Phase transitions in a two hidden layer neural network tra [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3. Mean-Squared-Error over Epochs for three mod [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 1
Figure 1. Figure 1: FIG. 1. Sketch of an error surface [PITH_FULL_IMAGE:figures/full_fig_p005_1.png]
Figure 2
Figure 2. Figure 2: FIG. 2. Phase transition in a two hidden layer neural network trai [PITH_FULL_IMAGE:figures/full_fig_p008_2.png]
Figure 3
Figure 3. Figure 3: FIG. 3. Accuracy of a NN with two hidden layers, trained on [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: FIG. 4. Phase transitions in a two hidden layer neural network tra [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5. MSE for increasing regularizer strength [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages

  1. [1]

    The Information Bottleneck Method,

    N. Tishby, F. Pereira, and W. Bialek, “The Information Bottleneck Method,” arXiv preprint arXiv:0004057 , 2000

    work page 2000

  2. [2]

    Information bottleneck for Gaussian variables,

    G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, “Information bottleneck for Gaussian variables,” Ad- vances in Neural Information Processing Systems , vol. 16, 2003

    work page 2003

  3. [3]

    Learnability for the information bottleneck,

    T. Wu, I. Fischer, I. L. Chuang, and M. Tegmark, “Learnability for the information bottleneck,” in Uncer- tainty in Artificial Intelligence , pp. 1050–1060, PMLR, 2020. 4 0 2500 5000 7500 10000 12500 15000 Epoch 0.44 0.48 0.52 0.56 0.60MSE Random Init Trivial Phase Intermediate Phase FIG. 3. Mean-Squared-Error over Epochs for three mod- els trained with di...

    work page 2020

  4. [4]

    Zeroth, first, and second-order phase transitions in deep neural networks,

    L. Ziyin and M. Ueda, “Zeroth, first, and second-order phase transitions in deep neural networks,” Physical Re- view Research, vol. 5, no. 4, p. 043243, 2023

    work page 2023

  5. [5]

    & Janssen, L

    Winter, M. & Janssen, L. Glassy dynamics in deep neural networks: A structural com- parison. Phys. Rev. Res. . 7, 023010 (2025,4), https://link.aps.org/doi/10.1103/PhysRevResearch.7.023010

    work page doi:10.1103/physrevresearch.7.023010 2025

  6. [6]

    & Gabriel, F

    Jacot, A., Golikov, E., Hongler, C. & Gabriel, F. Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsity. (2022), https://arxiv.org/abs/2205.15809

    arXiv 2022

  7. [7]

    & Ringel, Z

    Rubin, N., Seroussi, I. & Ringel, Z. grokking as a First Order Phase Transition in Two Layer Networks. (2024), https://arxiv.org/abs/2310.03789

    arXiv 2024

  8. [8]

    & Pos- ner, I

    DeMoss, B., Sapora, S., Foerster, J., Hawes, N. & Pos- ner, I. The complexity dynamics of grokking. Physica D: Nonlinear Phenomena . pp. 134859 (2025)

    work page 2025

Show all 27 references
  1. [9]

    & Tegmark, M

    Liu, Z., Michaud, E. & Tegmark, M. Omnigrok: grokking Beyond Algorithmic Data. The Eleventh International Confer- ence On Learning Representations . (2023), https://openreview.net/forum?id=zDiHoIWa0q1

    2023

  2. [10]

    Natural gradient works efficiently in learn- ing,

    S. Amari, “Natural gradient works efficiently in learn- ing,” Neural Computation , vol. 10, no. 2, pp. 251–276, 1998

    1998

  3. [11]

    Amari, Information Geometry and Its Applications , vol

    S. Amari, Information Geometry and Its Applications , vol. 194, Springer, 2016

    2016

  4. [12]

    Identifying and attacking the saddle point problem in high-dimensional non-convex optimization,

    Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio, “Identifying and attacking the saddle point problem in high-dimensional non-convex optimization,” Advances in Neural Information Pro- cessing Systems , vol. 27, 2014

    2014

  5. [13]

    The loss surfaces of multilayer net- works,

    A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, “The loss surfaces of multilayer net- works,” in Artificial Intelligence and Statistics , pp. 192– 204, PMLR, 2015

    2015

  6. [14]

    C. M. Bishop, Pattern Recognition and Machine Learn- ing, Springer, 2006

    2006

  7. [15]

    Goodfellow, Y

    I. Goodfellow, Y. Bengio, and A. Courville, Deep Learn- ing, MIT Press, 2016

    2016

  8. [16]

    Ridge regression: Bi- ased estimation for nonorthogonal problems,

    A. E. Hoerl and R. W. Kennard, “Ridge regression: Bi- ased estimation for nonorthogonal problems,” Techno- metrics, vol. 12, no. 1, pp. 55–67, 1970

    1970

  9. [17]

    Feature Learning in L2-regularized DNNs: Attrac- tion/Repulsion and Sparsity,

    E. Golikov, A. Jacot, C. Hongler, and F. Gabriel, “Feature Learning in L2-regularized DNNs: Attrac- tion/Repulsion and Sparsity,” in NeurIPS 2022 , 2022

    2022

  10. [18]

    Phase Transitions for the Information Bottleneck in Representation Learning,

    T. Wu and I. S. Fischer, “Phase Transitions for the Information Bottleneck in Representation Learning,” CoRR, vol. abs/2001.01878, 2020

    2001 arXiv

  11. [19]

    Information-Driven Learning from High-Dimensional Data,

    A. Montanari and P. R. Sgc, “Information-Driven Learning from High-Dimensional Data,” Annual Review of Statistics and Its Application , vol. 6, pp. 121–146, 2019

    2019

  12. [20]

    Deep learning,

    Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  13. [21]

    Zeroth, first, and second-order phase transitions in deep neural networks,

    L. Ziyin and M. Ueda, “Zeroth, first, and second-order phase transitions in deep neural networks,” Phys. Rev. Res., vol. 5, no. 4, p. 043243, 2023

    2023

  14. [22]

    Watanabe, Algebraic Geometry and Statistical Learn- ing Theory , vol

    S. Watanabe, Algebraic Geometry and Statistical Learn- ing Theory , vol. 25, Cambridge University Press, 2009

    2009

  15. [23]

    The global landscape of neural networks: An overview,

    R. Sun, D. Li, S. Liang, T. Ding, and R. Srikant, “The global landscape of neural networks: An overview,” IEEE Signal Processing Magazine , vol. 37, no. 5, pp. 95–108, 2020

    2020

  16. [24]

    On the geometry of generalization and memorization in deep neural networks,

    C. Stephenson, S. Padhy, A. Ganesh, Y. Hui, H. Tang, and S. Chung, “On the geometry of generalization and memorization in deep neural networks,” arXiv preprint arXiv:2105.14602, 2021

    2021 arXiv

  17. [25]

    Lang, Fundamentals of Differential Geometry , vol

    S. Lang, Fundamentals of Differential Geometry , vol. 191, Springer, 2012

    2012

  18. [26]

    J. M. Lee, Introduction to Riemannian Manifolds , vol. 2, Springer, 2018

    2018

  19. [27]

    Auto-encoding varia- tional bayes,

    D. P. Kingma and M. Welling, “Auto-encoding varia- tional bayes,” in ICLR, Banff, Canada, 2013. 1 GEOMETR Y OF THE MODEL SP ACE Error and Loss Landscapes as Submanifolds in Model Space Here, we derive and investigate the geometric prop- erties of error and loss landscapes as d ...

    2013

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.