Pith. sign in

REVIEW 2 major objections 4 minor 59 references

Singular perturbations and hierarchical learning in two-layer neural networks

T0 review · 2 major / 4 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Joint two-layer training recovers link-function components on sharp, separated timescales set by a speed-separation parameter.

desk verdict Solid rigorous proofs of the BMZ constant/linear timescales plus a reusable constrained-flow approximation for the quadratic onset; soft spots are declared modelling assumptions, not hidden gaps. read the letter →

arxiv 2607.10869 v1 pith:3BSJMQ6H submitted 2026-07-12 cs.LG math.DSmath.OC

Cédric Gerbelot , Jean-Christophe Mourrat This is my paper
classification cs.LGmath.DSmath.OC MSC 68T0734E1537N40
keywords hierarchicallearningtwo-layerneuralnetworkssingularperturbationmean-fieldlimitsingle-indexmodelgradientflowHermiteexpansion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When an infinitely wide two-layer network is trained jointly on a high-dimensional single-index target whose activation is unknown, and the outer layer is forced to move much faster than the inner layer, the network recovers the constant, linear, and then quadratic pieces of the hidden link on successively longer timescales. The paper proves the first two of those timescales exactly, matching earlier non-rigorous predictions, and shows that once those pieces are learned they keep shaping every later stage through integral constraints that the dynamics must preserve. The same analysis reveals that the weight measure becomes singular: a few neurons grow rapidly while the rest rearrange to keep the already-learned coefficients intact. A sympathetic reader cares because the result supplies the first rigorous, quantitative description of hierarchical feature learning under ordinary joint gradient flow, without layer-wise oracles.

What carries the argument

Quantitative tube approximations for singularly perturbed infinite-dimensional ODEs that remain near a manifold defined by integral constraints (the already-learned Hermite coefficients); the radii of the tubes adapt to the hierarchical timescales and yield the sharp thresholds.

What would settle it

Numerically integrate the mean-field ODEs for small ε and check whether the hitting times for |φ_{0}-σ_{0}a|~ε and for as reaching (1-α)φ_{1}/σ_{1} match the explicit constants 1/σ_{0}^{2} and 1/(4σ_{1}φ_{1}) predicted by Propositions 3.1–3.2; a systematic mismatch for positive coefficients would refute the claimed thresholds.

Watch

Extended reading notes

Core claim

Under a small speed-separation parameter ε the population gradient flow of the infinite-width two-layer network recovers the constant Hermite coefficient of the link to precision nearly ε on a timescale of order ε log(1/ε), reaches a non-trivial fraction of the linear coefficient by time (1/(4σ_{1}φ_{1}))√(ε log(1/ε)), and thereafter stays close to an auxiliary constrained flow that exactly preserves the already-learned integral constraints while the quadratic component begins to grow.

Load-bearing premise

Every Hermite coefficient of both the activation and the link is strictly positive, and the high-dimensional finite-width dynamics has already been reduced to the autonomous mean-field ODEs that the paper studies.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper rigorously analyzes the population gradient flow of an infinitely wide two-layer network learning a misspecified single-index model, with a small parameter ε controlling the relative speed of the second layer. Building on the mean-field reduction of Berthier–Montanari–Zhou, it proves that the constant and linear Hermite components of the link are recovered on the predicted timescales (Propositions 3.1–3.2, Theorem 3.3) via quantitative tubes around explicitly integrable idealized ODEs. For the quadratic component it constructs an auxiliary constrained flow that exactly preserves the already-learned integral constraints (Proposition 3.6, Appendix B) and shows that earlier components continue to influence the dynamics; simulations illustrate the associated singular rearrangement of the empirical measure of the weights.

Significance. The work supplies the first fully rigorous confirmation of the hierarchical timescales conjectured in Berthier–Montanari–Zhou for the constant and linear components, together with a general approximation lemma for singularly perturbed flows near integral-constraint manifolds that is of independent interest. The demonstration that previously learned coefficients cannot be neglected at the quadratic stage, and the accompanying phenomenological description of singular weight-measure behaviour, clarify the structure of joint two-layer training beyond the kernel regime. The proofs are self-contained once the mean-field ODEs and positivity of Hermite coefficients are granted, and the tube constructions and Gronwall estimates are sharp enough to recover the leading constants.

major comments (2)
  1. The quadratic analysis (Section 3.2, equations (3.11)–(3.12) and Proposition 3.6) is performed on a truncated system that retains only the first three Hermite terms. While the truncation is declared, the paper does not quantify the error incurred by discarding higher-order terms on the ε^{1/4} timescale; a short a-priori bound showing that those terms remain negligible under the same tube radii would make the claim that earlier components continue to influence the dynamics fully rigorous for the original infinite series.
  2. The reduction of the high-dimensional finite-width dynamics to the autonomous mean-field ODEs (2.8)–(2.9) is taken as given from Berthier–Montanari–Zhou. The related-work discussion (Section 4) correctly notes that existing propagation-of-chaos bounds are insufficient on the relevant timescales; a brief remark on the precise regime (m,d versus 1/ε) in which the present conclusions transfer to the original particle system would strengthen the modelling claim.
minor comments (4)
  1. Assumption 1 requires σ_k > 0 and φ_k > 0 for every k; a short discussion of what happens when a low-order coefficient vanishes (or a pointer to the corresponding open question) would help the reader assess robustness.
  2. Figures 1–6 are informative but the captions could more explicitly state the precise truncation used in each panel and the numerical values of the constants C, δ appearing in the theorems.
  3. The notation for the idealized linear system (β,γ versus b,g) is introduced in two places (Section 3 and Appendix A); a single consistent definition would improve readability.
  4. Typographical inconsistencies appear in the arXiv header (date July 14, 2026) and in a few places where “ε” is written as “e”; these should be cleaned before final publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: timescales and tubes are derived from the ODEs by retaining leading terms or enforcing integral constraints, with the source conjecture used only as motivation.

full rationale

The paper takes the mean-field ODEs (2.8)–(2.9) as given from Berthier–Montanari–Zhou and then constructs idealized systems (b, β, γ and the constrained flow z1, z2 with Lagrange multipliers α0, α1) by dropping higher-order Hermite terms or by solving the linear system that keeps already-learned integral constraints exactly zero. All error bounds (Theorems 7.1–7.2, Proposition B.1, Corollary B.2) are obtained by differential inequalities, Gronwall, and Taylor expansions around those idealized flows; no free parameters are fitted to data, and no uniqueness or ansatz is imported from the authors’ own prior work. The citation to the conjecture paper supplies only the setting and the predicted timescales that are subsequently proved; the proofs themselves are self-contained once Assumption 1 and the autonomous ODEs are granted. Consequently the claimed sharp thresholds and the quadratic-onset description do not reduce to their inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The central claims rest on standard ODE theory, the mean-field reduction already established by Berthier–Montanari–Zhou, positivity of all Hermite coefficients, and the singular-perturbation scaling ε→0. No free parameters are fitted to data; ε is an asymptotic parameter. The only invented technical objects are the auxiliary constrained flows and the time-dependent tubes used for approximation.

assumptions (4)
  • domain assumption All Hermite coefficients σ_k > 0 and φ_k > 0 for every k ≥ 0 (Assumption 1).
    Used throughout §3 and the proofs to guarantee that every component is eventually learnable and that the idealized systems remain non-degenerate.
  • domain assumption The high-dimensional finite-width population gradient flow reduces, in the joint m,d o∞ limit, to the autonomous mean-field ODEs (2.8)–(2.9) with s(0,·)=0.
    Taken as given from Berthier–Montanari–Zhou; the entire analysis is performed on these reduced ODEs.
  • standard math Cauchy–Lipschitz local existence/uniqueness for the locally Lipschitz vector fields that appear (Remark 2.1).
    Invoked for every initial-value problem considered.
  • ad hoc to paper The matrix G(u) of integrated inner products of the constraint gradients and the fast vector fields is full-rank on a non-empty open set U (App. B).
    Required for the Lagrange-multiplier coefficients α(t) to be well-defined; verified for the constant/linear constraints but left as an assumption for higher-order extensions.
invented entities (2)
  • Auxiliary constrained flow (z_{1},z_{2}) with time-dependent Lagrange multipliers α_{0}(t),α_{1}(t) that exactly preserve the already-learned integral constraints
    purpose: Provides the idealized dynamics against which the true singularly-perturbed flow is compared once the constant and linear components have been learned.
    Constructed in §3.2 and Prop. B.1; no independent experimental handle outside the paper.
  • Time-dependent tubes of radii controlled by eta(t) and powers of ε around the idealized constant/linear solutions b,b,g
    purpose: Quantitative approximation device that yields the sharp recovery timescales.
    Defined and controlled in Thm. 7.1–7.2; purely analytic construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Singular perturbations and hierarchical learning in two-layer neural networks." pith.science (2026). https://pith.science/paper/3BSJMQ6H

@misc{pith2026260710869,
  author       = {Pith},
  title        = {Pith review of: Singular perturbations and hierarchical learning in two-layer neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BSJMQ6H}},
  note         = {Machine review of arXiv:2607.10869}
}
read the original abstract

We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in \cite{berthier2024learning}, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.

Figures

Figures reproduced from arXiv: 2607.10869 by the authors.

Figure 1
Figure 1. Separation of timescales for the constant and linear components: evolution of the averaged quantities a, as, and cost function for the first two coefficients φ0, φ1 as a function of time on a logarithmic scale, for ε = 10−3 (left) and ε = 10−4 (right) [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Theorem 3.3: comparison of a (orange curve) with b (blue curve), and of as (red curve) with bg (green curve), for ε = 10−3 (left) and ε = 10−4 (right). On a timescale of order ε log(1/ε), the blue and orange curves are nearly indis￾tinguishable, and, when moving to a timescale of order √ ε log(1/ε), the error becomes larger, as predicted by the bound (3.8) (see also Remark 3.5). The curves collapse a… view at source ↗
Figure 3
Figure 3. Illustration of Theorem 3.3: comparison of ∥a⊥(t, .) − b(t, .)∥ with β(t) (top two figures), and ∥s⊥(t, .) − g(t, .)∥ with γ(t) (bottom two figures), for ε = 10−3 (left panel) and ε = 10−4 (right panel). The approximation errors roughly behave as small multiples of β(t), as predicted by the bound (3.9) (see also Remark 3.4), with the error decreasing with ε. coefficients once they have been learned. Indeed, we canno… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Separation of timescales for the constant, linear and quadratic component: evolution of the quantities a, as, as2, along with the cost function for the first three coefficients φ0, φ1, φ2, as a function of time on a logarithmic scale, for ε = 10−5 (left) and ε = 10−6 (…
Figure 5
Figure 5. Figure 5: Illustration of Proposition 3.6: comparison of the evolution of aˆsˆ 2 (blue curve), aˆqsˆ 2 q (green curve) and z1z 2 2 (orange curve), for ε = 10−5 (left) and ε = 10−6 (right). The explosion time for the quantity aˆqsˆ 2 q is significantly smaller than that of aˆsˆ 2…
Figure 6
Figure 6. Figure 6: Evolution of the maps z1(t, .), z2(t, .), in the setting of [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 4 linked inside Pith

  1. [1]

    Foundations of Computational Mathematics , pages=

    Learning time-scales in two-layers neural networks , author=. Foundations of Computational Mathematics , pages=. 2024 , publisher=

    2024

  2. [2]

    2012 , publisher=

    Ordinary differential equations and dynamical systems , author=. 2012 , publisher=

    2012

  3. [3]

    Conference on Learning Theory , pages=

    Neural networks can learn representations with gradient descent , author=. Conference on Learning Theory , pages=. 2022 , organization=

    2022

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    High-dimensional asymptotics of feature learning: How one gradient step improves the representation , author=. Advances in Neural Information Processing Systems , volume=

  5. [5]

    2017 , publisher=

    Deep learning , author=. 2017 , publisher=

    2017

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    When do neural networks outperform kernel methods? , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    Advances in neural information processing systems , volume=

    On the global convergence of gradient descent for over-parameterized models using optimal transport , author=. Advances in neural information processing systems , volume=

  8. [8]

    Proceedings of the National Academy of Sciences , volume=

    A mean field view of the landscape of two-layer neural networks , author=. Proceedings of the National Academy of Sciences , volume=. 2018 , publisher=

    2018

Show all 59 references
  1. [9]

    Communications on Pure and Applied Mathematics , volume=

    Trainability and accuracy of artificial neural networks: An interacting particle system approach , author=. Communications on Pure and Applied Mathematics , volume=. 2022 , publisher=

    2022

  2. [10]

    Advances in neural information processing systems , volume=

    Neural tangent kernel: Convergence and generalization in neural networks , author=. Advances in neural information processing systems , volume=

  3. [11]

    Advances in neural information processing systems , volume=

    On lazy training in differentiable programming , author=. Advances in neural information processing systems , volume=

  4. [12]

    Advances in neural information processing systems , volume=

    On the power and limitations of random features for understanding neural networks , author=. Advances in neural information processing systems , volume=

  5. [13]

    Conference on learning theory , pages=

    Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss , author=. Conference on learning theory , pages=. 2020 , organization=

    2020

  6. [14]

    Mathematical Programming , volume=

    Sparse optimization on measures with over-parameterized gradient descent , author=. Mathematical Programming , volume=. 2022 , publisher=

    2022

  7. [15]

    Journal of Physics A: Mathematical and general , volume=

    Learning by on-line gradient descent , author=. Journal of Physics A: Mathematical and general , volume=

  8. [16]

    Physical Review Letters , volume=

    Exact solution for on-line learning in multilayer neural networks , author=. Physical Review Letters , volume=. 1995 , publisher=

    1995

  9. [17]

    Advances in neural information processing systems , volume=

    Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup , author=. Advances in neural information processing systems , volume=

  10. [18]

    Journal of Machine Learning Research , volume=

    Online stochastic gradient descent on non-convex losses from high-dimensional inference , author=. Journal of Machine Learning Research , volume=

  11. [19]

    Conference on Learning Theory , pages=

    Learning a single neuron with gradient methods , author=. Conference on Learning Theory , pages=. 2020 , organization=

    2020

  12. [20]

    The Thirty Sixth Annual Conference on Learning Theory , pages=

    Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics , author=. The Thirty Sixth Annual Conference on Learning Theory , pages=. 2023 , organization=

    2023

  13. [21]

    Advances in neural information processing systems , volume=

    Learning single-index models with shallow neural networks , author=. Advances in neural information processing systems , volume=

  14. [22]

    Communications on Pure and Applied Mathematics , volume=

    On learning Gaussian multi-index models with gradient flow part I: General properties and two-timescale learning , author=. Communications on Pure and Applied Mathematics , volume=. 2025 , publisher=

    2025

  15. [23]

    Journal of Machine Learning Research , volume=

    How two-layer neural networks learn, one (giant) step at a time , author=. Journal of Machine Learning Research , volume=

  16. [24]

    International Conference on Machine Learning , pages=

    Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  17. [25]

    International Conference on Learning Representations , volume=

    Sgd finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the xor problem , author=. International Conference on Learning Representations , volume=

  18. [26]

    International Conference on Learning Representations , volume=

    High-dimensional SGD aligns with emerging outlier eigenspaces , author=. International Conference on Learning Representations , volume=

  19. [27]

    Communications on Pure and Applied Mathematics , volume=

    High-dimensional limit theorems for SGD: Effective dynamics and critical scaling , author=. Communications on Pure and Applied Mathematics , volume=. 2024 , publisher=

    2024

  20. [28]

    Conference on Learning Theory , pages=

    Kernel and rich regimes in overparametrized models , author=. Conference on Learning Theory , pages=. 2020 , organization=

    2020

  21. [29]

    Journal of Statistical Mechanics: Theory and Experiment , volume=

    Disentangling feature and lazy training in deep neural networks , author=. Journal of Statistical Mechanics: Theory and Experiment , volume=. 2020 , publisher=

    2020

  22. [30]

    Journal of Machine Learning Research , volume=

    Online stochastic gradient descent with arbitrary initialization solves non-smooth, non-convex phase retrieval , author=. Journal of Machine Learning Research , volume=

  23. [31]

    Journal of Machine Learning Research , volume=

    Breaking the curse of dimensionality with convex neural networks , author=. Journal of Machine Learning Research , volume=

  24. [32]

    arXiv preprint arXiv:2505.21336 , year=

    Joint Learning in the Gaussian Single Index Model , author=. arXiv preprint arXiv:2505.21336 , year=

    arXiv

  25. [33]

    Advances in Neural Information Processing Systems , volume=

    Neural network learns low-dimensional polynomials with sgd near the information-theoretic limit , author=. Advances in Neural Information Processing Systems , volume=

  26. [34]

    Advances in Neural Information Processing Systems , volume=

    Learning parities with neural networks , author=. Advances in Neural Information Processing Systems , volume=

  27. [35]

    2012 , publisher=

    Introduction to perturbation methods , author=. 2012 , publisher=

    2012

  28. [36]

    2023 , publisher=

    An introduction to optimization on smooth manifolds , author=. 2023 , publisher=

    2023

  29. [37]

    Conference On Learning Theory , pages=

    Learning single-index models in gaussian space , author=. Conference On Learning Theory , pages=. 2018 , organization=

    2018

  30. [38]

    The Annals of Probability , volume=

    Algorithmic thresholds for tensor PCA , author=. The Annals of Probability , volume=. 2020 , publisher=

    2020

  31. [39]

    Advances in neural information processing systems , volume=

    Dynamics of on-line gradient descent learning for multilayer neural networks , author=. Advances in neural information processing systems , volume=

  32. [40]

    Advances in Neural Information Processing Systems , volume=

    Leveraging the two-timescale regime to demonstrate convergence of neural networks , author=. Advances in Neural Information Processing Systems , volume=

  33. [41]

    International conference on machine learning , pages=

    Gradient descent finds global minima of deep neural networks , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  34. [42]

    International conference on machine learning , pages=

    A closer look at memorization in deep networks , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  35. [43]

    Conference on Learning Theory , pages=

    The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks , author=. Conference on Learning Theory , pages=. 2022 , organization=

    2022

  36. [44]

    , author=

    On the symmetries in the dynamics of wide two-layer neural networks. , author=. Electronic Research Archive , volume=

  37. [45]

    Conference on learning theory , pages=

    Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit , author=. Conference on learning theory , pages=. 2019 , organization=

    2019

  38. [46]

    Advances in Neural Information Processing Systems , volume=

    Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks , author=. Advances in Neural Information Processing Systems , volume=

  39. [47]

    The Thirty Sixth Annual Conference on Learning Theory , pages=

    From high-dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks , author=. The Thirty Sixth Annual Conference on Learning Theory , pages=. 2023 , organization=

    2023

  40. [48]

    arXiv preprint arXiv:2005.13530 , year=

    On the convergence of gradient descent training for two-layer relu-networks in the mean field regime , author=. arXiv preprint arXiv:2005.13530 , year=

    2005 arXiv

  41. [49]

    Proceedings of Machine Learning Research vol , volume=

    Mean-field analysis of polynomial-width two-layer neural network beyond finite time horizon , author=. Proceedings of Machine Learning Research vol , volume=

  42. [50]

    arXiv preprint arXiv:2605.22010 , year=

    Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks , author=. arXiv preprint arXiv:2605.22010 , year=

    arXiv

  43. [51]

    Advances in Neural Information Processing Systems , volume=

    Beyond ntk with vanilla gradient descent: A mean-field analysis of neural networks with polynomial width, samples, and time , author=. Advances in Neural Information Processing Systems , volume=

  44. [52]

    Advances in Neural Information Processing Systems , volume=

    Saddle-to-saddle dynamics in diagonal linear networks , author=. Advances in Neural Information Processing Systems , volume=

  45. [53]

    Journal of Machine Learning Research , volume=

    Incremental learning in diagonal linear networks , author=. Journal of Machine Learning Research , volume=

  46. [54]

    Proceedings of the International Conference on Learning Represenatations 2014 , year=

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , author=. Proceedings of the International Conference on Learning Represenatations 2014 , year=

    2014

  47. [55]

    arXiv preprint arXiv:2106.15933 , year=

    Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity , author=. arXiv preprint arXiv:2106.15933 , year=

    arXiv

  48. [56]

    Systems & Control Letters , volume=

    Stochastic approximation with two time scales , author=. Systems & Control Letters , volume=. 1997 , publisher=

    1997

  49. [57]

    Convergence rate of linear two-time-scale stochastic approximation , author=

  50. [58]

    Conference On Learning Theory , pages=

    Finite sample analysis of two-timescale stochastic approximation with applications to reinforcement learning , author=. Conference On Learning Theory , pages=. 2018 , organization=

    2018

  51. [59]

    SIAM Journal on Optimization , volume=

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic , author=. SIAM Journal on Optimization , volume=. 2023 , publisher=

    2023

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.