Pith. sign in

REVIEW 2 major objections 42 references

A Theory of Saddle Escape in Deep Nonlinear Networks

T0 review · 2 major / 0 minor · reviewed 2026-07-01 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An exact identity for Frobenius-norm imbalance in deep nonlinear networks reduces saddle escape to a scalar ODE whose time scale depends on bottleneck layer count r rather than total depth L.

desk verdict Exact identity on Frobenius norm imbalance is new and general, but the r-2 escape scaling rests on an unquantified approximate balance law. read the letter →

arxiv 2605.01288 v3 pith:3VKFXRQM submitted 2026-05-02 cs.LG cond-mat.dis-nnstat.ML

Divit Rawal , Michael R. DeWeese This is my paper · ORCID
classification cs.LGcond-mat.dis-nnstat.ML
keywords deepnonlinearnetworkssaddleescapeFrobeniusnormimbalanceuniversalityclassespermutation-symmetricsubmanifoldcriticaldepthscalarODEreductionactivationfunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper derives an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for any smooth activation and any differentiable loss. This identity partitions activation functions into four universality classes. On the permutation-symmetric submanifold the identity combines with an approximate balance law to collapse the full matrix flow to a scalar ordinary differential equation. Solving the ODE produces an escape-time law from saddle points that scales as epsilon to the power of minus (r minus 2), where r is the number of layers at the bottleneck scale. The same exponent appears under He-normal initialization when the r bottleneck layers are rescaled by epsilon, and the predictions match numerical simulations.

What carries the argument

Exact identity for the imbalance of Frobenius norms of layer weight matrices; it classifies activations into four universality classes and, together with the approximate balance law, reduces the matrix flow to a scalar ODE on the permutation-symmetric submanifold.

What would settle it

Numerical experiments that vary the number r of bottleneck layers while holding total depth L fixed and measure an escape-time scaling that deviates from epsilon to the power of minus (r minus 2).

Watch

Extended reading notes

Core claim

The authors establish an exact identity for the imbalance of Frobenius norms of the layer weight matrices that is valid for any smooth activation function and any differentiable loss function. When restricted to the permutation-symmetric submanifold and combined with an approximate balance law, this identity reduces the high-dimensional matrix flow to a one-dimensional ODE. The resulting critical-depth escape time is governed by the exponent r-2, where r counts the layers at the bottleneck scale rather than the total depth L. The same scaling is recovered under He-normal initialization with r bottleneck layers rescaled by epsilon, and the predictions agree closely with numerical simulations.

Load-bearing premise

The approximate balance law on the permutation-symmetric submanifold must hold in order to reduce the full matrix flow to the scalar ODE.

Editorial extensions

If this is right

  • Escape time from saddles is independent of total depth L and is controlled only by the bottleneck layer count r.
  • Activation functions are grouped into four universality classes according to the form taken by the norm-imbalance identity.
  • The r-2 exponent is recovered under He-normal initialization once the r bottleneck layers are rescaled by epsilon.
  • The scalar-ODE reduction produces predictions that match numerical simulations of the training dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the approximate balance law holds beyond the symmetric submanifold, the scalar reduction could simplify analysis of training phases that begin from asymmetric initializations.
  • The four universality classes suggest that activation choice could be used to adjust the escape exponent without altering network depth or width.
  • The critical-depth result implies that optimization speed near saddles is set by the narrowest scale rather than overall network size.
  • The same reduction technique might be applied to other phases of gradient flow once an analogous balance law is identified.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper derives an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for arbitrary smooth activations and differentiable losses, using it to classify activations into four universality classes. On the permutation-symmetric submanifold this identity is combined with an approximate balance law to reduce the matrix dynamics to a scalar ODE, producing the escape-time scaling τ★ = Θ(ε^{-(r-2)}) controlled by bottleneck depth r rather than total depth L. The same exponent is recovered under He-normal initialization with r rescaled bottleneck layers (where the symmetry manifold is preserved but not attracting), and the predictions are reported to agree closely with simulations.

Significance. If the approximate balance law can be shown to hold with controlled error in the relevant small-ε regime, the exact identity and resulting critical-depth scaling would constitute a substantive advance in the analysis of saddle escape for deep nonlinear networks, moving beyond the well-understood linear and shallow cases. The parameter-free character of the identity itself is a clear strength.

major comments (2)
  1. [abstract] Abstract (paragraph on reduction to scalar ODE): the headline scaling τ★ = Θ(ε^{-(r-2)}) is obtained only after invoking an unspecified 'approximate balance law' on the permutation-symmetric submanifold; no error bound, scaling regime, or estimate of the neglected terms is supplied. If those terms are O(ε^α) with α < r-2 they can dominate the leading balance and change both the exponent and the claim that only r (not L) governs escape.
  2. [abstract] Abstract (He-normal initialization paragraph): the statement that the r-2 exponent is recovered when the symmetry manifold is preserved but not attracting does not address whether the same approximate balance law remains valid or whether the neglected terms again affect the leading-order scaling; this is load-bearing for the universality of the critical-depth law.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the careful reading and for identifying the exact identity as a strength. We respond point-by-point to the major comments on the approximate balance law.

read point-by-point responses

  1. Referee: [abstract] Abstract (paragraph on reduction to scalar ODE): the headline scaling τ★ = Θ(ε^{-(r-2)}) is obtained only after invoking an unspecified 'approximate balance law' on the permutation-symmetric submanifold; no error bound, scaling regime, or estimate of the neglected terms is supplied. If those terms are O(ε^α) with α < r-2 they can dominate the leading balance and change both the exponent and the claim that only r (not L) governs escape.

    Authors: We agree that the manuscript invokes the approximate balance law without supplying a rigorous error bound or explicit scaling regime for the neglected terms. The law follows from combining the exact imbalance identity with the observation that, on the permutation-symmetric submanifold under small initialization, the layer norms remain close; the resulting scalar ODE is therefore a leading-order reduction. Simulations across multiple activations and depths show that the neglected terms remain subdominant and do not alter the r-2 exponent or the independence from total depth L. In revision we will add a paragraph after the reduction derivation that states the working assumptions, reports the observed numerical error scaling, and clarifies that the claim concerns the leading-order escape time. This is a partial revision because a fully rigorous a-priori bound is not supplied. revision: partial

  2. Referee: [abstract] Abstract (He-normal initialization paragraph): the statement that the r-2 exponent is recovered when the symmetry manifold is preserved but not attracting does not address whether the same approximate balance law remains valid or whether the neglected terms again affect the leading-order scaling; this is load-bearing for the universality of the critical-depth law.

    Authors: Under He-normal initialization with r rescaled bottleneck layers the flow exactly preserves the symmetry manifold, so the same exact identity applies and the identical reduction to the scalar ODE is used. Additional simulations (to be added) confirm that the approximate balance law continues to hold with error of the same order as in the attracting case, yielding the same r-2 scaling. The revision will expand the relevant paragraph and abstract sentence to note this numerical verification explicitly, thereby supporting the universality statement. Again this is partial because the error control remains numerical rather than analytic. revision: partial

standing simulated objections not resolved
  • A rigorous, a-priori error bound establishing that the neglected terms in the approximate balance law are o(ε^{r-2}) throughout the small-ε regime (required for a fully controlled proof of the leading-order scaling).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation chain is self-contained

full rationale

The paper states an exact identity for Frobenius-norm imbalance that holds for arbitrary smooth activations and differentiable losses, then invokes a separate approximate balance law on the permutation-symmetric submanifold to obtain the scalar ODE and the τ★ = Θ(ε^{-(r-2)}) scaling. No step reduces the claimed result to its own inputs by construction, no fitted parameter is relabeled as a prediction, and no load-bearing premise rests on a self-citation chain. The approximate law is an additional assumption whose error is not quantified in the provided text, but this is a question of rigor rather than circularity; the central identity and reduction steps remain independent of the final scaling law.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Review performed on abstract only; the primary additional premise is the approximate balance law used for the reduction. No free parameters or invented entities are mentioned in the abstract.

assumptions (1)
  • domain assumption Approximate balance law on the permutation-symmetric submanifold
    Combined with the exact identity to reduce matrix flow to scalar ODE, as stated in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Theory of Saddle Escape in Deep Nonlinear Networks." pith.science (2026). https://pith.science/paper/3VKFXRQM

@misc{pith2026260501288,
  author       = {Pith},
  title        = {Pith review of: A Theory of Saddle Escape in Deep Nonlinear Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3VKFXRQM}},
  note         = {Machine review of arXiv:2605.01288}
}
abstract

In deep networks with small initialization, training exhibits long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for any smooth activation and any differentiable loss and use this to classify activation functions into four universality classes. On the permutation-symmetric submanifold, the identity combines with an approximate balance law to reduce the full matrix flow to a scalar ODE, giving a critical-depth escape time law $\tau_\star = \Theta(\varepsilon^{-(r-2)})$ governed by the number $r$ of layers at the bottleneck scale rather than the total depth $L$. We find that this same $r-2$ exponent is recovered under He-normal initialization with $r$ bottleneck layers rescaled by $\varepsilon$, where the symmetry manifold is preserved by the flow but not attracting. We find close agreement between our theory and numerical simulations.

Figures

Figures reproduced from arXiv: 2605.01288 by the authors.

Figure 1
Figure 1. Empirical confirmation of ansatz reduction. (a) Loss vs. time: reduced ODE (solid line) overlaid with empirical NL-parameter gradient descent (circles). (b) Layer scales Xℓ versus time t: scalar ODE trajectories match the full-parameter dynamics. Because the ansatz is flow-invariant (Section B.1), the population gradient flow on W descends to a flow on the L scalars (X1, . . . , XL). The reduction below is stated in… view at source ↗
Figure 2
Figure 2. Escape time obeys critical-depth law on the manifold. (a) Escape time tesc vs initializa￾tion scale ε for balanced init: closed form (solid) and reduced ODE (diamonds), polynomial scaling ε −(L−2) steepens with depth. (b) Same at fixed L = 6 with r layers at bottleneck scale: diamonds track the ε −(r−2) law of Theorem 6. Theoretical prediction and experiment diverge at large ε. Proof sketch. Separability of eq. (8) … view at source ↗
Figure 3
Figure 3. Universality across activations. (a) Raw escape time tesc vs ε for three Class B activations (tanh, erf, sin; solid line) and two Class C (GELU, Swish; dashed). (b) After rescaling by K(σ) : Class B curves collapse onto the master curve of Corollary 7; Class C deviates by O(γC ε) per Section C. Proof sketch. Partition [s 2 1 , 1] into shells [s 2 j , s2 j+1]. Under the strict-hierarchy assumption the shells with j <… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Off-manifold critical-depth exponent. tesc vs ε for L = 8 tanh with r ∈ {3, 5, 8} bot￾tleneck layers, He-normal init, SGD: slopes track the ε −(r−2) law of Theorem 11. Theorem 11 makes the single-mode expo￾nent intrinsic to M and γ rather than to any ansatz. The produc…
Figure 5
Figure 5. Figure 5: Three-mode tanh cascade and escape-time decomposition. Black: training loss; blue: mode-1 alignment ∥W1v1∥2/ √ N. Light purple: leading-order single-mode prediction of Theo￾rem 5. Dark purple: homotopy identity T(1) = T(0)+R 1 0 A(ν)dν on the homotopy from decoupled si…

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages

  1. [1]

    Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics.ArXiv, abs/2302.11055, 2023

    Emmanuel Abbe, Enric Boix-Adserà, and Theodor Misiakiewicz. Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics.ArXiv, abs/2302.11055, 2023. URL https://api.semanticscholar.org/CorpusID:257078637

    work page arXiv 2023

  2. [2]

    The merged-staircase prop- erty: a necessary and nearly sufficient condition for sgd learning of sparse functions on two- layer neural networks, 2024

    Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. The merged-staircase prop- erty: a necessary and nearly sufficient condition for sgd learning of sparse functions on two- layer neural networks, 2024. URLhttps://arxiv.org/abs/2202.08658

    work page arXiv 2024

  3. [3]

    High-dimensional dynamics of generalization error in neural networks

    Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks, 2017. URLhttps://arxiv.org/abs/1710.03667

    work page Pith review arXiv 2017

  4. [4]

    Escaping mediocrity: how two-layer networks learn hard generalized linear models with sgd, 2024

    Luca Arnaboldi, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Escaping medi- ocrity: how two-layer networks learn hard generalized linear models with sgd, 2024. URL https://arxiv.org/abs/2305.18502

    work page arXiv 2024

  5. [5]

    On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

    Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization, 2018. URLhttps://arxiv.org/abs/1802.06509

    work page Pith review arXiv 2018

  6. [6]

    Why gradient clipping accel- erates training: A theoretical justification for adaptivity

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization, 2019. URLhttps://arxiv.org/abs/1905.13655

    work page arXiv 2019

  7. [7]

    Online stochastic gradient descent on non-convex losses from high-dimensional inference, 2021

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference, 2021. URLhttps://arxiv.org/ abs/2003.10409

    work page arXiv 2021

  8. [8]

    High-dimensional limit theorems for sgd: Effective dynamics and critical scaling, 2023

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling, 2023. URLhttps://arxiv.org/abs/ 2206.04030

    work page arXiv 2023

Show all 42 references
  1. [9]

    Simon, and Cengiz Pehlevan

    Alexander Atanasov, Alexandru Meterez, James B. Simon, and Cengiz Pehlevan. The opti- mization landscape of sgd across the feature learning strength, 2025. URLhttps://arxiv. org/abs/2410.04642

    2025

  2. [10]

    Simon, and Arthur Jacot

    Ioannis Bantzis, James B. Simon, and Arthur Jacot. Saddle-to-saddle dynamics in deep relu networks: Low-rank bias in the first saddle escape, 2026. URLhttps://arxiv.org/abs/ 2505.21722

    2026 arXiv

  3. [11]

    Learning by on-line gradient descent.Journal of Physics A: Mathematical and General, 28:643–656, 02 1995

    Michael Biehl and H Schwarze. Learning by on-line gradient descent.Journal of Physics A: Mathematical and General, 28:643–656, 02 1995. doi: 10.1088/0305-4470/28/3/018

    1995 doi

  4. [12]

    Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape, 2019

    Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape, 2019. URLhttps://arxiv.org/abs/1907.02911

    2019 arXiv

  5. [13]

    Lipschitz flow-box theorem, 2006

    Craig Calcaterra and Axel Boldt. Lipschitz flow-box theorem, 2006. URLhttps://arxiv. org/abs/math/0305207

    2006 arXiv

  6. [14]

    On lazy training in differentiable program- ming, 2020

    Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming, 2020. URLhttps://arxiv.org/abs/1812.07956

    2020

  7. [15]

    Du, Wei Hu, and Jason D

    Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced.Advances in Neural Information Processing Systems, 2018-December:384–395, 2018. ISSN 1049-5258. Publisher Copyright: © 2018 Cur- ran Assoc...

    2018

  8. [16]

    Effect of batch learning in multilayer neural networks

    Kenji Fukumizu. Effect of batch learning in multilayer neural networks. InInternational Con- ference on Neural Information Processing, 1998. URLhttps://api.semanticscholar. org/CorpusID:605683. 10

    1998

  9. [17]

    Sebastian Goldt, Madhu S Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup*.Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124010, December

    2020

  10. [18]

    doi: 10.1088/1742-5468/abc61e

    ISSN 1742-5468. doi: 10.1088/1742-5468/abc61e. URLhttp://dx.doi.org/10. 1088/1742-5468/abc61e

    doi

  11. [19]

    Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification, 2015. URLhttps://arxiv. org/abs/1502.01852

    2015 arXiv

  12. [20]

    A. C. Hindmarsh and L. R. Petzold. LSODA, ordinary differential equation solver for stiff or non-stiff system. Nuclear Energy Agency of the OECD (NEA), Sep 2005. URLhttps: //www.osti.gov/etdeweb/biblio/21352532

    2005

  13. [21]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson.Matrix Analysis. Cambridge University Press, 1990. ISBN 0521386322. URLhttp://www.amazon.com/Matrix-Analysis-Roger-Horn/ dp/0521386322%3FSubscriptionId%3D192BW6DQ43CK9FN0ZGG2%26tag%3Dws% 26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26...

    1990

  14. [22]

    Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity,

    Arthur Jacot, François Ged, Berfin ¸ Sim¸ sek, Clément Hongler, and Franck Gabriel. Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity,

  15. [23]

    URLhttps://arxiv.org/abs/2106.15933

  16. [24]

    Gradient descent aligns the layers of deep linear networks, 2019

    Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks, 2019. URLhttps://arxiv.org/abs/1810.02032

    2019 arXiv

  17. [25]

    Kakade, and Michael I

    Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape saddle points efficiently, 2017. URLhttps://arxiv.org/abs/1703.00887

    2017 arXiv

  18. [26]

    Directional convergence near small initializations and saddles in two-homogeneous neural networks, 2024

    Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in two-homogeneous neural networks, 2024. URLhttps://arxiv.org/abs/2402.09226

    2024

  19. [27]

    Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel L. K. Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dy- namics, 2021. URLhttps://arxiv.org/abs/2012.04728

    2021

  20. [28]

    Get rich quick: exact solutions reveal how unbalanced initializations pro- mote rapid feature learning

    Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations pro- mote rapid feature learning. InProceedings of the 38th International Conference on Neural Inform...

    2024

  21. [29]

    Simon, Michael R

    Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B. Simon, Michael R. DeWeese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A the- ory of feature learning in two-layer neural networks, 2025. URLhttps://arxiv.org/abs/ 2506.06489

    2025

  22. [30]

    A mean field view of the landscape of two-layer neural networks.Proceedings of the National Academy of Sciences, 115(33), 2018

    Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks.Proceedings of the National Academy of Sciences, 115(33), 2018. ISSN 1091-6490. doi: 10.1073/pnas.1806579115. URLhttp://dx.doi.org/10.1073/ pnas.1806579115

    2018 doi

  23. [31]

    Convergence and implicit bias of gradient flow on overparametrized linear networks, 2022

    Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. Convergence and implicit bias of gradient flow on overparametrized linear networks, 2022. URLhttps://arxiv.org/ abs/2105.06351

    2022

  24. [32]

    Saddle-to-saddle dynamics in diagonal linear networks,

    Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks,

  25. [33]

    URLhttps://arxiv.org/abs/2304.00488

  26. [34]

    Hamprecht, Yoshua Bengio, and Aaron Courville

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019. URL https://arxiv.org/abs/1806.08734. 11

    2019 arXiv

  27. [35]

    Dynamics of on-line gradient descent learning for multilayer neural networks

    David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. 04 1999

    1999

  28. [36]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014. URLhttps://arxiv.org/abs/ 1312.6120

    2014 arXiv

  29. [37]

    Saxe, Shagun Sodhani, and Sam Lewallen

    Andrew M. Saxe, Shagun Sodhani, and Sam Lewallen. The neural race reduction: Dynamics of abstraction in gated networks, 2022. URLhttps://arxiv.org/abs/2207.10430

    2022

  30. [38]

    Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J

    James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning, 2023. URLhttps://arxiv. org/abs/2303.15438

    2023

  31. [39]

    Geometry of the loss landscape in overparameterized neural net- works: Symmetries and invariances, 2021

    Berfin ¸ Sim¸ sek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural net- works: Symmetries and invariances, 2021. URLhttps://arxiv.org/abs/2105.12221

    2021

  32. [40]

    Noether’s learning dynamics: Role of symmetry breaking in neural networks, 2021

    Hidenori Tanaka and Daniel Kunin. Noether’s learning dynamics: Role of symmetry breaking in neural networks, 2021. URLhttps://arxiv.org/abs/2105.02716. 12 A Proof and Extension of Theorem 1 We give the derivation of Theorem 1, state the matrix-valued refinement, and record the...

    2021

  33. [41]

    strict hierarchy

    =O(∥X∥ L+q−1)sharply; the sharpness argument of (24) (withh (q) σ ̸= 0 for every Class B activation used in this paper) shows the bound is attained, not merely an upper bound. Proposition 13 is the nonlinear analog of deep-linear balance: the Class B bound holds to order ∥X∥ L...

  34. [42]

    At each quadrature node, the forward trajectory ˙xν =f ν(xν)and backward adjoint˙p ν =−(∂ xfν)⊤pν are integrated jointly. The Jacobian∂ xfν is constructed analytically in closed form from the reduced-variable equations of motion (layer scales, off-block amplitudes, cross-block...

Pith tools

Reviewed July 1, 2026 · model on record in the stance chip above.