Pith. sign in

REVIEW 4 minor 56 references

Even in the simplest deep linear chains, the best depth-wise learning-rate scaling depends on the data, and data-agnostic rules fail to transfer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 16:02 UTC pith:45C7SB4A

load-bearing objection Clean, elementary counter-example: even scalar linear chains need a data-dependent depth correction for LR transfer; data-agnostic L^{-1} fails.

arxiv 2607.07884 v1 pith:45C7SB4A submitted 2026-07-08 cs.LG

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

Yedi Zhang , Peter E. Latham , Leena Chennuru Vankadara , Andrew Saxe This is my paper
classification cs.LG
keywords deep linear networkslearning-rate scalinghyperparameter transfergradient flowresidual networksexact dynamicsdepth dependence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern practice often assumes that a single depth-wise learning-rate rule, free of data statistics, will transfer from shallow to deep models. This short note shows that the assumption already fails in the most elementary deep network: a product of scalar weights. Exact gradient-flow solutions for any depth reveal that the largest stable learning rate must contain a correction that depends on the ratio of input-output to input-input moments. Under that data-dependent scaling the relative-weight dynamics becomes independent of the data and only weakly dependent on depth, so every depth—including the infinite-depth limit—converges at the same linear rate. Data-agnostic power laws, even when calibrated at one depth, systematically over- or under-scale the rate at other depths. The same data dependence appears in scalar residual chains of block depth one and two. The result supplies a clean counter-example to the hope that depth-wise hyperparameter transfer can be made fully data-agnostic, and it isolates the precise data correction that restores transfer.

Core claim

In a depth-L scalar linear network the maximal stable learning rate scales as η ∝ L⁻¹ r⁻²⁺²/L, where r = μ_yx/μ_xx is a data statistic; the extra finite-depth factor is data-dependent, so pure power-law rules fail to transfer across depths, while the data-dependent rule yields dynamics independent of data and a constant linear convergence rate for every depth including infinity.

What carries the argument

The conservation law that keeps equal initial weights equal reduces the L-dimensional gradient flow to a single ODE for the relative total weight α; evaluating the sharpness of that ODE at the global minimum produces the exact maximal stable learning rate, after which the α-dynamics becomes data-free.

Load-bearing premise

All layers start with identical positive weights, so the multi-layer dynamics collapses to a single ordinary differential equation whose curvature can be read off at the minimum.

What would settle it

Train scalar linear chains of several depths on two datasets with markedly different r = μ_yx/μ_xx, transfer a learning rate tuned at one depth by a pure L⁻¹ rule, and check whether the deeper nets remain stable and converge at the predicted rate; if they do for every r, the claimed data dependence is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper studies continuous-time gradient descent on the scalar linear chain f(x)=\prod_l w_l x (and residual variants with block depth one and two). Using the conservation law under equal positive initialization, the L-dimensional dynamics reduce to a one-dimensional ODE. Evaluating the sharpness of the loss at the global minimum yields a maximal stable learning rate that scales as \eta=\tau^{-1} \mu_xx^{-1} L^{-1} r^{-2+2/L} with r=\mu_yx/\mu_xx (Eq. 5). Under this data-dependent scaling the relative total weight \alpha obeys a data-independent ODE (Eq. 6) whose solutions are expressed via the hypergeometric function (finite L) or the Lambert W function (L\to\infty). The same data dependence appears for residual architectures (Eqs. 31, 37). Finite-horizon loss sweeps (Fig. 2) confirm that the data-dependent rule transfers across depth while pure power-law rules do not.

Significance. The note supplies a clean, fully solvable counter-example to the claim that data-agnostic depth-wise learning-rate scalings transfer. The derivations are elementary, the exact time-course solutions (hypergeometric and Lambert W) are new refinements of classical deep-linear results, and the residual extensions show the phenomenon is not an artifact of the pure product architecture. Code that reproduces the figures is provided. While the model class is minimal, the explicit data-dependent correction and the demonstration that it cannot be absorbed into a calibrated power law are useful for the hyperparameter-transfer literature.

minor comments (4)
  1. [Abstract / §3] The abstract and introduction assert that data-agnostic rules 'fail to transfer,' yet the final paragraph of §3 notes that the data dependence becomes weak for large L and that transfer from intermediate depths under L^{-1} may still suffice. A single clarifying sentence in the abstract would prevent over-statement.
  2. [Figure 2] Figure 2 caption should state the precise values of r=\mu_yx/\mu_xx used in each panel so that the reader can verify the direction of the mismatch under the agnostic rule.
  3. [Appendix B.2] The remark after Eq. (16) on the order of differentiation versus substitution is important for reproducibility; it would be clearer if moved into the main text near the sharpness calculation rather than left only in the appendix.
  4. [§2] A brief sentence noting that the one-dimensional reduction fails for unbalanced or sign-changing initializations (already flagged in footnote 1) would make the scope of the claim fully explicit for readers who skip the footnote.

Circularity Check

0 steps flagged

No circularity: maximal stable LR is the Hessian at the known global min; dynamics follow by separation of variables with data as explicit inputs.

full rationale

The paper's central derivation is self-contained and non-circular. The maximal stable learning rate (Eq. 5 / Eq. 16) is obtained by evaluating the second derivative of the reduced loss along the balanced-weight manifold at the analytically known global minimum w1 = (μ_yx/μ_xx)^{1/L}; this is a direct calculation, not a fit. Substituting that η into the reduced ODE yields the data-independent dynamics τ α̇ = α^{2-2/L}(1-α) (Eq. 6), which is solved by separation of variables (hypergeometric / Lambert W). Data moments μ_yx, μ_xx appear as explicit inputs, never as free parameters tuned to produce the claimed transfer. Residual-block extensions (Eqs. 31, 37) follow the same Hessian-at-minimum construction. Finite-horizon sweeps in Fig. 2 are empirical corroboration, not fitted predictions re-labeled as theory. Self-citations to Saxe et al. (2014; 2019) supply prior exact solutions that the present note refines; they are not load-bearing uniqueness theorems that force the data-dependent claim. No step reduces by construction to its own input.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The central claim rests on standard gradient-flow analysis of a scalar product network under balanced positive initialization and ℓ2 loss. No free parameters are fitted to data; τ is an arbitrary positive time-constant that simply sets units. The only non-standard modeling choices are the equal-initialization assumption and the continuous-time approximation, both stated explicitly.

free parameters (1)
  • τ (time constant) = any value in (0.5, ∞) for stability
    Arbitrary positive scalar that sets the overall time unit of the rescaled dynamics; not fitted to any dataset.
axioms (3)
  • domain assumption Gradient flow (continuous-time limit of gradient descent) accurately describes the stable discrete dynamics when η < 2/S.
    Invoked throughout Section 2 and justified by reference to Cohen et al. (2025); standard in the deep-linear literature.
  • domain assumption All layer weights are initialized equal and positive, remaining equal by the conservation law d/dt(w_l² - w_l'²) = 0.
    Stated after Eq. 3; reduces the system to a single ODE whose sharpness supplies the claimed scaling.
  • domain assumption Training uses mean-squared error on a finite dataset summarized by the two moments μ_yx and μ_xx.
    Standard ℓ2 loss; moments appear already in Eq. 2.

pith-pipeline@v1.1.0-grok45 · 20034 in / 2211 out tokens · 26238 ms · 2026-07-10T16:02:43.594470+00:00 · methodology

0 comments
read the original abstract

In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.

Figures

Figures reproduced from arXiv: 2607.07884 by Andrew Saxe, Leena Chennuru Vankadara, Peter E. Latham, Yedi Zhang.

Figure 1
Figure 1. Figure 1: The loss landscape of a scalar linear network has a sharper global minimum as the depth [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Learning rates transfer under the optimal data-dependent scaling (left two columns), but [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dynamics of α(t) with different depths L and learning rates η. The learning rate η is given by Equation (5) with τ −1 = 1, 1.5, 1.95, 2.05 for the four panels from left to right. When 0 < τ −1 ≤ 1, the gradient descent dynamics is monotonic and well described by the gradient flow dynamics. When 1 < τ −1 < 2, the gradient descent dynamics is oscillatory but converging. When τ −1 ≥ 2, the gradient descent dy… view at source ↗
Figure 4
Figure 4. Figure 4: Dynamics of total weights (top row) and loss (bottom row) with different depths [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The loss landscape of scalar linear residual networks with block depth one. Similar to [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Loss trajectories of deep scalar linear residual networks with block depth one with different [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The loss landscape of scalar linear residual networks with block depth two. Specifically, [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Loss trajectories of deep scalar linear residual networks with block depth two with different [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages · 1 internal anchor

  1. [1]

    Neural networks and principal component analysis: Learning from examples without local minima , journal =

    Pierre Baldi and Kurt Hornik , keywords =. Neural networks and principal component analysis: Learning from examples without local minima , journal =. 1989 , issn =. doi:https://doi.org/10.1016/0893-6080(89)90014-2 , url =

    work page doi:10.1016/0893-6080(89)90014-2 1989

  2. [2]

    Gen , volume=

    Effect of batch learning in multilayer neural networks , author=. Gen , volume=

    work page

  3. [3]

    Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages =

    Understanding the difficulty of training deep feedforward neural networks , author =. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages =. 2010 , editor =

    work page 2010

  4. [4]

    International Conference on Learning Representations , year=

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , author=. International Conference on Learning Representations , year=

    work page

  5. [5]

    Exponential expressivity in deep neural networks through transient chaos , url =

    Poole, Ben and Lahiri, Subhaneil and Raghu, Maithra and Sohl-Dickstein, Jascha and Ganguli, Surya , booktitle =. Exponential expressivity in deep neural networks through transient chaos , url =

    work page

  6. [6]

    International Conference on Learning Representations , year=

    Deep Information Propagation , author=. International Conference on Learning Representations , year=

    work page

  7. [7]

    Zico Kolter , booktitle =

    Brandon Amos and J. Zico Kolter , booktitle =. 2017 , editor =

    work page 2017

  8. [8]

    Inverse Problems , abstract =

    Haber, Eldad and Ruthotto, Lars , title =. Inverse Problems , abstract =. 2017 , month =. doi:10.1088/1361-6420/aa9a90 , url =

    work page doi:10.1088/1361-6420/aa9a90 2017

  9. [9]

    Chen, Ricky T. Q. and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David K , booktitle =. Neural Ordinary Differential Equations , url =

    work page

  10. [10]

    International Conference on Learning Representations , year=

    An analytic theory of generalization dynamics and transfer learning in deep linear networks , author=. International Conference on Learning Representations , year=

    work page

  11. [11]

    Dynamical Isometry and a Mean Field Theory of

    Xiao, Lechao and Bahri, Yasaman and Sohl-Dickstein, Jascha and Schoenholz, Samuel and Pennington, Jeffrey , booktitle =. Dynamical Isometry and a Mean Field Theory of. 2018 , editor =

    work page 2018

  12. [12]

    Zico and Koltun, Vladlen , booktitle =

    Bai, Shaojie and Kolter, J. Zico and Koltun, Vladlen , booktitle =. Deep Equilibrium Models , url =

    work page

  13. [13]

    Mathematics , VOLUME =

    Hanin, Boris , TITLE =. Mathematics , VOLUME =. 2019 , NUMBER =

    work page 2019

  14. [14]

    Saxe and James L

    Andrew M. Saxe and James L. McClelland and Surya Ganguli , title =. Proceedings of the National Academy of Sciences , volume =. 2019 , doi =

    work page 2019

  15. [15]

    Proceedings of the Thirty-Second Conference on Learning Theory , pages =

    Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks , author =. Proceedings of the Thirty-Second Conference on Learning Theory , pages =. 2019 , editor =

    work page 2019

  16. [16]

    Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks , url =

    Gidel, Gauthier and Bach, Francis and Lacoste-Julien, Simon , booktitle =. Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks , url =

    work page

  17. [17]

    International Conference on Learning Representations , year=

    The Implicit Bias of Depth: How Incremental Learning Drives Generalization , author=. International Conference on Learning Representations , year=

    work page

  18. [18]

    2020 , eprint=

    Scaling Laws for Neural Language Models , author=. 2020 , eprint=

    work page 2020

  19. [19]

    An empirical analysis of compute-optimal large language model training , url =

    Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and de Las Casas, Diego and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and Hennigan, Thomas and Noland, Eric and Millican, Katherine and van den Driessche, George and Damoc, Bogdan and Guy, Aurelia and Osindero, Simon and...

    work page

  20. [20]

    Advani and Andrew M

    Madhu S. Advani and Andrew M. Saxe and Haim Sompolinsky , keywords =. High-dimensional dynamics of generalization error in neural networks , journal =. 2020 , issn =. doi:https://doi.org/10.1016/j.neunet.2020.08.022 , url =

    work page doi:10.1016/j.neunet.2020.08.022 2020

  21. [21]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Understanding the Dynamics of Gradient Flow in Overparameterized Linear models , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    work page 2021

  22. [22]

    Exact learning dynamics of deep linear networks with prior knowledge , url =

    Braun, Lukas and Domin\'. Exact learning dynamics of deep linear networks with prior knowledge , url =. Advances in Neural Information Processing Systems , editor =

    work page

  23. [23]

    International Conference on Learning Representations , year=

    A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , author=. International Conference on Learning Representations , year=

    work page

  24. [24]

    Proceedings of the 35th International Conference on Machine Learning , pages =

    On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization , author =. Proceedings of the 35th International Conference on Machine Learning , pages =. 2018 , editor =

    work page 2018

  25. [25]

    International Conference on Learning Representations , year=

    Gradient descent aligns the layers of deep linear networks , author=. International Conference on Learning Representations , year=

    work page

  26. [26]

    Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced , url =

    Du, Simon S and Hu, Wei and Lee, Jason D , booktitle =. Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced , url =

    work page

  27. [27]

    Neural Tangent Kernel: Convergence and Generalization in Neural Networks , url =

    Jacot, Arthur and Gabriel, Franck and Hongler, Clement , booktitle =. Neural Tangent Kernel: Convergence and Generalization in Neural Networks , url =

    work page

  28. [28]

    Learning dynamics of deep linear networks with multiple pathways , url =

    Shi, Jianghong and Shea-Brown, Eric and Buice, Michael , booktitle =. Learning dynamics of deep linear networks with multiple pathways , url =

    work page

  29. [29]

    International Conference on Learning Representations , year=

    Neural Networks as Kernel Learners: The Silent Alignment Effect , author=. International Conference on Learning Representations , year=

    work page

  30. [30]

    On Lazy Training in Differentiable Programming , url =

    Chizat, L\'. On Lazy Training in Differentiable Programming , url =. Advances in Neural Information Processing Systems , editor =

    work page

  31. [31]

    Proceedings of Thirty Third Conference on Learning Theory , pages =

    Kernel and Rich Regimes in Overparametrized Models , author =. Proceedings of Thirty Third Conference on Learning Theory , pages =. 2020 , editor =

    work page 2020

  32. [32]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Curvature-corrected learning dynamics in deep neural networks , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    work page 2020

  33. [33]

    Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity

    Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity , author=. arXiv preprint arXiv:2106.15933 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [34]

    The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit , url =

    Noci, Lorenzo and Li, Chuning and Li, Mufan and He, Bobby and Hofmann, Thomas and Maddison, Chris J and Roy, Dan , booktitle =. The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit , url =

    work page

  35. [35]

    Journal of Machine Learning Research , year =

    Raphaël Berthier , title =. Journal of Machine Learning Research , year =

    work page

  36. [36]

    The Eleventh International Conference on Learning Representations , year=

    Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions , author=. The Eleventh International Conference on Learning Representations , year=

    work page

  37. [37]

    Saddle-to-Saddle Dynamics in Diagonal Linear Networks , url =

    Pesme, Scott and Flammarion, Nicolas , booktitle =. Saddle-to-Saddle Dynamics in Diagonal Linear Networks , url =

    work page

  38. [38]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Width and Depth Limits Commute in Residual Networks , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    work page 2023

  39. [39]

    Transactions on Machine Learning Research , issn=

    On the infinite-depth limit of finite-width neural networks , author=. Transactions on Machine Learning Research , issn=. 2023 , url=

    work page 2023

  40. [40]

    2023 , eprint=

    Depth Dependence of P Learning Rates in ReLU MLPs , author=. 2023 , eprint=

    work page 2023

  41. [41]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Understanding Unimodal Bias in Multimodal Deep Linear Networks , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    work page 2024

  42. [42]

    The Fourteenth International Conference on Learning Representations , year=

    Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures , author=. The Fourteenth International Conference on Learning Representations , year=

    work page

  43. [43]

    Scaling ResNets in the Large-depth Regime , journal =

    Pierre Marion and Adeline Fermanian and G. Scaling ResNets in the Large-depth Regime , journal =. 2025 , volume =

    work page 2025

  44. [44]

    Infinite Limits of Multi-head Transformer Dynamics , url =

    Bordelon, Blake and Chaudhry, Hamza and Pehlevan, Cengiz , booktitle =. Infinite Limits of Multi-head Transformer Dynamics , url =. doi:10.52202/079017-1130 , editor =

    work page doi:10.52202/079017-1130

  45. [45]

    The Twelfth International Conference on Learning Representations , year=

    Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit , author=. The Twelfth International Conference on Learning Representations , year=

    work page

  46. [46]

    Super Consistency of Neural Network Landscapes and Learning Rate Transfer , url =

    Noci, Lorenzo and Meterez, Alexandru and Hofmann, Thomas and Orvieto, Antonio , booktitle =. Super Consistency of Neural Network Landscapes and Learning Rate Transfer , url =. doi:10.52202/079017-3262 , editor =

    work page doi:10.52202/079017-3262

  47. [47]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Scaling Exponents Across Parameterizations and Optimizers , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    work page 2024

  48. [48]

    Tensor Programs

    Greg Yang and Dingli Yu and Chen Zhu and Soufiane Hayou , booktitle=. Tensor Programs. 2024 , url=

    work page 2024

  49. [49]

    The Thirteenth International Conference on Learning Representations , year=

    Understanding Optimization in Deep Learning with Central Flows , author=. The Thirteenth International Conference on Learning Representations , year=

    work page

  50. [50]

    The Thirteenth International Conference on Learning Representations , year=

    From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks , author=. The Thirteenth International Conference on Learning Representations , year=

    work page

  51. [51]

    The Thirteenth International Conference on Learning Representations , year=

    Three Mechanisms of Feature Learning in a Linear Network , author=. The Thirteenth International Conference on Learning Representations , year=

    work page

  52. [52]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , editor =

    work page 2025

  53. [53]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Don't be lazy: CompleteP enables compute-efficient deep transformers , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    work page

  54. [54]

    2025 , eprint=

    The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagrams , author=. 2025 , eprint=

    work page 2025

  55. [55]

    2025 , eprint=

    On the inductive bias of infinite-depth ResNets and the bottleneck rank , author=. 2025 , eprint=

    work page 2025

  56. [56]

    2026 , eprint=

    The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks , author=. 2026 , eprint=

    work page 2026