Pith. sign in

REVIEW 2 major objections 2 minor 50 references

Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

T0 review · 2 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient flow on two-layer ReLU networks with orthogonal data converges in the small-initialization limit to a saddle-to-saddle jump process that activates one neuron per saddle.

desk verdict This paper derives a saddle-to-saddle jump process for gradient flow on orthogonal data, giving incremental neuron activation plus a new sqrt(n) scaling for the learned interpolator norm. read the letter →

arxiv 2605.27097 v1 pith:IB2775ON submitted 2026-05-26 cs.LG stat.ML

James Town , Etienne Boursier , Ben Lewis , Matthias Englert , Ranko Lazic This is my paper
classification cs.LGstat.ML
keywords ReLUnetworksgradientflowimplicitbiasoverparameterizationincrementallearningsaddle-to-saddledynamicsorthogonaldatainterpolation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies gradient flow dynamics of two-layer ReLU networks trained from small initialization on orthogonal data. It establishes that as the initialization scale tends to zero the continuous flow converges to a discrete saddle-to-saddle jump process in which a fresh neuron activates at each saddle. This incremental characterization recovers the known high-probability interpolation result that width m of order log n suffices, and it yields the new implicit-bias statement that the squared ℓ₂-norm of the learned interpolator scales as √n, matching the minimal-norm interpolator up to a constant factor.

What carries the argument

The saddle-to-saddle jump process, in which the flow lingers near saddles before a new neuron activates and the trajectory jumps to the next saddle.

What would settle it

Numerical integration of the gradient flow on orthogonal data that fails to exhibit one-neuron-per-saddle activation or produces a final squared norm far from order √n.

Watch

Extended reading notes

Core claim

Gradient flow on mildly overparameterized two-layer ReLU networks with orthogonal inputs converges in the small-initialization limit to a saddle-to-saddle jump process in which a new neuron activates at each saddle; this process produces an interpolating solution whose squared ℓ₂-norm scales as √n.

Load-bearing premise

The training data consists of exactly orthogonal vectors.

Editorial extensions

If this is right

  • The network reaches exact interpolation of the training set with high probability once its width satisfies m ≳ log(n).
  • The interpolating solution obtained by the flow has squared ℓ₂-norm of order √n and is therefore within a constant factor of the minimal-norm interpolator.
  • Mild overparameterization suffices for the network to reach an interpolator whose complexity is of the same order as the optimal one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same saddle-to-saddle structure may persist under small perturbations of orthogonality, allowing the incremental phenomenon to hold for nearly orthogonal data.
  • Analogous limiting jump processes could govern training dynamics in deeper ReLU networks or with other piecewise-linear activations when data satisfy suitable separation conditions.
  • The √n-norm scaling indicates that the implicit bias of gradient flow from tiny initialization already selects near-minimal solutions without needing explicit regularization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper analyzes gradient flow dynamics of two-layer ReLU networks trained on orthogonal data from small initialization. It proves that as the initialization scale tends to zero the flow converges to a saddle-to-saddle jump process in which neurons activate incrementally one at a time. This recovers the known high-probability interpolation result for width m ≳ log(n) and derives a new implicit-bias statement that the squared ℓ₂-norm of the learned interpolator scales as √n (within a constant factor of the minimal-norm interpolator).

Significance. If the central claims hold, the work supplies the first rigorous characterization of an incremental learning process for ReLU networks and a novel quantitative implicit-bias result in the mildly overparameterized regime. The orthogonal-data assumption is used explicitly to decouple neuron contributions, enabling an explicit limiting ODE and jump process; this is a clear methodological strength. Recovery of the Dana et al. interpolation threshold together with the new √n scaling provides concrete, falsifiable predictions about both convergence and solution complexity.

major comments (2)
  1. [§4.2] §4.2, the passage from the finite-scale gradient-flow ODE to the saddle-to-saddle jump process: the argument that the initialization-scale limit produces well-defined jump times at which a new neuron activates while previous neurons remain at equilibrium relies on orthogonality decoupling; the verification that the resulting jump process indeed yields the claimed √n norm scaling (rather than a different constant or scaling) is load-bearing for the implicit-bias claim and requires an explicit check against the minimal-norm interpolator on orthogonal data.
  2. [§5.1] §5.1, Eq. (27) (or the corresponding statement of the norm result): the derivation that the incremental activation produces ||w||² ~ √n assumes a specific ordering of activation times; if the ordering is random or data-dependent, the constant factor relative to the minimal-norm solution must be shown to remain O(1) uniformly, otherwise the “within a constant factor” claim is not fully supported.
minor comments (2)
  1. The citation to Dana et al. (2025) appears in the abstract and introduction; the reference list should give the full arXiv identifier and title for consistency with journal style.
  2. Notation for the network width m and sample size n is introduced early but the dependence on the initialization scale ε is sometimes written as o(1) without an explicit ε→0 statement in every theorem; a uniform convention would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful review and valuable suggestions. We address each major comment below, providing clarifications and indicating where revisions will be made to strengthen the manuscript.

read point-by-point responses

  1. Referee: [§4.2] §4.2, the passage from the finite-scale gradient-flow ODE to the saddle-to-saddle jump process: the argument that the initialization-scale limit produces well-defined jump times at which a new neuron activates while previous neurons remain at equilibrium relies on orthogonality decoupling; the verification that the resulting jump process indeed yields the claimed √n norm scaling (rather than a different constant or scaling) is load-bearing for the implicit-bias claim and requires an explicit check against the minimal-norm interpolator on orthogonal data.

    Authors: The orthogonality of the data is explicitly used to decouple the neuron dynamics, enabling the derivation of the limiting jump process as described in §4.2. For the norm scaling, the proof establishes that the incremental process produces an interpolator with ||w||² = Θ(√n). On orthogonal data, the minimal-norm interpolator also satisfies ||w_min||² = Θ(√n). We will include an explicit comparison in the revised version to verify the constant factor is bounded. revision: partial

  2. Referee: [§5.1] §5.1, Eq. (27) (or the corresponding statement of the norm result): the derivation that the incremental activation produces ||w||² ~ √n assumes a specific ordering of activation times; if the ordering is random or data-dependent, the constant factor relative to the minimal-norm solution must be shown to remain O(1) uniformly, otherwise the “within a constant factor” claim is not fully supported.

    Authors: The activation ordering in the jump process is data-dependent but, under the orthogonal data assumption, the total squared norm is the sum of contributions from each activated neuron, each adding a term of order 1/√n. This sum is invariant to the order of summation, yielding the same Θ(√n) scaling uniformly. We will add a remark in §5.1 clarifying that the constant factor remains O(1) independent of the activation sequence. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation self-contained on orthogonal data

full rationale

The paper derives the saddle-to-saddle jump process from the gradient-flow ODE limit as initialization scale tends to zero, under the explicit orthogonal-data assumption that decouples neuron contributions. From this process it recovers (via external citation to Dana et al.) the interpolation probability m ≳ log n and separately derives the novel ||·||² ~ √n scaling. No equation is shown to equal its own input by construction, no fitted parameter is relabeled as a prediction, and the sole citation is to non-overlapping prior work. The central claims therefore remain independent of the target results.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

Review performed from abstract only; the paper invokes standard gradient-flow analysis and the orthogonality assumption but supplies no further free parameters or invented entities in the visible text.

assumptions (2)
  • domain assumption Continuous-time gradient flow governs the training dynamics
    The limiting behavior is analyzed under gradient flow as initialization scale tends to zero.
  • domain assumption Training inputs are pairwise orthogonal
    The title and abstract restrict the setting to orthogonal data to obtain the saddle-to-saddle structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias." pith.science (2026). https://pith.science/paper/IB2775ON

@misc{pith2026260527097,
  author       = {Pith},
  title        = {Pith review of: Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IB2775ON}},
  note         = {Machine review of arXiv:2605.27097}
}
abstract

The successful training of neural networks hinges on the use of first order optimization methods, yet the theoretical characterization of these methods remains incomplete. This is especially true in settings with mild overparameterization. In this work, we study the gradient flow dynamics of two-layer ReLU networks from small initialization with orthogonal training data. We prove the limiting flow converges to a saddle-to-saddle jump process as the initialization scale tends to zero, revealing an incremental learning phenomenon in which a new neuron activates at each saddle. This analysis recovers the known result of Dana et al. (2025, arXiv:2502.16977) that the network interpolates the training data with high probability as soon as $m \gtrsim \log(n)$, where $m$ is the network width and $n$ is the number of training samples. This incremental process characterization also allows us to derive a novel implicit bias result: the learned interpolator has a squared $\ell_2$-norm scaling as $\sqrt{n}$, which is within a constant factor of the minimal $\ell_2$-norm interpolator. More broadly, our work provides the first rigorous proof of an incremental learning process for ReLU networks, whilst suggesting mildly overparameterized networks can converge to interpolating solutions whose complexity is of the same order as that of the optimal interpolator.

Figures

Figures reproduced from arXiv: 2605.27097 by the authors.

Figure 1
Figure 1. We train a two-layer ReLU network of six neurons on 64 orthonormal data points in 64 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Timeline of the training dynamics about the jump from saddle [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. The same as the right plot of fig. 1 but with the y-axis on a logarithmic scale. The neuron norms increase exponentially. When new data points are fitted, the rate of this exponential increase reduces for the remaining neurons that were active on at least one of those newly fitted data points. significant proportion of the label, the output of the network at the data point very rapidly closes in on the label value. … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: For the same training as in fig. 1, these polar plots show the norm of each neuron (on a logarithmic scale) as well as the direction of the neuron projected onto a random two-dimensional plane, at different points in time. The projections of the neuron directions from …
Figure 5
Figure 5. Figure 5: We train a two-layer ReLU network with increasing data dimensionality and number of data [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: We train two-layer ReLU networks of varying widths [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 2 canonical work pages

  1. [1]

    Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias

    2, 20 Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias. In NeurIPS, pages 12978–12991, 2021. 3 Wesley J. Maddox, Gregory W. Benton, and Andrew Gordon Wilson. Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited. CoRR, abs/2003.02139, 2020. 4 Har...

    work page arXiv 2021

  2. [2]

    On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime

    4 Stephan Wojtowytsch. On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime. CoRR, abs/2005.13530, 2020. 2 Blake E. Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and Rich Regimes in Overparametrized Models. In COLT, pages 3635–3673,

    work page arXiv 2005

  3. [3]

    Wide Residual Networks

    2 Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016. 1 14 Appendix Table of contents A Experiments 15 A.1 Experimental details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Additional experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B Technical backgro...

    2016

  4. [4]

    ∀j ∈ N (k) U , ℓ◦ j(κ) + logα(∥ewj(κ)∥) < ξ

  5. [5]

    ∀j ∈ N (k) F , n D(k) j − ∥ewj(κ)∥2 < α δ

  6. [6]

    ∀i ∈ S(k) F , |heθ(κ)(xi) − yi| < α δ

  7. [7]

    In the above, and throughout the proof, we use ∥·∥ to denote the ℓ2-norm

    ∀j ∈ [m], ew ⊤ j (κ)D (k) j > 1 − αδ. In the above, and throughout the proof, we use ∥·∥ to denote the ℓ2-norm. The reader should note that δ depends on κ in Hypothesis 1; in particular we have limκ→0 δ = 0. This does not present a problem, since the proof is of convergence on compact subsets of R which exclude (tk)p k=1. Therefore, κ may be viewed as a q...

  8. [8]

    ∀j ∈ Js, s jew ⊤ j (κ)D (0) j > 1 − αδ

Show all 50 references
  1. [9]

    ∀j ∈ Js, ∀t ∈ (0, κ), ℓ◦ j(t) + logα(∥ewj∥) < ξ . 23

  2. [10]

    The monotonicity of the gradient flow means there exists a maximum rate of growth it is possible for any neuron to achieve

    ∀i ∈ I−s, ∀j ∈ Js, ew⊤ j (κ)xi ≤ 0. The monotonicity of the gradient flow means there exists a maximum rate of growth it is possible for any neuron to achieve. Therefore, the upper bound on κ appears as a technicality to ensure that even if a neuron were to have grown at this ...

  3. [11]

    ∀i ∈ S(k) F , ∀t ∈ [κ, τ1], |heθ(xi) − yi| < α ε

  4. [12]

    ∀j ∈ [m], ∀t ∈ [κ, τ1], ew ⊤ j D (k) j > 1 − αε/2

  5. [13]

    ∀j ∈ N (k) F , ∀t ∈ [κ, τ1], r n D(k) j − αε/4 < ∥ewj∥ < r n D(k) j + αε/4

  6. [14]

    tk+1 − 3ε 2 D(k) j⋆ < τ 1 < t k+1 − ε 2 D(k) j⋆

  7. [15]

    ∀j ∈ N (k) U , ∀t ∈ [κ, τ1], ℓ◦ j(t) + logα(∥ewj∥) < ε 2. Proof. Proof of Item 1 Lemma 4. Let δ be such that the conditions of Hypothesis 1 hold. In order to check that |heθ(xi) − yi| < α ε, we will bound the growth in norm of the quantity E(k) = − 1 n X i∈S(k) F heθ(xi) − yi ...

  8. [16]

    τ2 − τ1 < 3ε D(k) j⋆

  9. [17]

    ∀j ∈ N (k) F , ∀t ∈ [τ1, τ2], r n D(k) j − αε/8 < ∥ewj∥ < r n D(k) j + αε/8

  10. [18]

    ∀i ∈ S(k) F , ∀t ∈ [τ1, τ2], maxi∈S(k) F ew ⊤ j⋆ xi + < α ε/6

  11. [19]

    ∀t ∈ [τ1, τ2], ew ⊤ j⋆ D (k) j⋆ > 1 − αε/12

  12. [20]

    ∀i ∈ S(k) F , ∀t ∈ [τ1, τ2], |heθ(xi) − yi| < α ε/6

  13. [21]

    ∥ewj⋆(τ2)∥ = η. We observe at this point that although Item 4 Lemma 5 would seem to imply a statement on realignment with fitted data, we are required in the proof to obtain Item 3 Lemma 5 first in order to prove Item 4 Lemma 5. This is an interesting result of the fact that a...

  14. [22]

    τ3 − τ2 < ε′ D(k) j⋆

  15. [23]

    ∀j ∈ N (k) F , ∀t ∈ [τ2, τ3], r n D(k) j − αε′/2 < ∥ewj∥ < r n D(k) j + αε′/2

  16. [24]

    ∀i ∈ S(k) F , ∀t ∈ [τ2, τ3], |heθ(xi) − yi| < α ε′

  17. [25]

    ∀i ∈ S(k) F , ∀t ∈ [τ2, τ3], maxi∈S(k) F ew ⊤ j⋆ xi + < α 5ε′/4

  18. [26]

    ∀t ∈ [τ2, τ3], ew ⊤ j⋆ D (k) j⋆ > 1 − αε′

  19. [27]

    ∥ewj⋆(τ3)∥2 = n D(k) j⋆ − αε′/4. Proof. We work out a useful preliminary bound to begin this section. We upper bound the network output on i ∈ S(k) j⋆ : heθ(xi) (1) < ew ⊤ j⋆ xi∥ewj⋆ ∥2 + mα2cU (2) <   yi n D(k) j⋆ + √ 2αε′/2   n D(k) j⋆ − αε′/4 + mα2cU (3) < y i − αε′/3. ...

  20. [28]

    tk+1 < τ 4 < t k+1 + 4ε D(k) j⋆ + ε′ D(k) j⋆

  21. [29]

    ∀i ∈ S(k+1) F , ∀t ∈ [τ3, τ4], |heθ(xi) − yi| < α ε′/6

  22. [30]

    ∀j ∈ N (k+1) F , ∀t ∈ [τ3, τ4], r n D(k) j − αε′/8 < ∥ewj∥ < r n D(k) j + αε′/8

  23. [31]

    ∀j ∈ N (k+1) U , ew ⊤ j (τ4)D (k+1) j > 1 − αε′/15

  24. [32]

    ∀i ∈ N (k+1) U , ∀t ∈ [τ1, τ4], ℓ◦ j(t) + logα(∥ewj∥) < ζ j(ε, ε′), where ζj(ε, ε′) := ε 2 + D(k) j + q 2L(0) n 9ε 2 D(k) j⋆ + ε′ D(k) j⋆ . Proof. The reader should note it will be convenient to derive the bounds in Item 1 Lemma 7 as conse- quences of analyses for the other po...

  25. [33]

    ∀i ∈ S(k+1) F , ∀t ∈ [τ4, τ5], |heθ(xi) − yi| < α ε′/8. 52

  26. [34]

    ∀j ∈ [m], ∀t ∈ [τ4, τ5], ew ⊤ j D (k+1) j > 1 − αε′/32

  27. [35]

    ∀j ∈ N (k+1) F , ∀t ∈ [τ4, τ5], r n D(k) j − αε′/10 < ∥ewj∥ < r n D(k) j + αε′/10

  28. [36]

    ∀j ∈ N (k+1) U , ∀t ∈ [τ4, τ5], ℓ◦ j(t) + logα(∥ewj∥) < ζ j(ε, ε′) + ε′, where ζj(ε, ε′) is as defined in Lemma 7. Proof. Given any κ′′ ∈ (tk+1, tk+2), Item 1 Lemma 7 guarantees that we may select ε and ε′ sufficiently small that τ4 < κ ′′. Assume that this has been done and t...

  29. [37]

    ∀j ∈ N (p) U , ℓ◦ j(κ) + logα(∥ewj(κ)∥) < ξ

  30. [38]

    ∀j ∈ N (p) F , n D(p) j − ∥ewj(κ)∥2 < α δ

  31. [39]

    ∀i ∈ [n], |heθ(κ)(xi) − yi| < α δ

  32. [40]

    Then given any κ′ ∈ [κ, ∞),eθα converges uniformly to θ◦ on [κ, κ′]

    ∀j ∈ [m], ew ⊤ j (κ)D (p) j > 1 − αδ. Then given any κ′ ∈ [κ, ∞),eθα converges uniformly to θ◦ on [κ, κ′]. Proof. We note that Item 3 Lemma 9 and the monotonicity of the gradient flow imply that for allj ∈ [m] and for all t ∈ [κ, ∞), ∥Dj∥ < αδ √n. To estimate the change in the...

  33. [41]

    ∀j ∈ N (k+1) U , ℓ◦ j(κ′′) + logα(∥ewj(κ′′)∥) < ξ ′′

  34. [42]

    ∀j ∈ N (k+1) F , n D(k+1) j − ∥ewj(κ′′)∥2 < α δ′′

  35. [43]

    ∀i ∈ S(k+1) F , |heθ(κ′′)(xi) − yi| < α δ′′ . 57

  36. [44]

    ∀j ∈ [m], ew ⊤ j (κ′′)D (k+1) j > 1 − αδ′′ . The proof consists of two applications of Lemma 8: one choice of ε, ε′ in order to get the existence of δ′′ such that the alignment, norm, and loss conditions all hold at time κ′′; the other choice of ε, ε′ in order to get the arbit...

  37. [45]

    ■ Finally, we are able to provide a proof of Theorem 2

    + ε′| < ξ ′′ for all j ∈ N (k+1) U gives the claim. ■ Finally, we are able to provide a proof of Theorem 2. This proof consists of iteratively concluding uniform convergence on arbitrary closed intervals contained in (tk, tk+1) using Hypothesis 1, and then using Lemma 2 to app...

    2017

  38. [46]

    Since |Isj | ≥ min(n−, n+) almost surely by definition, we then have that a column j is zero with probability at most 1 2 min(n−,n+)

    Conditionally on aj, (Ai,j)i∈[n] are independent variables, each of them being either the deterministic variable 0 if i ̸∈ Isj , or a Bernoulli 1 2 i ∈ Isj . Since |Isj | ≥ min(n−, n+) almost surely by definition, we then have that a column j is zero with probability at most 1...

  39. [47]

    Note that two non-zero columns j and j′ are equal if and only if both sgn(aj) = sgn( aj′) and 1(w⊤ j xi) = 1(w⊤ j′ xi) for all i ∈ Isj . All these variables are independent Bernoulli 1 2 , so that a union bound argument again yields that the probability for two non-zero column...

  40. [48]

    F ∈ C 1([T1, T2]) and F ′ is Lipschitz continuous

  41. [49]

    F ≥ 0 and F (T1) = 0

  42. [50]

    Then for every t ∈ [T1, T2], F ′(t) ≤ q√p + F ′(T1) exp(√p(t − T1))

    F ′′(t) ≤ pF (t) + q for almost every t ∈ [T1, T2], where p, q > 0. Then for every t ∈ [T1, T2], F ′(t) ≤ q√p + F ′(T1) exp(√p(t − T1)). Proof. Let G = F + q/p. Then G obeys for almost every t ∈ [T1, T2] the differential inequality G′′ < pG. We compute using this inequality th...

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.