The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks

Nicolas Schreuder; Sholom Schechtman

The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks

T0 review · 0 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Constant-step stochastic subgradient descent on homogeneous ReLU networks converges in direction to the critical points of the normalized margin, extending the max-margin implicit bias to nonsmooth stochastic training.

desk verdict A real first extension of Lyu–Li to nonsmooth stochastic SGD, but the central theorem is conditional on a stability event that the paper never shows can actually occur for constant-step SGD on ReLU networks. read the letter →

arxiv 2502.05668 v3 pith:XF5I7QWD submitted 2025-02-08 cs.LG cs.NEmath.OCstat.ML

Sholom Schechtman , Nicolas Schreuder This is my paper · ORCID

classification cs.LGcs.NEmath.OCstat.ML MSC 68Q3290C2662L20

keywords implicitbiasstochasticsubgradientdescenthomogeneousneuralnetworksmarginmaximizationconservativeset-valuedfieldsdifferentialinclusionsnonsmoothoptimizationconstantstep-sizeSGD

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the implicit bias of gradient-based training—the tendency of overparameterized networks to steer toward max-margin solutions—survives when training is stochastic and the network is nonsmooth. Analyzing constant-step SGD on L-homogeneous ReLU-type networks with exponential or logistic loss, it shows that, whenever the normalized margin stays positive forever (event E), the normalized weight directions converge to the set of critical points of the normalized margin on the unit sphere. When the conservative fields are Clarke subgradients, these critical points are exactly the scaled KKT points of the max-margin problem previously characterized for gradient flow and smooth GD. The proof works by viewing the normalized iterates as a noisy Euler discretization of a conservative-field flow, with an effective step size that shrinks even though the nominal step size is constant.

What carries the argument

The load-bearing object is the normalized margin m(u) = min_i p_i(u) on the unit sphere and its Riemannian conservative field \bar D_s, defined as the tangent projection of the conservative field \bar D of the margin. The paper proves that the normalized SGD recursion is an Euler-like stochastic approximation of the flow \dot u(t) ∈ \bar D_s(u(t)): the recursion is rewritten with an effective step size \bar γ_k that tends to zero and a martingale noise whose cumulative effect vanishes. The convergence to Z_s then follows from the stochastic-approximation theorem for differential inclusions once one shows that −m is a Lyapunov function whose critical values have measure zero by Sard's theorem.

What would settle it

Take a two-layer ReLU network, binary separable data, exponential loss, and constant step size, and run SGD from an initialization where the normalized margin stays positive; if for some such run the normalized directions do not approach any critical point of the normalized margin on the sphere (for instance, if they keep cycling or approach a point where the projected conservative field is nonzero), Theorem 5 is false. A cheaper check is to verify on a simple homogeneous nonsmooth example whether a direction u with m(u)>0 and \bar D_s(u)≠0 can be an accumulation point of the iterates while E holds.

Watch

Extended reading notes

Core claim

The central discovery is that the late-stage dynamics of (stochastic) subgradient descent on homogeneous neural networks are governed by a conservative set-valued field of the normalized margin. The paper constructs the projected field \bar D_s(u) = {v − ⟨v,u⟩u : v ∈ \bar D(u)} on the unit sphere and proves that the normalized iterates u_k approximately follow the differential inclusion \dot u(t) ∈ \bar D_s(u(t)) with perturbations that vanish. Consequently, under the event E = [liminf m(u_k) > 0], the margin m(u_k) converges to a positive limit and dist(u_k, Z_s) → 0, where Z_s is the set of critical points of the normalized margin. When the conservative fields are Clarke subgradients, membership in Z_s coincides with being a scaled KKT point of the max-margin problem, so the result carries the implicit-bias characterization from smooth gradient descent to nonsmooth stochastic subgradient descent.

Load-bearing premise

The argument collapses without the event E = [liminf m(u_k) > 0]: if even one late-stage iterate misclassifies the training data, the norm-growth bound, the bounded noise, and the decaying effective step size all lose their footing, and the paper proves nothing about the directions.

Editorial extensions

If this is right

With constant step size, the directions of both SGD and GD on ReLU-type homogeneous networks converge to critical points of the normalized margin, not just to some zero-error region; behind a single positive-margin iterate plus the stability event, stochasticity does not break the implicit bias.
Under the Clarke-subgradient selection, running nonsmooth SGD is asymptotically equivalent to running margin maximization: any limit direction is a scaled KKT direction of the min-norm problem.
The effective step size \bar γ_k decays polynomially even though the nominal step size γ is constant, because the weight norm grows logarithmically; hence constant-step nonsmooth SGD becomes a convergent stochastic approximation.
The same proof scheme extends to broader losses such as e^{−q^a} and log(1+e^{−q^a}) and to a larger event E1 where the margin may tend to zero, in which case the directions still converge to Z_s but the KKT interpretation can fail.
A byproduct is that the normalized margin itself converges to a positive limit on E, giving a quantitative late-stage description: the norm grows like log k and the loss decays like k^{−ε c1}.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the stability event E could be proven from initialization and overparameterization—a question the paper leaves open—Theorem 5 would become an unconditional statement about constant-step SGD on ReLU networks rather than a conditional one.
The automatic decay of the effective step size suggests that batch-size schedules could be treated as a tuning knob for the rate of convergence to Z_s without altering the limit; this is a testable and currently unexplored implication.
The conservative-field reformulation is not tied to backpropagation artifacts, so the same convergence statement should hold for any homogeneous objective with a min-structure, including multi-class margins or adversarial-margin variants; verifying that would confirm that the mechanism is the margin geometry rather than the specific loss.
When the limit margin is zero (the E1 regime), the limit direction can fail to be a max-margin direction, so the implicit bias is not unconditional: there should be a phase transition between stable (E) and unstable (E1-only) training where the asymptotic solution changes character.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The main thing to know: this paper does what it says on the tin, but only on an event E that is assumed, not established. The result is the first discrete-time, stochastic, nonsmooth extension of Lyu–Li: under Assumptions 1–2, on the event E = [liminf m(u_k) > 0], normalized constant-step SGD iterates converge to the critical set Z_s of the margin on the sphere. That is a genuine advance, not a repackaging. The conservative-field stochastic-approximation viewpoint is a real change of tools, and the appendices do the heavy lifting honestly.

Credit where it is due: the paper is explicit that E is a stability assumption, and it explains why the old descent-lemma arguments cannot rule out leaving the correctly classified region for nonsmooth stochastic dynamics. The proof of Proposition 4 is detailed, and Appendix D checks the Lyapunov and Sard-type conditions needed for the SA theorem. The general setting in Appendix F with vanishing step sizes and the softer event E1 is a good-faith attempt to widen the scope.

Now the soft spots, in proportion. The stress-test concern is not a manufactured flaw: the paper gives no example, simulation, or sufficient condition showing that constant-step SGD on a ReLU network satisfies E. If E is rare or unverifiable, Theorem 5 has no force in the very setting that motivates it. The authors acknowledge this, but the acknowledgment does not make the theorem less conditional. This is the main caveat a reader should carry away. Second, Theorem 5's proof relies on 'minor adaptations' of results from Benaïm and Davis et al. Appendix D sketches the adaptation, but the details are compressed; a referee should ask whether the martingale-noise truncation argument fully goes through, especially the step indexing and the use of Proposition 15. These are referee-level issues, not obvious failures. Also minor: the 'arbitrary step-size' comment is accurate on E but should not be read as a statement about constant-step SGD in general.

Who is this for? Researchers in implicit bias and nonsmooth optimization theory. It deserves a serious referee: the proof structure is sound enough to audit, the question is important, and the conditional limitation is clearly labeled. My recommendation: send it to review, and insist the authors either prove a nontrivial sufficient condition for E or present a concrete network/data setting where E holds. Without that, the result remains a conditional theorem whose applicability is unverified.

Referee Report

0 major / 6 minor

Summary. The paper studies constant-step stochastic subgradient descent (SGD) for binary classification with L-homogeneous nonsmooth neural networks (e.g., ReLU networks). It defines normalized iterates u_k = w_k/||w_k|| and, under an event E = [liminf m(u_k) > 0], proves that m(u_k) converges to a positive limit and dist(u_k, Z_s) -> 0, where Z_s is the set of critical points of the normalized margin on the unit sphere with respect to a conservative set-valued field. The proof interprets normalized SGD as a stochastic approximation of a differential inclusion associated with the margin, proves norm growth (Proposition 3) and an Euler-like decomposition (Proposition 4), and verifies the assumptions of a stochastic approximation result (Appendix D). The paper also discusses the relation to KKT points of the Lyu-Li max-margin problem and provides an extension with vanishing step sizes and more general losses (Appendix F).

Significance. If correct, the result is a substantial extension of Lyu-Li (2020) from smooth GD to nonsmooth and stochastic settings, showing that late-stage SGD converges to margin critical points. The proof is largely self-contained and makes elegant use of conservative fields and stochastic approximation for differential inclusions. The conditional nature of the result on the event E is explicitly acknowledged and is a limitation, not an internal inconsistency. The paper does not provide experiments, but for a theory contribution this is acceptable. Strengths include the explicit Euler-like decomposition, the Sard-type argument for m(Z_s), and the honest discussion of the stability assumption E.

minor comments (6)

[Section 4, Eq. (6)] The main theorem is conditional on the event E = [liminf m(u_k) > 0], and the paper provides no sufficient condition or example showing that E can hold for nonsmooth constant-step SGD. Since the authors explicitly call this a stability assumption and discuss its difficulty, this is a limitation rather than a flaw; I suggest adding a short remark (or a small numerical illustration) in Section 4 or the discussion to indicate settings in which E is plausible, such as the vanishing-step-size framework of Appendix F.
[Appendix B, Remark 14] The example with p1(w)=min(0,w) and p2(w)=min(0,-w) does not satisfy m(w)=0 for w≠0 (for w=1, m(1)=-1); use instead p1(w)=max(0,w) and p2(w)=max(0,-w), which gives m≡0, \bar D(0)=[-1,1], and ∂m(0)={0}, illustrating the claimed distinction.
[Section 4, 'On Assumption 2'] The displayed identity a_i(λw)=λ^L a_i(w) should read a_i(λw)=λ^{L-1}a_i(w), consistent with the (L-1)-homogeneity of the conservative gradient.
[Abstract/Introduction] Please correct 'Up to our knowledge' to 'To the best of our knowledge' and the typo 'sastify' to 'satisfy'.
[Appendix F, Proposition 19] In item 3, the constant is named c1 in the text and then written as c6 in the inequality; the notation should be consistent.
[Section 6, Eq. (15)] The definition of η_{k+1} is correct, but a one-line derivation showing how it arises from rewriting the SGD update would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theorem is conditional on the explicitly stated stability event E, and the convergence proof rests on external stochastic-approximation results rather than on the conclusion being assumed.

full rationale

The central claim (Theorem 5) is a conditional statement: under Assumptions 1 and 2, on the event E = [liminf m(u_k) > 0], the normalized margin converges to a positive limit and the normalized directions approach Z_s. The event E is a stated modeling assumption (Equation (6)), and the paper explicitly notes that it is a stability assumption that is not automatically implied by a single correctly classifying iterate. This is not circular: the conclusion 'dist(u_k, Z_s) -> 0' is not contained in E, and the convergence of m(u_k) goes beyond the liminf assumption. The proof imports convergence results for stochastic approximations of differential inclusions (Benaim 2006; Davis et al. 2020; Bolte and Pauwels 2021), all external to the paper. The identification of Z_s with scaled KKT points of Lyu and Li (2020) is a mathematical equivalence proved in the appendices, not an input. No parameters are fitted, no predictions are produced from their own defining data, and no load-bearing chain of self-citations is used. The concern that E may be hard to verify for nonsmooth constant-step SGD is a question of vacuity or correctness risk, not of circularity, since the paper honestly flags this limitation rather than smuggling the conclusion into the assumptions.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No data-fitting parameters appear in the paper. The constants c1, c2, c3, epsilon, and k0 are existential proof constants, not tuned to data. The result is conditional on event E and Assumptions 1 and 2; all mathematical objects are standard or derived directly from the margin.

assumptions (5)

domain assumption Event E: liminf_k m(u_k) > 0.
Defines the late-stage training phase and is used to prove norm growth, effective step-size decay, and convergence to Z_s. The paper acknowledges it is a stability assumption with no a priori guarantee (Section 4, Equation (6)).
domain assumption Assumption 1: each p_i is positively L-homogeneous, locally Lipschitz, semialgebraic; loss is exponential or logistic.
Core setting of the paper, covering ReLU and LeakyReLU networks without biases (Section 4).
domain assumption Assumption 2: selected fields a_i satisfy a_i(w) in D_i(w), D_i(λw)=λ^{L-1}D_i(w), and L p_i(w)=<a_i(w),w>.
Euler-type relation for the chosen conservative fields; proved for Clarke subgradients and for backpropagation selections; used throughout the proof of Proposition 4 and norm growth.
standard math Sard's theorem and definable stratification.
Used in Appendix D to show m(Z_s) has zero measure, which supplies the Lyapunov condition in the stochastic approximation theorem.
standard math Stochastic approximation convergence theorems for differential inclusions.
Results from Benaim, Davis et al., and Duchi-Ruan are used as black boxes to pass from the Euler-like discretization of Proposition 4 to convergence in Theorem 5.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks." pith.science (2026). https://pith.science/paper/XF5I7QWD

@misc{pith2026250205668,
  author       = {Pith},
  title        = {Pith review of: The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XF5I7QWD}},
  note         = {Machine review of arXiv:2502.05668}
}

read the original abstract

We analyze the implicit bias of constant step stochastic subgradient descent (SGD). We consider the setting of binary classification with homogeneous neural networks - a large class of deep neural networks with ReLU-type activation functions such as MLPs and CNNs without biases. We interpret the dynamics of normalized SGD iterates as an Euler-like discretization of a conservative field flow that is naturally associated to the normalized classification margin. Owing to this interpretation, we show that normalized SGD iterates converge to the set of critical points of the normalized margin at late-stage training (i.e., assuming that the data is correctly classified with positive normalized margin). Up to our knowledge, this is the first extension of the analysis of Lyu and Li (2020) on the discrete dynamics of gradient descent to the nonsmooth and stochastic setting. Our main result applies to binary classification with exponential or logistic losses. We additionally discuss extensions to more general settings.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 39 canonical work pages

[1]

Mart\' i n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Man\' e , Rajat Monga, Sherry Moore, Der...

2015
[2]

Reconciling modern machine-learning practice and the classical bias--variance trade-off

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias--variance trade-off. Proceedings of the National Academy of Sciences, 116 0 (32): 0 15849--15854, 2019

2019
[3]

Dynamics of stochastic approximation algorithms

Michel Bena \" m. Dynamics of stochastic approximation algorithms. In Seminaire de probabilites XXXIII, pages 1--68. Springer, 2006

work page 2006
[4]

Stochastic approximations and differential inclusions

Michel Bena \" m, Josef Hofbauer, and Sylvain Sorin. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization, 44 0 (1): 0 328--348, 2005

work page 2005
[5]

Semianalytic and subanalytic sets

Edward Bierstone and Pierre D Milman. Semianalytic and subanalytic sets. Publications Math \'e matiques de l'IH \'E S , 67: 0 5--42, 1988

work page 1988
[6]

Bolte, A

J. Bolte, A. Daniilidis, A. Lewis, and M. Shiota. Clarke subgradients of stratifiable functions. SIAM Journal on Optimization, 18 0 (2): 0 556--572, 2007

work page 2007
[7]

Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

J \'e r \^o me Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188: 0 19--51, 2021

work page 2021
[8]

Subgradient sampling for nonsmooth nonconvex minimization

J \'e r \^o me Bolte, Tam Le, and Edouard Pauwels. Subgradient sampling for nonsmooth nonconvex minimization. SIAM Journal on Optimization, 33 0 (4): 0 2542--2569, 2023

work page 2023

Show all 55 references

[9]

Stochastic approximation: a dynamical systems viewpoint, volume 9

Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 9. Springer, 2008

2008
[10]

The ode method for convergence of stochastic approximation and reinforcement learning

Vivek S Borkar and Sean P Meyn. The ode method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization, 38 0 (2): 0 447--469, 2000

2000
[11]

An introduction to optimization on smooth manifolds

Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023. doi:10.1017/9781009166164. URL https://www.nicolasboumal.net/book

2023 doi
[12]

Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization

Yuhang Cai, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter Bartlett. Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization. Advances in Neural Information Processing Systems, 37: 0 71306--71351, 2024

2024
[13]

Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory, pages 1305--1338. PMLR, 2020

2020
[14]

Nonsmooth analysis and control theory, volume 178

Francis H Clarke, Yuri S Ledyaev, Ronald J Stern, and Peter R Wolenski. Nonsmooth analysis and control theory, volume 178. Springer-Verlag, New York, 1998

1998
[15]

An introduction to o-minimal geometry

Michel Coste. An introduction to o-minimal geometry. Istituti editoriali e poligrafici internazionali Pisa, 2000

2000
[16]

Stochastic subgradient method converges on tame functions

Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D Lee. Stochastic subgradient method converges on tame functions. Foundations of computational mathematics, 20 0 (1): 0 119--154, 2020

2020
[17]

Curves of descent

Dmitriy Drusvyatskiy, Alexander D Ioffe, and Adrian S Lewis. Curves of descent. SIAM Journal on Control and Optimization, 53 0 (1): 0 114--138, 2015

2015
[18]

Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced

Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018

2018
[19]

Stochastic methods for composite and weakly convex optimization problems

John C Duchi and Feng Ruan. Stochastic methods for composite and weakly convex optimization problems. SIAM Journal on Optimization, 28 0 (4): 0 3229--3259, 2018

2018
[20]

The little book of deep learning

Fran c ois Fleuret. The little book of deep learning. A lovely concise introduction, page 297, 2023

2023
[21]

Complements of subanalytic sets and existential formulas for analytic functions

Andrei Gabrielov. Complements of subanalytic sets and existential formulas for analytic functions. Inventiones mathematicae, 125 0 (1): 0 1--12, 1996

1996
[22]

Projections of semi-analytic sets

Andrei M Gabrielov. Projections of semi-analytic sets. Functional Analysis and its applications, 2 0 (4): 0 282--291, 1968

1968
[23]

Deep learning, volume 196

Ian Goodfellow. Deep learning, volume 196. MIT press, 2016

2016
[24]

Lee, Daniel Soudry, and Nathan Srebro

Suriya Gunasekar, Jason D. Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ss...

2018
[25]

Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018 b

2018
[26]

An invitation to tame optimization

Alexander D Ioffe. An invitation to tame optimization. SIAM Journal on Optimization, 19 0 (4): 0 1894--1917, 2009

1917
[27]

Gradient descent aligns the layers of deep linear networks

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv:1810.02032, 2018 a

2018 arXiv
[28]

Risk and parameter convergence of logistic regression

Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018 b

2018 arXiv
[29]

Directional convergence and alignment in deep learning

Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33: 0 17176--17186, 2020

2020
[30]

Global stability of first-order methods for coercive tame functions

C \'e dric Josz and Lexiao Lai. Global stability of first-order methods for coercive tame functions. Mathematical Programming, 207 0 (1): 0 551--576, 2024

2024
[31]

The asymmetric maximum margin bias of quasi-homogeneous neural networks

Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. arXiv preprint arXiv:2210.03820, 2022

2022 arXiv
[32]

An Introduction to Differential Manifolds

Jacques Lafontaine. An Introduction to Differential Manifolds. Springer International Publishing, 2015. doi:10.1007/978-3-319-20735-3. URL https://doi.org/10.1007

2015 doi
[33]

Nonsmooth nonconvex stochastic heavy ball

Tam Le. Nonsmooth nonconvex stochastic heavy ball. Journal of Optimization Theory and Applications, 201 0 (2): 0 699--719, 2024

2024
[34]

Training invariances and the low-rank phenomenon: beyond linear networks

Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. arXiv preprint arXiv:2201.11968, 2022

2022 arXiv
[35]

Gradient descent maximizes the margin of homogeneous neural networks

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=SJeLIgBKPS

2020
[36]

Analysis of nonsmooth stochastic approximation: the differential inclusion approach

Szymon Majewski, B a \.z ej Miasojedow, and Eric Moulines. Analysis of nonsmooth stochastic approximation: the differential inclusion approach. arXiv preprint arXiv:1805.01916, 2018

2018 arXiv
[37]

Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models

Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pages 4683--4692. PMLR, 2019 a

2019
[38]

Convergence of gradient descent on separable data

Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420--3428. PMLR, 2019 b

2019
[39]

Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate

Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3051--3059. PMLR, 2019 c

2019
[40]

In search of the real inductive bias: On the role of implicit regularization in deep learning

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA,...

2015 arXiv
[41]

Automatic differentiation in pytorch

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

2017
[42]

A generalization of the borkar-meyn theorem for stochastic recursive inclusions

Arunselvan Ramaswamy and Shalabh Bhatnagar. A generalization of the borkar-meyn theorem for stochastic recursive inclusions. Mathematics of Operations Research, 42 0 (3): 0 648--661, 2017

2017
[43]

The measure of the critical values of differentiable maps

Arthur Sard. The measure of the critical values of differentiable maps. Bull. Amer. Math. Soc., 48 0 (12): 0 883--890, 1942. URL http://dml.mathdoc.fr/item/1183504867

1942
[44]

The implicit bias of gradient descent on separable data

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19 0 (70): 0 1--57, 2018

2018
[45]

A decision method for elementary algebra and geometry

Alfred Tarski. A decision method for elementary algebra and geometry. In Quantifier elimination and cylindrical algebraic decomposition, pages 24--84. Springer, 1951

1951
[46]

Tame topology and o-minimal structures, volume 248

Lou Van den Dries. Tame topology and o-minimal structures, volume 248. Cambridge university press, 1998

1998
[47]

Geometric categories and o-minimal structures

Lou van den Dries and Chris Miller. Geometric categories and o-minimal structures. Duke Math. J., 85 0 (1): 0 497--540, 1996

1996
[48]

The elementary theory of restricted analytic fields with exponentiation

Lou van den Dries, Angus Macintyre, and David Marker. The elementary theory of restricted analytic fields with exponentiation. Annals of Mathematics, 140 0 (1): 0 183--205, 1994

1994
[49]

Statistical learning theory

Vladimir Vapnik. Statistical learning theory. Wiley, 1998. ISBN 978-0-471-03003-4

1998
[50]

On the implicit bias in deep-learning algorithms

Gal Vardi. On the implicit bias in deep-learning algorithms. Communications of the ACM, 66 0 (6): 0 86--93, 2023

2023
[51]

On margin maximization in linear and relu networks

Gal Vardi, Ohad Shamir, and Nati Srebro. On margin maximization in linear and relu networks. Advances in Neural Information Processing Systems, 35: 0 37024--37036, 2022

2022
[52]

The implicit bias for adaptive optimization algorithms on homogeneous neural networks

Bohan Wang, Qi Meng, Wei Chen, and Tie-Yan Liu. The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In International Conference on Machine Learning, pages 10849--10858. PMLR, 2021

2021
[53]

Model completeness results for expansions of the ordered field of real numbers by restricted pfaffian functions and the exponential function

Alex Wilkie. Model completeness results for expansions of the ordered field of real numbers by restricted pfaffian functions and the exponential function. Journal of the American Mathematical Society, 9 0 (4): 0 1051--1094, 1996

1996
[54]

A unifying view on implicit bias in training linear neural networks

Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. arXiv preprint arXiv:2010.02501, 2020

2010 arXiv
[55]

Understanding deep learning (still) requires rethinking generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64 0 (3): 0 107--115, 2021

2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Mart\' i n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Man\' e , Rajat Monga, Sherry Moore, Der...

2015

[2] [2]

Reconciling modern machine-learning practice and the classical bias--variance trade-off

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias--variance trade-off. Proceedings of the National Academy of Sciences, 116 0 (32): 0 15849--15854, 2019

2019

[3] [3]

Dynamics of stochastic approximation algorithms

Michel Bena \" m. Dynamics of stochastic approximation algorithms. In Seminaire de probabilites XXXIII, pages 1--68. Springer, 2006

work page 2006

[4] [4]

Stochastic approximations and differential inclusions

Michel Bena \" m, Josef Hofbauer, and Sylvain Sorin. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization, 44 0 (1): 0 328--348, 2005

work page 2005

[5] [5]

Semianalytic and subanalytic sets

Edward Bierstone and Pierre D Milman. Semianalytic and subanalytic sets. Publications Math \'e matiques de l'IH \'E S , 67: 0 5--42, 1988

work page 1988

[6] [6]

Bolte, A

J. Bolte, A. Daniilidis, A. Lewis, and M. Shiota. Clarke subgradients of stratifiable functions. SIAM Journal on Optimization, 18 0 (2): 0 556--572, 2007

work page 2007

[7] [7]

Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

J \'e r \^o me Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188: 0 19--51, 2021

work page 2021

[8] [8]

Subgradient sampling for nonsmooth nonconvex minimization

J \'e r \^o me Bolte, Tam Le, and Edouard Pauwels. Subgradient sampling for nonsmooth nonconvex minimization. SIAM Journal on Optimization, 33 0 (4): 0 2542--2569, 2023

work page 2023