Pith. sign in

REVIEW 5 minor 194 references

On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations

T0 review · 0 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For a simple quadratic problem with asymmetric data, the MUON optimizer provably fails to converge to the minimizer for almost every finite mini-batch size, leaving an error floor that only larger batches can lower.

desk verdict A genuinely new non-convergence result for a widely used optimizer, with a careful error analysis; the load-bearing asymptotic lemma is delicate but coherent. read the letter →

arxiv 2608.04607 v1 pith:KUMWQOJI submitted 2026-08-05 math.OC cs.LG

Thang Do , Steffen Dereich , Arnulf Jentzen This is my paper
classification math.OCcs.LG MSC 90C1560F0560E1062L2068T07
keywords MUONoptimizerNewton-Schulziterationstochasticoptimizationnon-convergencePolarExpressmini-batchbiasthirdcentralmomentmomentummethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MUON, an optimizer that applies approximate matrix orthogonalization to momentum gradients, is analyzed here in a one-dimensional stochastic quadratic problem. The paper first proves an error bound: after n steps with mini-batch size M and learning rate sequence γ_n, the mean square distance to the minimizer is at most C($M^{{-1}}$+γ_n). Its second and sharper result is that for almost every finite M this process does not converge to the minimizer when the data are asymmetric: the expected distance stays bounded away from zero as n→∞ even though γ_n→0. The cause is a systematic bias created by the Newton-Schulz step that persists exactly at the optimum, which standard SGD and momentum SGD do not possess in this setting.

What carries the argument

The argument passes through a MUON vector field $f(\theta)=E[\Phi((1-\alpha)\sum_{k=0}^\infty \alpha^k (1/M)\sum_{m=1}^M \nabla L(\theta,X_{k+1,m}))]$, where $\Phi$ is the Newton-Schulz polynomial map. A limit theorem (Proposition 6.2) shows that any point at which a MUON-type process converges must be a zero of $f$. The decisive computation (Proposition 6.11) shows that for bounded centered data with nonzero third moment, the stationary momentum variable $Z_M$ obeys $M^{3/2}E[Z_M|Z_M|]\to C\neq 0$ as $M\to\infty$, a sharp rate obtained from characteristic-function expansions. Substituted into a polynomial expansion of the Newton-Schulz map (Lemma 6.12), this makes the vector field nonzero at the optimum (Corollary 6.14), which rules out convergence to it.

What would settle it

Run the one-dimensional quadratic example with $X$ taking values 0 and 1 with unequal probabilities, the original five-step Newton-Schulz polynomial, constant momentum $\alpha=0.95$, and learning rates $\gamma_n=c\,n^{-r}$ with $r\in(0,1)$, and estimate $E[\min\{1,|\Theta^M_n-E[X]|\}]$ for large $n$ at several fixed mini-batch sizes; vanishing for any single $M$ would contradict Theorem 1.3. A cheaper direct check is to Monte-Carlo evaluate the MUON vector field $f(E[X])$ and test whether it is zero, since the paper predicts a nonzero value of order $M^{-3/2}$.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 1.3 (with its general form in Theorem 6.16): for the loss $L(\theta,x)=\lambda|\theta-x|^2$ with bounded i.i.d. data satisfying $E[(X_{1,1}-E[X_{1,1}])^3]\neq 0$, for almost every mini-batch size $M$ the MUON recursion with any admissible collection of generalized Newton-Schulz steps, including the original MUON polynomial and Polar Express, satisfies $\limsup_{n\to\infty} E[\min\{1, |\Theta^M_n-E[X_{1,1}]|\}]>0$. In words, the minimizer is not an accumulation point of the optimization process. The companion error analysis, Theorem 1.2, shows $E|\Theta^M_n-\vartheta|^2\le C(M^{-1}+\gamma_n)$, so the non-convergence is quantitative: with $M$ fixed, the process cannot get closer than roughly $M^{-1/2}$ in mean, no matter how many gradient steps are taken.

Load-bearing premise

The non-convergence conclusion rests on the sharp asymptotic $M^{3/2}E[Z_M|Z_M|]\to C\neq 0$ for bounded centered data with nonzero third moment; if the third moment vanished, the bias would disappear and the conclusion would no longer hold.

Editorial extensions

If this is right

  • For a fixed mini-batch size, taking more gradient steps cannot make MUON reach the exact minimizer on asymmetric problems; an error floor of order $M^{-1/2}$ persists as $n\to\infty$.
  • Because the original MUON polynomial and Polar Express are special cases of the generalized Newton-Schulz family, the non-convergence conclusion applies to the step sequences used in deployed implementations.
  • For symmetric data, where the third central moment vanishes, the bias mechanism disappears and the deterministic-setting convergence result (Corollary 4.7) holds instead.
  • The error bound $C(M^{-1}+\gamma_n)$ indicates that the practical remedy for the bias is to increase the mini-batch size, not to extend training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same third-moment expansion should predict a bias floor in any optimizer whose normalization is a polynomial approximation of the sign function; the mechanism is carried by the approximate orthogonalization map, not by the quadratic loss itself.
  • In large-scale training this suggests the skewness of mini-batch gradients, not just their variance, governs the steady-state bias of MUON-style optimizers; measuring gradient skewness would give a testable predictor of when the bias is visible.
  • An exact polar-factor step is an odd function of the momentum and would not generate the signed-square term, so the paper's mechanism implies that idealized MUON with infinitely many Newton-Schulz iterations should not exhibit this particular bias floor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper proposes a generalized MUON optimizer with an arbitrary finite number of Newton–Schulz steps and arbitrary polynomial coefficients, covering the original MUON implementation and the Polar Express variant as special cases. The first main result, Theorem 1.2, gives a mean-square error bound of order C(M^{-1} + γ_n) for a class of strongly convex stochastic optimization problems with bounded i.i.d. data. The second main result, Theorem 1.3, and its general form Theorem 6.16, show for a one-dimensional quadratic stochastic problem that if the data distribution has a nonzero third central moment, then for every sufficiently large mini-batch size M the MUON process fails to converge to the unique minimizer as the number of gradient steps tends to infinity. The proof combines pathwise a priori bounds, recursive error inequalities, a MUON-type limit theorem (Proposition 6.2), a sharp asymptotic for the signed-square expectation of a stationary momentum random variable (Proposition 6.11), and a non-vanishing evaluation of the MUON vector field (Corollary 6.14).

Significance. If correct, the non-convergence result is significant: it identifies a systematic failure mode of finite-batch MUON that standard SGD and momentum SGD do not exhibit in the same simple quadratic setting, and it shows that the M^{-1} term in the error analysis cannot be removed in general. The result also covers the Newton–Schulz polynomial used in the PyTorch MUON implementation and the Polar Express coefficients, via Lemmas 5.1 and 5.2, which makes the statement relevant beyond an idealized method. The paper is unusually thorough: the main theorems are supported by complete proofs, the delicate asymptotic Proposition 6.11 is argued through characteristic-function expansions and dominated convergence, and the discrepancy between the general upper bound and the non-convergence theorem is clearly exposed. I did not identify a gap in the load-bearing argument, although I did not machine-check the longer asymptotic expansions.

minor comments (5)
  1. [Theorems 1.3 and 6.16] The displayed assumption "lim sup_{n->∞}((γ_{n+1})^{-1}γ_n) < α^{-1} < Σ_{n=1}^∞ γ_n = ∞" is not well-formed as printed; the intended hypothesis appears to be the conjunction of lim sup_{n->∞} γ_n/γ_{n+1} < α^{-1} and Σ_{n=1}^∞ γ_n = ∞, and the proofs use exactly these two conditions. Please rewrite the assumption as two separate statements.
  2. [Section 1.3] The text says that the additional error is "beside the standard error term C√γ_n," whereas Theorem 1.2 is stated for the squared error with the rate C(M^{-1} + γ_n); the notation should be aligned to avoid an apparent inconsistency.
  3. [Section 6, Lemma 6.5 and Lemma 6.6] The global Lipschitz hypothesis on ψ in Lemma 6.5 is stronger than what the subsequent moment estimates require; this is not an error but could be relaxed to a local Lipschitz condition together with the polynomial growth bound.
  4. [Section 1.7] The disclosure that some lemmas and their proofs were developed through discussions with GPT models is transparent, but the statement that the core ideas for statements and proofs are "due to" the models is unusual for a mathematics paper; please clarify the authors' verification role or move this material to the acknowledgements.
  5. [Global] There are several typographical errors that should be corrected in a revision: "Newton-Schultz" in the abstract, "opimization" in Section 1.3, "mearsurable" in the proof of Lemma 3.11, "thar" in the proof of Lemma 5.3, "satsfiy" in Definition 3.14, and "covex" in the proof of Corollary 4.6.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the non-convergence mechanism is derived from the dynamics and characteristic-function asymptotics, not assumed.

full rationale

The load-bearing non-convergence claim, Theorem 1.3, is an application of Theorem 6.16, whose proof does not reduce to a fitted input or to a self-citation. Proposition 6.11 proves, under the explicit domain condition E[X^3] != 0, the sharp asymptotic M^{3/2} E[Z_M |Z_M|] -> C != 0 via characteristic-function expansions (Lemma 6.8), an integral representation of the signed square (Lemma 6.9), and a variance non-degeneracy lemma (Lemma 6.10). Corollary 6.14 then combines Proposition 6.11 with the polynomial representation of the NS map (Lemma 6.12) to show that the MUON vector field is nonzero at the optimum for all sufficiently large mini-batch sizes. Theorem 6.16 finishes by contradiction: convergence to the optimum would, by the proven limit theorem Proposition 6.2, force the vector field to vanish at the optimum, contradicting Corollary 6.14. Every load-bearing step is proved in the paper from the stated model equations. The self-citations to the authors' Adam papers, such as [14, Theorem 1.1], are used only to explain the overall proof strategy and to import elementary background lemmas (e.g., Lemma 2.1 and Lemma 2.2), not to assume the MUON conclusion; Proposition 6.2 and the auxiliary lemmas are self-contained. No fitted parameter is renamed as a prediction, and no known result is merely relabeled. Minor exposition issues, such as the unnecessarily strong global Lipschitz hypothesis in Lemma 6.5 and the delicate M_n versus M^n notation in Proposition 6.2, do not affect the derivation's independence.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted; the NS polynomial coefficients come from the existing optimizers. The MUON vector field is a derived construction, not a new physical entity. The analysis rests on standard mathematics plus explicit smoothness and data assumptions.

assumptions (5)
  • domain assumption Data variables are bounded and i.i.d., with nonzero third central moment in the non-convergence theorem (Theorem 1.3).
    This assumption enters the characteristic-function expansion in Proposition 6.11; without it, the leading M^{-3/2} term in E[Z_M |Z_M|] vanishes and the vector field at the optimum may be zero.
  • domain assumption Strong convexity and Lipschitz continuity of the gradient of the loss in the error analysis (Theorems 1.2 and 3.17).
    These conditions provide the recursive inequality and the quadratic lower bound used in the error analysis in Sections 3 and 4.
  • domain assumption Positivity of the Newton-Schulz polynomials: for all i in {0,...,K} and all x in R, sum_j b_{i,j} x^{2j} > 0.
    This ensures the NS map has the lower bound in Lemma 3.13; the authors verify it for the original MUON and Polar Express polynomials in Lemmas 5.1 and 5.2.
  • standard math Standard probability and analysis results: central limit theorem, Marcinkiewicz-Zygmund inequality, SVD, and measurable selection (Arsenin-Kunugui).
    These are used throughout, for example in Lemma 3.4, Lemma 3.11, and Proposition 6.11.
  • domain assumption Learning-rate regularity conditions: gamma_n non-increasing with limsup(gamma_n + (gamma_n)^{-2}(gamma_n - gamma_{n+1})) = 0 for the upper bound, and limsup((gamma_{n+1})^{-1} gamma_n) < alpha^{-1} with sum gamma_n = infinity for the non-convergence result.
    These are standard vanishing-but-not-too-fast step-size conditions, stated explicitly in Theorems 1.2 and 1.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations." pith.science (2026). https://pith.science/paper/KUMWQOJI

@misc{pith2026260804607,
  author       = {Pith},
  title        = {Pith review of: On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KUMWQOJI}},
  note         = {Machine review of arXiv:2608.04607}
}
abstract

Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs). In many relevant artificial intelligence (AI) systems - such as popular large language models (LLMs)-not the standard SGD scheme is used as the optimization method but instead suitable accelerated variants of SGD are employed. One of the most popular methods of such accelerated SGD variants is the momentum orthogonalized by Newton-Schulz (MUON) optimizer proposed by Jordan et al. in 2024. The MUON optimizer exploits the special matrix structure of the weight parameters in the training of the DNNs and, in its original form, employs five Newton-Schultz (NS) matrix steps in each MUON iteration. In this work we propose and study a generalized variant of the MUON optimizer involving an arbitrary number of generalized NS steps with polynomials of possibly arbitrary high degree. The considered optimizer covers MUON with the original NS polynomial as well as MUON combined with the recently proposed Polar Express method as special cases. For a simple class of stochastic optimization problems (SOPs) we show for almost every mini-batch size that MUON fails to converge to the solution of the SOP as the number of gradient steps converges to infinity. We also establish an error analysis for MUON with the generalized NS steps that provides convergence rates in terms of the number of gradient steps and in terms of the size of the mini-batch. We illustrate our general error analysis for MUON in the case of several concrete examples including quadratic stochastic optimization problems (SOPs) as well as $\ell_2$ regularized logistic regression for binary classification.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

194 extracted references · 32 canonical work pages

  1. [66]

    Poljak, B. T. , TITLE =. Z. Vy cisl. Mat i Mat. Fiz. , FJOURNAL =. 1964 , PAGES =

    1964

  2. [1]

    Cheridito, Patrick and Jentzen, Arnulf and Riekert, Adrian and Rossmannek, Florian , TITLE =. J. Complexity , FJOURNAL =. 2022 , PAGES =. doi:10.1016/j.jco.2022.101646 , URL =

    work page arXiv 2022

  3. [2]

    2004 , PAGES =

    Nesterov, Yurii , TITLE =. 2004 , PAGES =. doi:10.1007/978-1-4419-8853-9 , URL =

    doi:10.1007/978-1-4419-8853-9 2004

  4. [3]

    Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning , url =

    Moulines, Eric and Bach, Francis , booktitle =. Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning , url =

  5. [4]

    Proceedings of The 33rd International Conference on Machine Learning , pages =

    Stochastic Variance Reduction for Nonconvex Optimization , author =. Proceedings of The 33rd International Conference on Machine Learning , pages =. 2016 , editor =

    2016

  6. [5]

    Yeonjong Shin and George Em Karniadakis , title =. J. Mach. Learn. Model. Comput , fjournal=. 2020 , volume =

    2020

  7. [7]

    , TITLE =

    Davis, Damek and Drusvyatskiy, Dmitriy and Kakade, Sham and Lee, Jason D. , TITLE =. Found. Comput. Math. , FJOURNAL =. 2020 , NUMBER =. doi:10.1007/s10208-018-09409-5 , URL =

    doi:10.1007/s10208-018-09409-5 2020

  8. [8]

    Steffen Dereich and Sebastian Kassing , title=

Show all 194 references
  1. [9]

    Shokhrukh Ibragimov and Arnulf Jentzen and Adrian Riekert , title=

  2. [10]

    Gentile and G

    R. Gentile and G. Welper , title=

  3. [11]

    Welper, Gerrit , TITLE =. J. Mach. Learn. , FJOURNAL =. 2024 , NUMBER =

    2024

  4. [12]

    Jentzen, Arnulf and Riekert, Adrian , TITLE =. J. Math. Anal. Appl. , FJOURNAL =. 2023 , NUMBER =. doi:10.1016/j.jmaa.2022.126601 , URL =

    2023

  5. [13]

    Itay Safran and Ohad Shamir , journal=

  6. [14]

    2017 , journal=

    Local minima in training of neural networks , author=. 2017 , journal=

    2017

  7. [15]

    Zhang, Yaoyu and Li, Yuqing and Zhang, Zhongwang and Luo, Tao and Xu, Zhi-Qin John , TITLE =. J. Mach. Learn. , FJOURNAL =. 2022 , NUMBER =

    2022

  8. [16]

    CSIAM Trans

    Bai, Zhiwei and Luo, Tao and Xu, Zhi-Qin John and Zhang, Yaoyu , TITLE =. CSIAM Trans. Appl. Math. , FJOURNAL =. 2024 , NUMBER =

    2024

  9. [17]

    Cheridito, Patrick and Jentzen, Arnulf and Rossmannek, Florian , TITLE =. J. Nonlinear Sci. , FJOURNAL =. 2022 , NUMBER =. doi:10.1007/s00332-022-09823-8 , URL =

    2022 doi

  10. [18]

    Exponential moments for numerical approximations of stochastic partial differential equations , JOURNAL =

    Jentzen, Arnulf and Pu. Exponential moments for numerical approximations of stochastic partial differential equations , JOURNAL =. 2018 , NUMBER =. doi:10.1007/s40072-018-0116-y , URL =

    2018 doi

  11. [19]

    2018 , eprint =

    Jentzen, Arnulf and Pu. 2018 , eprint =

    2018

  12. [20]

    arXiv:1911.05620 , archivePrefix =

    Johannes Ruf and Weiguan Wang , year =. arXiv:1911.05620 , archivePrefix =

    1911 arXiv

  13. [21]

    Kingma and Jimmy Ba , year =

    Diederik P. Kingma and Jimmy Ba , year =. arXiv:1412.6980 , archivePrefix =

    arXiv

  14. [22]

    Cheridito, Patrick and Jentzen, Arnulf and Rossmannek, Florian , TITLE =. J. Complexity , FJOURNAL =. 2021 , PAGES =. doi:10.1016/j.jco.2020.101540 , URL =

    2021

  15. [23]

    2402.05155 , archivePrefix=

    Arnulf Jentzen and Adrian Riekert , year=. 2402.05155 , archivePrefix=

    arXiv

  16. [24]

    Lu, Lu and Shin, Yeonjong and Su, Yanhui and Karniadakis, George Em , TITLE =. Commun. Comput. Phys. , FJOURNAL =. 2020 , NUMBER =. doi:10.4208/cicp.oa-2020-0165 , URL =

    2020 doi

  17. [25]

    D. Trans. on Mach. Learn. Res. , issn =. 2022 , url =

    2022

  18. [26]

    Reddi, Sashank J and Kale, Satyen and Kumar, Sanjiv , eprint =

  19. [27]

    arXiv:1811.09358 , archivePrefix =

    Zou, Fangyu and Shen, Li and Jie, Zequn and Zhang, Weizhong and Liu, Wei , year =. arXiv:1811.09358 , archivePrefix =

    arXiv

  20. [28]

    arXiv:2208.09632 , archivePrefix =

    Zhang, Yushun and Chen, Congliang and Shi, Naichen and Sun, Ruoyu and Luo, Zhi-Quan , year =. arXiv:2208.09632 , archivePrefix =

    arXiv

  21. [29]

    Duncan , year =

    Juntang Zhuang and Tommy Tang and Yifan Ding and Sekhar Tatikonda and Nicha Dvornek and Xenophon Papademetris and James S. Duncan , year =. arXiv:2010.07468 , archivePrefix =

    2010 arXiv

  22. [30]

    Learning the random variables in

    Becker, Sebastian and Jentzen, Arnulf and M\". Learning the random variables in. Math. Finance , FJOURNAL =. 2024 , NUMBER =. doi:10.1111/mafi.12405 , URL =

    2024 doi

  23. [31]

    IEEE Trans

    Zhou, Yangfan and Huang, Kaizhu and Cheng, Cheng and Wang, Xuguang and Hussain, Amir and Liu, Xin , TITLE =. IEEE Trans. Neural Netw. Learn. Syst. , FJOURNAL =. 2023 , NUMBER =. doi:10.1109/tnnls.2022.3143554 , URL =

    2023

  24. [32]

    arXiv:1905.02957 , archivePrefix =

    Guanghui Wang and Shiyin Lu and Weiwei Tu and Lijun Zhang , year =. arXiv:1905.02957 , archivePrefix =

    1905 arXiv

  25. [33]

    arXiv:1902.09843 , archivePrefix =

    Liangchen Luo and Yuanhao Xiong and Yan Liu and Xu Sun , year =. arXiv:1902.09843 , archivePrefix =

    1902 arXiv

  26. [34]

    arXiv:2109.08282 , archivePrefix =

    Xiaoxia Wu and Yuege Xie and Simon Du and Rachel Ward , year =. arXiv:2109.08282 , archivePrefix =

    arXiv

  27. [35]

    arXiv:2312.15295 , archivePrefix =

    Lu Xia and Stefano Massei , year =. arXiv:2312.15295 , archivePrefix =

    arXiv

  28. [36]

    arXiv:2011.02150 , archivePrefix =

    Wei Yuan and Kai-Xin Gao , year =. arXiv:2011.02150 , archivePrefix =

    2011 arXiv

  29. [37]

    2017 , eprint=

    An overview of gradient descent optimization algorithms , author=. 2017 , eprint=

    2017

  30. [38]

    IMA J. Numer. Anal. , fjournal =. 2020 , abstract =

    Jentzen, Arnulf and Kuckuck, Benno and Neufeld, Ariel and von Wurstemberger, Philippe , title = ". IMA J. Numer. Anal. , fjournal =. 2020 , abstract = ". doi:10.1093/imanum/drz055 , url =

    2020 doi

  31. [39]

    OpenAI and Josh Achiam and Steven Adler and Sandhini Agarwal and Lama Ahmad and Ilge Akkaya and Florencia Leoni Aleman and Diogo Almeida and Janko Altenschmidt and Sam Altman and Shyamal Anadkat and Red Avila and Igor Babuschkin and Suchir Balaji and Valerie Balcom and Paul Ba...

    arXiv

  32. [40]

    Petersen, Philipp and Raslan, Mones and Voigtlaender, Felix , TITLE =. Found. Comput. Math. , FJOURNAL =. 2021 , NUMBER =

    2021

  33. [41]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Chil...

    2005 arXiv

  34. [42]

    2024 , eprint=

    Gemini: A Family of Highly Capable Multimodal Models , author=. 2024 , eprint=

    2024

  35. [43]

    Sirignano, Justin and Spiliopoulos, Konstantinos , TITLE =. J. Comput. Phys. , FJOURNAL =. 2018 , PAGES =. doi:10.1016/j.jcp.2018.08.029 , URL =

    2018 doi

  36. [44]

    2105.09506 , archivePrefix=

    Shengze Cai and Zhiping Mao and Zhicheng Wang and Minglang Yin and George Em Karniadakis , year=. 2105.09506 , archivePrefix=

    arXiv

  37. [45]

    2021 , eprint=

    Fourier Neural Operator for Parametric Partial Differential Equations , author=. 2021 , eprint=

    2021

  38. [46]

    Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations , author=. Commun. Math. Stat. , fjournal=. 2017 , publisher=. doi:10.1007/s40304-017-0117-6 , url=

    2017 doi

  39. [47]

    Discrete Contin

    Beck, Christian and Hutzenthaler, Martin and Jentzen, Arnulf and Kuckuck, Benno , title =. Discrete Contin. Dyn. Syst. Ser. B , fjournal =. 2023 , volume =. doi:10.3934/dcdsb.2022238 , url =

    2023 doi

  40. [48]

    Han, Jiequn and Jentzen, Arnulf and E, Weinan , TITLE =. Proc. Natl. Acad. Sci. USA , FJOURNAL =. 2018 , NUMBER =. doi:10.1073/pnas.1718942115 , URL =

    2018 doi

  41. [49]

    2016 , eprint=

    Deep Learning Approximation for Stochastic Control Problems , author=. 2016 , eprint=

    2016

  42. [50]

    Becker, Sebastian and Cheridito, Patrick and Jentzen, Arnulf , TITLE =. J. Mach. Learn. Res. , FJOURNAL =. 2019 , PAGES =

    2019

  43. [51]

    and Perdikaris, P

    Raissi, M. and Perdikaris, P. and Karniadakis, G. E. , TITLE =. J. Comput. Phys. , FJOURNAL =. 2019 , PAGES =. doi:10.1016/j.jcp.2018.10.045 , URL =

    2019 doi

  44. [52]

    2024 , eprint=

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis , author=. 2024 , eprint=

    2024

  45. [53]

    2102.12092 , archivePrefix=

    Aditya Ramesh and Mikhail Pavlov and Gabriel Goh and Scott Gray and Chelsea Voss and Alec Radford and Mark Chen and Ilya Sutskever , year=. 2102.12092 , archivePrefix=

    arXiv

  46. [54]

    Nonlinearity , FJOURNAL =

    E, Weinan and Han, Jiequn and Jentzen, Arnulf , TITLE =. Nonlinearity , FJOURNAL =. 2022 , NUMBER =. doi:10.1088/1361-6544/ac337f , URL =

    2022 doi

  47. [55]

    (2004) , publisher=

    Convex Optimization , author=. (2004) , publisher=. doi:10.1017/CBO9780511804441 , URL =

    2004 doi

  48. [56]

    (2014), xii+638 , publisher=

    Probability Theory , author=. (2014), xii+638 , publisher=

    2014

  49. [57]

    (2021), xii+946 , PAGES =

    Kallenberg, Olav , TITLE =. (2021), xii+946 , PAGES =. doi:10.1007/978-3-030-61871-1 , URL =

    2021 doi

  50. [58]

    2012 , numpages =

    Hinton, Geoffrey and Srivastava, Nitish and Swersky, Kevin , title =. 2012 , numpages =

    2012

  51. [59]

    Zitkovic, Gordan , journal =

  52. [60]

    (2014), xviii+493 , PAGES =

    Da Prato, Giuseppe and Zabczyk, Jerzy , TITLE =. (2014), xviii+493 , PAGES =. doi:10.1017/CBO9781107295513 , URL =

    2014 doi

  53. [61]

    Dudley, R. M. , TITLE =. (2002), x+555 , PAGES =. doi:10.1017/CBO9780511755347 , URL =

    2002 doi

  54. [62]

    and Lahiri, Soumendra N

    Athreya, Krishna B. and Lahiri, Soumendra N. , TITLE =. (2006), xviii+618 , PAGES =

    2006

  55. [63]

    Nature Reviews Physics , volume=

    Physics-informed machine learning , author=. Nature Reviews Physics , volume=. 2021 , publisher=

    2021

  56. [64]

    European J

    Becker, Sebastian and Cheridito, Patrick and Jentzen, Arnulf and Welti, Timo , TITLE =. European J. Appl. Math. , FJOURNAL =. 2021 , NUMBER =. doi:10.1017/S0956792521000073 , URL =

    2021 doi

  57. [65]

    Duchi, John and Hazan, Elad and Singer, Yoram , title =. J. Mach. Learn. Res. , issue_date =. 2011 , issn =

    2011

  58. [67]

    , TITLE =

    Blechschmidt, Jan and Ernst, Oliver G. , TITLE =. GAMM-Mitt. , FJOURNAL =. 2021 , NUMBER =. doi:10.1002/gamm.202100006 , URL =

    2021 doi

  59. [68]

    Cuomo, Salvatore and Schiano Di Cola, Vincenzo and Giampaolo, Fabio and Rozza, Gianluigi and Raissi, Maziar and Piccialli, Francesco , TITLE =. J. Sci. Comput. , FJOURNAL =. 2022 , NUMBER =. doi:10.1007/s10915-022-01939-z , URL =

    2022 doi

  60. [69]

    (2023), 426–452 , X=

    Germain, Maximilien and Pham, Huy. (2023), 426–452 , X=. doi:10.1017/9781009028943.023 , booktitle=

    2023 doi

  61. [70]

    Optimization for deep learning: theory and algorithms , year =

    Ruoyu Sun , journal =. Optimization for deep learning: theory and algorithms , year =

  62. [71]

    2023 , archiveprefix =

    Antoine Godichon-Baggioni and Pierre Tarrago , title =. 2023 , archiveprefix =. 2303.01370 , journal =

    2023 arXiv

  63. [72]

    SIAM Journal on Mathematics of Data Science , volume =

    Berner, Julius and Grohs, Philipp and Jentzen, Arnulf , title =. SIAM Journal on Mathematics of Data Science , volume =. 2020 , page=. doi:10.1137/19M125649X , URL =

    2020 doi

  64. [73]

    2021 , number =

    Lu, Lu and Jin, Pengzhan and Pang, Guofei and Zhang, Zhongqiang and Karniadakis, George Em , journal =. 2021 , number =. doi:10.1038/s42256-021-00302-5 , groups =

    2021 doi

  65. [74]

    Barakat, Anas and Bianchi, Pascal , TITLE =. SIAM J. Optim. , FJOURNAL =. 2021 , NUMBER =. doi:10.1137/19M1263443 , URL =

    2021 doi

  66. [75]

    2402.13794 , archivePrefix=

    Yusu Hong and Junhong Lin , year=. 2402.13794 , archivePrefix=

    arXiv

  67. [76]

    2024 , eprint=

    Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses , author=. 2024 , eprint=

    2024

  68. [77]

    Cucker, Felipe and Smale, Steve , TITLE =. Bull. Amer. Math. Soc. (N.S.) , FJOURNAL =. 2002 , NUMBER =. doi:10.1090/S0273-0979-01-00923-5 , URL =

    2002 doi

  69. [78]

    2016 , journal=

    An overview of gradient descent optimization algorithms , author=. 2016 , journal=

    2016

  70. [79]

    2310.20360 , archivePrefix=

    Arnulf Jentzen and Benno Kuckuck and Philippe von Wurstemberger , year=. 2310.20360 , archivePrefix=

    arXiv

  71. [80]

    2024 , ISBN =

    Francis Bach , TITLE =. 2024 , ISBN =

    2024

  72. [81]

    , TITLE =

    Evans, Lawrence C. , TITLE =. 1998 , PAGES =. doi:10.1090/gsm/019 , URL =

    1998 doi

  73. [82]

    2019 , eprint=

    Optimization for deep learning: theory and algorithms , author=. 2019 , eprint=

    2019

  74. [83]

    Kingma and Jimmy Ba , title =

    Diederik P. Kingma and Jimmy Ba , title =

  75. [84]

    Incorporating

    Dozat, Timothy , booktitle =. Incorporating

  76. [85]

    Zeiler , title=

    Matthew D. Zeiler , title=

  77. [86]

    Journal of Machine Learning Research , year =

    John Duchi and Elad Hazan and Yoram Singer , title =. Journal of Machine Learning Research , year =

  78. [87]

    Proceedings of the USSR Academy of Sciences , year=

    A method for solving the convex programming problem with convergence rate O(1/k^2) , author=. Proceedings of the USSR Academy of Sciences , year=

  79. [88]

    SIAM/ASA J

    Jentzen, Arnulf and Riekert, Adrian , TITLE =. SIAM/ASA J. Uncertain. Quantif. , FJOURNAL =. 2025 , NUMBER =. doi:10.1137/24M1639464 , URL =

    2025 doi

  80. [89]

    Muller and Philippe von Wurstemberger , title=

    Sebastian Becker and Arnulf Jentzen and Marvin S. Muller and Philippe von Wurstemberger , title=

  81. [90]

    Jentzen, Arnulf and Riekert, Adrian , TITLE =. J. Mach. Learn. , FJOURNAL =. 2022 , NUMBER =

    2022

  82. [91]

    Arnulf Jentzen and Adrian Riekert , title=

  83. [92]

    The analysis of linear partial differential operators

    Lars H\". The analysis of linear partial differential operators. 2003 , doi=

    2003

  84. [93]

    Martin Hutzenthaler and Arnulf Jentzen, Katharina Pohl and Adrian Riekert and Luca Scarpa , title=

  85. [94]

    A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with

    Arnulf Jentzen and Adrian Riekert , year=. A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with. 2108.04620 , archivePrefix=

    arXiv

  86. [95]

    A mathematical model for automatic differentiation in machine learning , url =

    Bolte, J\'. A mathematical model for automatic differentiation in machine learning , url =. Advances in Neural Information Processing Systems , editor =

  87. [96]

    On Correctness of Automatic Differentiation for Non-Differentiable Functions , url =

    Lee, Wonyeol and Yu, Hangyeol and Rival, Xavier and Yang, Hongseok , booktitle =. On Correctness of Automatic Differentiation for Non-Differentiable Functions , url =

  88. [97]

    NeurIPS , year=

    Provably Correct Automatic Subdifferentiation for Qualified Programs , author=. NeurIPS , year=

  89. [98]

    Bolte, Jérôme and Boustany, Ryan and Pauwels, Edouard and Pesquet-Popescu, Béatrice , title=

  90. [99]

    Jérôme Bolte and Edouard Pauwels and Samuel Vaiter , title=

  91. [100]

    Bolte, J\'er\^ome and Pauwels, Edouard , TITLE =. Math. Program. , FJOURNAL =. 2021 , NUMBER =. doi:10.1007/s10107-020-01501-5 , URL =

    2021 doi

  92. [101]

    Adam Paszke and Sam Gross and Francisco Massa and Adam Lerer and James Bradbury and Gregory Chanan and Trevor Killeen and Zeming Lin and Natalia Gimelshein and Luca Antiga and Alban Desmaison and Andreas Köpf and Edward Yang and Zach DeVito and Martin Raison and Alykhan Tejani...

  93. [102]

    Jerome Bolte and Edouard Pauwels , title=

  94. [103]

    and Drusvyatskiy, D

    Davis, D. and Drusvyatskiy, D. and Kakade, S. and Lee, J.D. , year =. Stochastic subgradient method converges on tame functions , journal =

  95. [104]

    Jentzen, Arnulf and Riekert, Adrian , TITLE =. Z. Angew. Math. Phys. , FJOURNAL =. 2022 , NUMBER =. doi:10.1007/s00033-022-01716-w , URL =

    2022 doi

  96. [105]

    Gower , title=

    Guillaume Garrigos and Robert M. Gower , title=

  97. [106]

    Steffen Dereich and Robin Graeber and Arnulf Jentzen , title=

  98. [107]

    Philipp Petersen and Jakob Zech , title=

  99. [108]

    Quoc Tung Le and Elisa Riccietti and Rémi Gribonval , title=

  100. [109]

    Tian Ding and Dawei Li and Ruoyu Sun , title=

  101. [110]

    and Bruna, Joan , TITLE =

    Venturi, Luca and Bandeira, Afonso S. and Bruna, Joan , TITLE =. J. Mach. Learn. Res. , FJOURNAL =. 2019 , PAGES =

    2019

  102. [111]

    Set-Valued Var

    Davis, Damek and Drusvyatskiy, Dmitriy , TITLE =. Set-Valued Var. Anal. , FJOURNAL =. 2022 , NUMBER =. doi:10.1007/s11228-021-00594-0 , URL =

    2022 doi

  103. [112]

    and Tian, Tonghua , TITLE =

    Lewis, Adrian S. and Tian, Tonghua , TITLE =. SIAM J. Optim. , FJOURNAL =. 2021 , NUMBER =. doi:10.1137/21M1393637 , URL =

    2021 doi

  104. [113]

    Set-Valued Var

    Pauwels, Edouard , TITLE =. Set-Valued Var. Anal. , FJOURNAL =. 2023 , NUMBER =. doi:10.1007/s11228-023-00682-3 , URL =

    2023 doi

  105. [114]

    Bolte, J\'er\^ome and Pauwels, Edouard and Silveti-Falls, Antonio , TITLE =. SIAM J. Optim. , FJOURNAL =. 2024 , NUMBER =. doi:10.1137/22M1541630 , URL =

    2024 doi

  106. [115]

    Nachuan Xiao and Xiaoyin Hu and Xin Liu and Kim-Chuan Toh , title=

  107. [116]

    Nachuan Xiao and Xiaoyin Hu and Kim-Chuan Toh , title=

  108. [117]

    Kuangyu Ding and Nachuan Xiao and Kim-Chuan Toh , title=

  109. [118]

    Sholom Schechtman , title=

  110. [119]

    Weinan E and Chao Ma and Stephan Wojtowytsch and Lei Wu , title=

  111. [120]

    2017 , url =

    Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory and Yang, Edward and DeVito, Zachary and Lin, Zeming and Desmaison, Alban and Antiga, Luca and Lerer, Adam , title =. 2017 , url =

    2017

  112. [121]

    Steffen Dereich and Arnulf Jentzen , title=

  113. [122]

    Steffen Dereich and Thang Do and Arnulf Jentzen and Philippe von Wurstemberger , title=

  114. [123]

    Ussr Computational Mathematics and Mathematical Physics , year=

    Some methods of speeding up the convergence of iteration methods , author=. Ussr Computational Mathematics and Mathematical Physics , year=

  115. [124]

    Steffen Dereich and Robin Graeber and Arnulf Jentzen and Adrian Riekert , title=

  116. [125]

    Robbins, Herbert and Monro, Sutton , TITLE =. Ann. Math. Statistics , FJOURNAL =. 1951 , PAGES =. doi:10.1214/aoms/1177729586 , URL =

    1951

  117. [126]

    Zhishuai Guo and Yi Xu and Wotao Yin and Rong Jin and Tianbao Yang , title=

  118. [127]

    Alexandre Défossez and Léon Bottou and Francis Bach and Nicolas Usunier , title=

  119. [128]

    Haochuan Li and Alexander Rakhlin and Ali Jadbabaie , title=

  120. [129]

    Li Shen and Congliang Chen and Fangyu Zou and Zequn Jie and Ju Sun and Wei Liu , title=

  121. [130]

    Bohan Wang and Huishuai Zhang and Qi Meng and Ruoyu Sun and Zhi-Ming Ma and Wei Chen , title=

  122. [131]

    Bohan Wang and Yushun Zhang and Huishuai Zhang and Qi Meng and Ruoyu Sun and Zhi-Ming Ma and Tie-Yan Liu and Zhi-Quan Luo and Wei Chen , title=

  123. [132]

    Qi Zhang and Yi Zhou and Shaofeng Zou , title=

  124. [133]

    Anas Barakat and Pascal Bianchi , title=

  125. [134]

    Sebastian Bock and Josef Goppold and Martin Weiß , title=

  126. [135]

    2019 International Joint Conference on Neural Networks (IJCNN) , year=

    Sebastian Bock and Martin Georg Wei. 2019 International Joint Conference on Neural Networks (IJCNN) , year=

    2019

  127. [136]

    Soham De and Anirbit Mukherjee and Enayat Ullah , title=

  128. [137]

    Steffen Dereich and Arnulf Jentzen and Adrian Riekert , title=

  129. [138]

    2021 , url=

    Naichen Shi and Dawei Li and Mingyi Hong and Ruoyu Sun , booktitle=. 2021 , url=

    2021

  130. [139]

    Sebastian Bock and Martin Georg Weiß , title=

  131. [140]

    Thang Do and Arnulf Jentzen and Adrian Riekert , title=

  132. [141]

    Steffen Dereich and Arnulf Jentzen and Sebastian Kassing , title=

  133. [142]

    Naum Dimitrieski and Maria Christine Honecker and Carsten Scherer and Christian Ebenbauer , title=

  134. [143]

    Zhiwei Bai and Jiajie Zhao and Zhangchen Zhou and Zhi-Qin John Xu and Yaoyu Zhang , title=

  135. [144]

    Steffen Dereich and Thang Do and Arnulf Jentzen , title=

  136. [145]

    Shokhrukh Ibragimov and Arnulf Jentzen , title=

  137. [146]

    2012 , publisher =

    Matrix Analysis , author =. 2012 , publisher =

    2012

  138. [147]

    Mher Safaryan and Peter Richtárik , title=

  139. [148]

    1997 , PAGES =

    Chow, Yuan Shih and Teicher, Henry , TITLE =. 1997 , PAGES =. doi:10.1007/978-1-4612-1950-7 , URL =

    1997 doi

  140. [149]

    2024 , journal =

    Keller Jordan and Yuchen Jin and Vlado Boza and Jiacheng You and Franz Cesista and Laker Newhouse and Jeremy Bernstein , title =. 2024 , journal =

    2024

  141. [150]

    Wei Shen and Ruichuan Huang and Minhui Huang and Cong Shen and Jiawei Zhang , title=

  142. [151]

    Jiaxiang Li and Mingyi Hong , title=

  143. [152]

    Lizhang Chen and Jonathan Li and Qiang Liu , title=

  144. [153]

    Maria-Eleni Sfyraki and Jun-Kun Wang , title=

  145. [154]

    Naoki Sato and Hiroki Naganuma and Hideaki Iiduka , title=

  146. [155]

    Da Chang and Yongxiang Liu and Ganzhao Yuan , title=

  147. [156]

    Jianhao Ma and Yu Huang and Yuejie Chi and Yuxin Chen , title=

  148. [157]

    Gyu Yeol Kim, Min-hwan Oh , title=

  149. [158]

    Shuntaro Nagashima and Hideaki Iiduka , title=

  150. [159]

    Tien-Phat Nguyen and Truong Nguyen and Minh-Phuc Truong and Tuc Nguyen and James Bailey and Trung Le , title=

  151. [160]

    Jeremy Bernstein and Yu-Xiang Wang and Kamyar Azizzadenesheli and Anima Anandkumar , title=

  152. [161]

    Jeremy Bernstein and Jiawei Zhao and Kamyar Azizzadenesheli and Anima Anandkumar , title=

  153. [162]

    Stich and Martin Jaggi , title=

    Sai Praneeth Karimireddy and Quentin Rebjock and Sebastian U. Stich and Martin Jaggi , title=

  154. [163]

    Michael Crawshaw and Mingrui Liu and Francesco Orabona and Wei Zhang and Zhenxun Zhuang , title=

  155. [164]

    2023 , volume =

    Sun, Tao and Wang, Qingsong and Li, Dongsheng and Wang, Bao , booktitle =. 2023 , volume =

    2023

  156. [165]

    2026 , howpublished =

    2026

  157. [166]

    , TITLE =

    Åke Björck and Bowie, C. , TITLE =. SIAM J. Numer. Anal. , FJOURNAL =. 1971 , PAGES =. doi:10.1137/0708036 , URL =

    1971 doi

  158. [167]

    Zdislav Kovarik , TITLE =. SIAM J. Numer. Anal. , FJOURNAL =. 1970 , PAGES =. doi:10.1137/0707031 , URL =

    1970 doi

  159. [168]

    , TITLE =

    Higham, Nicholas J. , TITLE =. 2008 , PAGES =. doi:10.1137/1.9780898717778 , URL =

    2008 doi

  160. [169]

    Jeremy Bernstein and Laker Newhouse , title=

  161. [170]

    Polyak, B. T. and Juditsky, A. B. , TITLE =. SIAM J. Control Optim. , FJOURNAL =. 1992 , NUMBER =. doi:10.1137/0330046 , URL =

    1992 doi

  162. [171]

    and Yang, Jichuan , TITLE =

    Kushner, Harold J. and Yang, Jichuan , TITLE =. SIAM J. Control Optim. , FJOURNAL =. 1993 , NUMBER =. doi:10.1137/0331047 , URL =

    1993 doi

  163. [172]

    , TITLE =

    Rosenthal, Haskell P. , TITLE =. Israel J. Math. , FJOURNAL =. 1970 , PAGES =. doi:10.1007/BF02771562 , URL =

    1970 doi

  164. [173]

    Nguyen and Phuong Ha Nguyen and Marten van Dijk and Peter Richtarik and Katya Scheinberg and Martin Takac , title=

    Lam M. Nguyen and Phuong Ha Nguyen and Marten van Dijk and Peter Richtarik and Katya Scheinberg and Martin Takac , title=

  165. [174]

    2025 , numpages =

    Bennno Kuckuck , title =. 2025 , numpages =

    2025

  166. [175]

    Egor Shulgin and Sultan AlRashed and Francesco Orabona and Peter Richtárik , title=

  167. [176]

    Davide Gallon and Arnulf Jentzen and Felix Lindner , title=

  168. [177]

    Steven Heilman and Sampad Mohanty , title=

  169. [178]

    Gower , title=

    Tetiana Parshakova and Ahmed Khaled and Michael Crawshaw and Guillaume Garrigos and Robert M. Gower , title=

  170. [179]

    Antoine Gonon and Andreea-Alexandra Musat and Nicolas Boumal , title=

  171. [180]

    Gower , title=

    Noah Amsel and David Persson and Christopher Musco and Robert M. Gower , title=

  172. [181]

    Julian Kranz and Davide Gallon and Steffen Dereich and Arnulf Jentzen , title=

  173. [182]

    2025 , howpublished =

    2025

  174. [183]

    2024 , howpublished =

    Keller Jordan , title =. 2024 , howpublished =

    2024

  175. [184]

    , TITLE =

    Nesterov, Yu.\ E. , TITLE =. Dokl. Akad. Nauk SSSR , FJOURNAL =. 1983 , NUMBER =

    1983

  176. [185]

    2013 , volume =

    Sutskever, Ilya and Martens, James and Dahl, George and Hinton, Geoffrey , booktitle =. 2013 , volume =

    2013

  177. [186]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  178. [187]

    Euhanna Ghadimi and Hamid Reza Feyzmahdavian and Mikael Johansson , title=

  179. [188]

    Online Learning and Stochastic Approximations , booktitle =

    Bottou, L. Online Learning and Stochastic Approximations , booktitle =. 1999 , doi =

    1999

  180. [189]

    Yan Yan and Tianbao Yang and Zhe Li and Qihang Lin and Yi Yang , title=

  181. [190]

    International Conference on Learning Representations , year=

    Gradient Descent Maximizes the Margin of Homogeneous Neural Networks , author=. International Conference on Learning Representations , year=

  182. [191]

    2022 , url=

    Gal Vardi and Ohad Shamir and Nathan Srebro , booktitle=. 2022 , url=

    2022

  183. [192]

    Naval Research Logistics Quarterly , volume =

    Frank, Marguerite and Wolfe, Philip , title =. Naval Research Logistics Quarterly , volume =. doi:https://doi.org/10.1002/nav.3800030109 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1002/nav.3800030109 , year =

    doi

  184. [193]

    Jentzen, Arnulf and von Wurstemberger, Philippe , TITLE =. J. Complexity , FJOURNAL =. 2020 , PAGES =. doi:10.1016/j.jco.2019.101438 , URL =

    2020

  185. [194]

    Davide Gallon and Arnulf Jentzen , title=

  186. [195]

    , TITLE =

    Kechris, Alexander S. , TITLE =. 1995 , PAGES =. doi:10.1007/978-1-4612-4190-4 , URL =

    1995 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.