Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Parallel averaged Adam (PADAM) runs several exponential-moving-average channels of a single Adam trajectory in parallel, selects at checkpoints the channel with the smallest optimization error, and claims this reaches essentially the…

desk verdict PADAM's headline result is undercut by test-loss oracle selection; the method is worth a rerun with honest evaluation, not a citation yet. read the letter →

arxiv 2505.22085 v1 pith:BX5RQDHC submitted 2025-05-28 math.OC cs.LGcs.NAmath.NA

Arnulf Jentzen , Julian Kranz , Adrian Riekert This is my paper
classification math.OCcs.LGcs.NAmath.NA MSC 68T0765K1090C15
keywords parallelaveragedAdamPADAMexponentialmovingaveragestochasticoptimizationscientificmachinelearningphysics-informedneuralnetworksdeepBSDEadaptiveaveraging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Averaging tricks such as Ruppert–Polyak averaging and exponential moving averages help stochastic optimizers, but the best averaging type and parameters depend on the problem. This paper proposes parallel averaged Adam (PADAM), which runs several different EMA-averaged copies of the same Adam trajectory at once and, during training, keeps whichever copy currently has the smallest optimization error. Because all copies share one underlying Adam trajectory and therefore the same gradients, PADAM costs no extra gradient evaluations. In 13 stochastic optimization and deep learning problems, including polynomial regression, Gaussian density learning, and scientific machine learning solvers for PDEs, optimal control, and optimal stopping, PADAM achieves essentially the smallest error among the compared optimizers in nearly all cases. The paper therefore argues that PADAM is a strong default choice for scientific machine learning and motivates further research into adaptive averaging procedures.

What carries the argument

The central object is the PADAM process in Definition 2.2: one Adam process $\vartheta^0$ whose gradients are shared by all channels, together with $K$ auxiliary EMA processes $\vartheta^k_n = \delta_{n,k}\,\vartheta^k_{n-1} + (1-\delta_{n,k})\,\Theta_n$ that track the same Adam iterate under different averaging weights $\delta_{n,k}$. At checkpoints the method evaluates the batch loss of each channel and returns the channel with the smallest loss, as specified in Algorithm 1. This carries the argument because it converts the problem of choosing an averaging schedule into a cheap online selection among schedules: each channel costs one vector recurrence per step, and all channels use the same gradients.

What would settle it

Take PADAM10 with the published schedules and run it on a new PDE or optimal-control problem where the best of the ten channels is known in hindsight; if the dynamically selected channel's error is consistently larger than the best channel's error, or if a single fixed schedule outside the bank beats all ten, the selection mechanism and schedule bank are not doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that parallel averaged Adam (PADAM), a wrapper that runs several EMA-averaged versions of a single Adam trajectory side by side and at checkpoints switches to the version with the lowest current loss, reaches essentially the smallest optimization error in nearly all of the 13 stochastic optimization and DNN learning problems studied. The comparison includes standard SGD, momentum SGD, Adam, Adam with EMA, Adam with Ruppert–Polyak averaging, and AdamW, and PADAM does this with the same number of gradient evaluations as plain Adam. The claim is empirical, established through numerical experiments spanning physics-informed neural networks, deep Galerkin and deep Ritz methods, deep BSDE, deep Kolmogorov, optimal control, and optimal stopping problems.

Load-bearing premise

PADAM's advantage rests on the hand-picked averaging schedules, which the paper says were found by trial and error, being well matched to the problems; if those schedules were effectively tuned on the 13 benchmarks, the method may not beat a single well-chosen average on new problems.

Editorial extensions

If this is right

  • PADAM3 and PADAM10 reach essentially the smallest optimization error in nearly all of the 13 problems, sometimes tied with other methods and sometimes exclusively, with no more gradient evaluations than plain Adam.
  • Because all channels share one Adam trajectory, PADAM is a drop-in addition to existing training loops for PINNs, deep Ritz, deep BSDE, deep Kolmogorov, optimal control, and optimal stopping solvers, with only minor extra computing time for large networks.
  • A small parallel bank of EMA schedules removes the need to commit in advance to a single averaging parameter; the selection rule picks among schedules online.
  • The strong empirical results support treating PADAM as a default optimizer for scientific machine learning problems and motivate further work on adaptive averaging procedures in deep neural network training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The selection mechanism does not depend on Adam specifically, so a natural extension is to run parallel averaging over AdamW, SGD with momentum, or other base optimizers and apply the same best-channel rule.
  • Channel selection uses the loss on one fresh mini-batch; on problems with very small batches or very noisy losses, a reader could test whether selecting on a larger validation batch changes the outcome, since the paper's runs use batches of 256 and checkpoint thresholds of 500 or 5000 steps.
  • The hand-tuned schedules are the fragile part; an adaptive scheduler that adds, removes, or updates channels during training could extend the idea beyond the fixed PADAM3 and PADAM10 channel sets.
  • On problems where one averaging schedule dominates, PADAM should reduce to that schedule and match its error; the open empirical question is whether the selected channel tracks the best fixed schedule on problems outside the benchmark set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes PADAM (parallel averaged Adam), a stochastic optimizer that maintains K exponentially/moving-averaged channels of a single Adam trajectory and selects among those channels during or after training. Because all channels share the same underlying gradient evaluations, the method uses no more gradient evaluations than standard Adam. The paper reports experiments on 13 problems—polynomial regression, Gaussian density fitting, deep Kolmogorov, deep Ritz, PINN, deep BSDE, optimal control, and optimal stopping—comparing PADAM3/PADAM10 against SGD, momentum SGD, AdamW, and Adam with Ruppert–Polyak or EMA averaging. The central claim is that PADAM attains essentially the smallest optimization error in nearly all examples, with Python code made publicly available.

Significance. If the empirical claim were established, PADAM would be a practically attractive, nearly free modification of Adam: the user obtains a menu of averaged trajectories at no extra gradient cost and can pick a better final iterate. The paper's strengths are its breadth (13 problems, up to 50 independent runs per problem) and the provision of source code. However, the plotted PADAM curves are a test-loss oracle envelope over the K channels, not the output of the selection rule in Algorithm 1; together with a mismatch between Definition 2.2 and Algorithm 1, this means the advertised advantage is not demonstrated for the implementable method. The trial-and-error schedules and uneven baseline learning-rate choices further weaken the general recommendation.

major comments (5)
  1. [§2.2, Algorithm 1] Algorithm 1, line 10 updates the second moment as 'v ← βn m + (1 − βn)g⊗2'. To match Definition 2.1, Eq. (3), this should read 'v ← βn v + (1 − βn)g⊗2'; as printed, the second-moment estimate is driven by the first moment m rather than by its own previous value, so the pseudocode does not implement Adam and would not reproduce the experiments. Please correct the pseudocode and confirm that the released code implements the corrected update.
  2. [§2.2, Definition 2.2 and Algorithm 1] Definition 2.2 does not describe Algorithm 1. In Definition 2.2(ii), each channel ϑ^k_n is updated using Θ_n, the PADAM output selected in (iii), whereas in Algorithm 1, line 15, channels are updated with the raw Adam iterate ϑ. Moreover, (iii) selects a channel at every step n and sets Θ_n to the selected channel, while Algorithm 1 updates all channels throughout and performs a single selection at the final step (lines 18–23). These are different methods, and the formal definition must be reconciled with the pseudocode or explicitly presented as a separate variant.
  3. [§2.2, paragraph after Algorithm 1] The curves labeled PADAM3 and PADAM10 are not produced by Algorithm 1's selection rule. The text states that 'we compute the test errors ... for the different channels ... and then plot the test error of the best performing channel'. This is a test-set oracle: at every evaluation point the plotted curve is the minimum over the K channels of their test errors. Algorithm 1 instead selects k* once at the end using a disjoint mini-batch loss. The plotted envelope cannot be worse than any fixed channel, so the claimed error reduction may be an artifact of selecting the best of K candidates with access to the evaluation labels. The central empirical claim should be demonstrated with the implementable selection rule, or the plots should be explicitly labeled as an idealized oracle-selection upper bound.
  4. [§2.2, averaging schedules] The averaging weights (i)–(x) for PADAM3 and PADAM10 are said to have been 'found by trial and error'. Because the same 13 problems are used both to tune these schedules and to demonstrate PADAM's advantage, the reported results may overstate performance on new problems. Please provide a protocol for selecting the schedules without test-set feedback (for example, using only training losses or a validation split), a sensitivity analysis over the δ_{n,k} values, or both, before drawing the general conclusion that PADAM should be considered for scientific machine learning.
  5. [§3.6, §3.11, §3.12] Several baseline comparisons use substantially smaller learning rates for SGD and momentum SGD than for the Adam-based methods, with the phrase 'to avoid divergence' (e.g., 3·10^-6 vs 3·10^-4 in §3.6, 10^-4 vs 10^-2 in §3.11, and 3·10^-4 vs 3·10^-3 in §3.12). This makes the comparison uneven, as the baselines may be far from their best settings while PADAM inherits Adam's learning rate. The authors should either tune the baselines per problem and report the best performance, or explicitly restrict the conclusion to the fixed-learning-rate protocol used here.
minor comments (5)
  1. [Abstract and throughout] There are several typos: 'movering' should be 'moving', 'parallely' should be 'in parallel', and the method name appears inconsistently as 'Padam' and 'PADAM'.
  2. [Introduction, first paragraph] 'Moroever' should be 'Moreover'; please also unify the capitalization of PADAM/Padam throughout the paper.
  3. [§3.7, §3.8, §3.12] Typos: 'Furthemore' in §3.7, 'errror' in §3.8, 'Dacry' and 'momentun' in §3.12.
  4. [Algorithm 1, line 15] Line 15 writes 'δ_{n,j}' but the schedule parameters are indexed as δ_{n,k}; please use a consistent subscript.
  5. [§2.2, evaluation paragraph] The paragraph defines n_T but does not explicitly state that the plotted quantity uses the Monte Carlo test-error approximations from the figure captions for channel selection; please make the distinction between the test loss used for plotting and the mini-batch loss used in Algorithm 1 explicit.

Circularity Check

2 steps flagged · score 6.0 of 10

PADAM's displayed error is the best-of-K channel selected on test loss, making its claimed advantage partly an artifact of selection.

  1. self definitional [Section 2.2, plotting paragraph after Algorithm 1, and the Section 3 figures]
    "For displaying the performance of the Padam algorithms, we fixed a threshhold nT ∈ {500, 5000} and compute the test errors Ltest(Θn,j) for the different channels Θn,1, . . . ,Θn,k whenever n is divisible by nT and then plot the test error of the best performing channel for the next nT gradient steps."

    The plotted PADAM error is, by construction, min_k Ltest(channel k) evaluated on the test loss used for the evaluation. Moreover, the listed PADAM3 and PADAM10 schedules include δ_n,1 = 0.999, which is exactly the 'Adam with EMA, delta = 0.999' baseline. Hence the reported PADAM curve is a lower envelope of K test-error realizations and is pointwise guaranteed to be no worse than that baseline, so the paper's central observation that PADAM beats Adam-with-EMA is a selection artifact rather than a property of the returned parameter.

  2. other [Definition 2.2(iii) versus Algorithm 1, lines 18–24]
    "Definition 2.2(iii): 'it holds for all n ∈ N that Θn = ϑ^K_n_n and (K_n+1)J_n ... = min_{k∈{1,2,...,K}} ...' ; Algorithm 1 instead ends with 'k∗ ← 1 ... return θk∗'."

    The definition selects the best channel at every step n using disjoint data blocks, while Algorithm 1 updates all channels and returns a single θ_k* chosen once at the end using different J_N-blocks for the comparison. The plotted curves follow the definition's per-n oracle choice rather than the implementable algorithm's returned parameter. Consequently, the numerical evidence does not establish that the runnable selection rule attains the plotted PADAM errors; the reported advantage is tied to a protocol that is not the optimizer as specified in Algorithm 1.

full rationale

The paper contains no mathematical derivation chain; its central claim is empirical. A substantial part of that empirical claim is constructed rather than measured: the PADAM curves shown in the figures are the best-performing channel chosen by test error, and PADAM3/PADAM10 include the exact EMA-0.999 baseline, so outperforming that baseline is guaranteed by taking a minimum. Additionally, Definition 2.2(iii) and Algorithm 1 are mutually inconsistent, so the plotted lower envelope is not what the implementable algorithm returns. The averaging schedules are admitted to have been 'found by trial and error', which further weakens any transfer claim to new problems. On the positive side, the method genuinely shares one Adam trajectory across all channels, so the no-extra-gradient-evaluations property is real and non-circular, and the final reported errors use independent simulation runs for some quantities. The central performance comparison, however, reduces in part to selecting the best of K test errors, so the paper is partially circular in its headline empirical claim.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. Its main baggage is the hand-tuned set of averaging schedules and the validation-based selection rule, which are captured as free parameters and domain assumptions.

free parameters (3)
  • Averaging schedules delta_{n,k} = e.g., 0.999, 1-n^{-0.7}, 1-0.1 exp(-2n ln(10)/N) for PADAM3; ten variants for PADAM10
    Described as 'found by trial and error', these are manually chosen hyperparameters that control the EMA channels.
  • Number of channels K = 3 and 10
    K is chosen by hand for the experiments; larger K increases memory but may improve selection.
  • Evaluation threshold n_T = 500 or 5000
    The frequency of test-error evaluation is manually set; it affects the dynamic selection behavior.
assumptions (3)
  • domain assumption The Monte Carlo approximations of the test errors with 50 (or 10) runs are accurate enough to rank the optimizers reliably.
    The paper's claim that PADAM achieves the smallest error relies on the measured errors being statistically meaningful; no significance tests are reported.
  • domain assumption The Adam trajectory is well-defined and stable for the chosen learning rates across all problems.
    The parallel channels are all EMA averages of the same Adam trajectory; if Adam diverges, PADAM also diverges.
  • standard math The loss function is differentiable and gradients are available in closed form or via autodiff.
    Definition 2.2 assumes differentiability of the loss, which holds for all benchmark problems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning." pith.science (2026). https://pith.science/paper/BX5RQDHC

@misc{pith2026250522085,
  author       = {Pith},
  title        = {Pith review of: PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BX5RQDHC}},
  note         = {Machine review of arXiv:2505.22085}
}
read the original abstract

Averaging techniques such as Ruppert--Polyak averaging and exponential movering averaging (EMA) are powerful approaches to accelerate optimization procedures of stochastic gradient descent (SGD) optimization methods such as the popular ADAM optimizer. However, depending on the specific optimization problem under consideration, the type and the parameters for the averaging need to be adjusted to achieve the smallest optimization error. In this work we propose an averaging approach, which we refer to as parallel averaged ADAM (PADAM), in which we compute parallely different averaged variants of ADAM and during the training process dynamically select the variant with the smallest optimization error. A central feature of this approach is that this procedure requires no more gradient evaluations than the usual ADAM optimizer as each of the averaged trajectories relies on the same underlying ADAM trajectory and thus on the same underlying gradients. We test the proposed PADAM optimizer in 13 stochastic optimization and deep neural network (DNN) learning problems and compare its performance with known optimizers from the literature such as standard SGD, momentum SGD, Adam with and without EMA, and ADAMW. In particular, we apply the compared optimizers to physics-informed neural network, deep Galerkin, deep backward stochastic differential equation and deep Kolmogorov approximations for boundary value partial differential equation problems from scientific machine learning, as well as to DNN approximations for optimal control and optimal stopping problems. In nearly all of the considered examples PADAM achieves, sometimes among others and sometimes exclusively, essentially the smallest optimization error. This work thus strongly suggest to consider PADAM for scientific machine learning problems and also motivates further research for adaptive averaging procedures within the training of DNNs.

Figures

Figures reproduced from arXiv: 2505.22085 by the authors.

Figure 1
Figure 1. Polynomial Regression Problem 0 25000 50000 75000 100000 125000 150000 175000 200000 gradient steps 10 4 10 3 10 2 10 1 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 Polynomial regression problem, 50 runs 9 [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Supervised deep ANN learning of Gaussian densities 0 10000 20000 30000 40000 50000 gradient steps 10 3 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 Supervised model approximating Gaussian density, 50 runs 3.3 Deep Kolmogorov method (DKM) for heat equation In the next example we employ the deep Kolmogorov method (DKM) from Beck et al. [6… view at source ↗
Figure 3
Figure 3. Deep Kolmogorov method for Heat equations [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Deep Kolmogorov method for Black Scholes equations [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Quadratic problem 0 20000 40000 60000 80000 100000 gradient steps 10 7 10 6 10 5 10 4 10 3 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 Quadratic Problem, 50 runs 17 [PITH_FULL_IMAGE:figu…
Figure 6
Figure 6. Figure 6: Deep Ritz for a Poisson equation 0 10000 20000 30000 40000 50000 gradient steps 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 DRM for 10d Poisson equation, 50 runs 19 [PITH_FULL_IMAGE:figu…
Figure 7
Figure 7. Figure 7: Deep Ritz for p-Laplace equation 0 5000 10000 15000 20000 25000 30000 gradient steps 10 4 10 3 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 DRM for p-Laplace equation (p=9), 50 runs 3.8 De…
Figure 8
Figure 8. Figure 8: Optimal Control Problem 0 500 1000 1500 2000 2500 3000 3500 4000 gradient steps 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 Optimal Control Problem, 10 runs 23 [PITH_FULL_IMAGE:figures/f…
Figure 9
Figure 9. Figure 9: BSDE 0 5000 10000 15000 20000 25000 30000 gradient steps 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 BSDE for 10d Hamilton-Jacobi-Bellmann equation, 50 runs 25 [PITH_FULL_IMAGE:figures/f…
Figure 10
Figure 10. Figure 10: PINN for Burgers equation 0 10000 20000 30000 40000 50000 gradient steps 10 5 10 4 10 3 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 PINN for Burger's equation, 50 runs 27 [PITH_FULL_IMA…
Figure 11
Figure 11. Figure 11: PINN for Allen Cahn equation 0 5000 10000 15000 20000 25000 30000 gradient steps 10 5 10 4 10 3 10 2 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 PINNs for 3-dimensional Allen-Cahn equation, 5…
Figure 12
Figure 12. Figure 12: PINNs for Darcy Flow 0 20000 40000 60000 80000 100000 gradient steps 10 3 10 2 10 1 10 0 10 1 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 PINNs for Darcy Flow, 50 runs 3.13 Deep optimal stopping (DOS) …
Figure 13
Figure 13. Figure 13: Optimal Stopping Problem 0 2000 4000 6000 8000 10000 gradient steps 10 1 10 0 error SGD SGD momentum ADAM ADAMW ADAM with Ruppert-Polyak average ADAM with EMA, delta = 0.999 PADAM3 PADAM10 Optimal Stopping Problem, 10 runs 4 Conclusion In this work we apply the propos…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization

    stat.ML 2026-01 conditional novelty 7.0 of 10

    Momentum SGD pays a provable tracking penalty under distribution shift — lag growing as (1−β)⁻¹ and tracking floors worse than vanilla SGD in drift-dominated regimes.

Reference graph

Works this paper leans on

55 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Adam with model exponential moving average is effective for nonconvex optimization

    Ahn, K., and Cutkosky, A. Adam with model exponential moving average is effective for nonconvex optimization. arXiv:2405.18199 (2024), 25 pages

    arXiv 2024

  2. [2]

    General framework for online-to- nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization

    Ahn, K., Magakyan, G., and Cutkosky, A. General framework for online-to- nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization. arXiv:2411.07061 (2024), 32 pages

    arXiv 2024

  3. [3]

    Athiwaratkun, B., Finzi, M., Izmailov, P., and Wilson, A. G. There are many consistent explanations of unlabeled data: Why you should average. In International Conference on Learning Representations (2019)

    work page 2019

  4. [4]

    Learning Theory from First Principles

    Bach, F. Learning Theory from First Principles . Adaptive Computation and Machine Learning series. MIT Press, 2024

    work page 2024

  5. [5]

    Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization

    Barakat, A., and Bianchi, P. Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization. SIAM J. Optim. 31 , 1 (2021), 244–274

    work page 2021

  6. [6]

    Solving the Kolmogorov PDE by means of deep learning

    Beck, C., Becker, S., Grohs, P., Jaafari, N., and Jentzen, A. Solving the Kolmogorov PDE by means of deep learning. Journal of Scientific Computing 88 , 3 (2021). 34

    work page 2021

  7. [7]

    An overview on deep learning-based approximation methods for partial differential equations

    Beck, C., Hutzenthaler, M., Jentzen, A., and Kuckuck, B. An overview on deep learning-based approximation methods for partial differential equations. Discrete Contin. Dyn. Syst. Ser. B 28 , 6 (2023), 3697–3746

    work page 2023

  8. [8]

    Solving high-dimensional optimal stopping problems using deep learning

    Becker, S., Cheridito, P., Jentzen, A., and Welti, T. Solving high-dimensional optimal stopping problems using deep learning. European J. Appl. Math. 32 , 3 (2021), 470–514

    work page 2021

Show all 55 references
  1. [9]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Lit...

    2020

  2. [10]

    G., Suau Cuadros, X., and Webb, R

    Busbridge, D., Ramapuram, J., Ablin, P., Likhomanenko, T., Dhekane, E. G., Suau Cuadros, X., and Webb, R. How to scale your EMA. In Advances in Neural Information Processing Systems (2023), A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36, Cur...

    2023

  3. [11]

    Scientific machine learning through physics-informed neural networks: where we are and what’s next

    Cuomo, S., Schiano Di Cola, V., Giampaolo, F., Rozza, G., Raissi, M., and Piccialli, F. Scientific machine learning through physics-informed neural networks: where we are and what’s next. J. Sci. Comput. 92 , 3 (2022), Paper No. 88, 62

    2022

  4. [12]

    A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A

    Defazio, A., Yang, X. A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A. The Road Less Scheduled. arXiv:2405.15682 (2024), 29 pages

    2024 arXiv

  5. [13]

    A Simple Convergence Proof of Adam and Adagrad

    D´efossez, A., Bottou, L., Bach, F., and Usunier, N. A Simple Convergence Proof of Adam and Adagrad. Transactions on Machine Learning Research (2022)

    2022

  6. [14]

    General multilevel adaptations for stochastic approximation algorithms II: CLTs

    Dereich, S. General multilevel adaptations for stochastic approximation algorithms II: CLTs. Stochastic Process. Appl. 132 (2021), 226–260

    2021

  7. [15]

    Convergence rates for the Adam optimizer

    Dereich, S., and Jentzen, A. Convergence rates for the Adam optimizer. arXiv:2407.21078 (2024), 43 pages

    2024 arXiv

  8. [16]

    On the existence of minimizers in shallow residual relu neural network optimization landscapes

    Dereich, S., Jentzen, A., and Kassing, S. On the existence of minimizers in shallow residual relu neural network optimization landscapes. SIAM J. Numer. Anal. 62 , 6 (2024), 2640–2666

    2024

  9. [17]

    Averaged adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems

    Dereich, S., Jentzen, A., and Riekert, A. Averaged adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems. arXiv:2501.06081 (2025), 25 pages. 35

    2025 arXiv

  10. [18]

    Central limit theorems for stochastic gradient descent with averaging for stable manifolds

    Dereich, S., and Kassing, S. Central limit theorems for stochastic gradient descent with averaging for stable manifolds. Electron. J. Probab. 28 (2023), Paper No. 57. 48

    2023

  11. [19]

    On the existence of optimal shallow feedforward networks with ReLU activation

    Dereich, S., and Kassing, S. On the existence of optimal shallow feedforward networks with ReLU activation. J. Mach. Learn. 3 , 1 (2024), 1–22

    2024

  12. [20]

    General multilevel adaptations for stochastic approximation algorithms of Robbins-Monro and Polyak-Ruppert type.Numer

    Dereich, S., and M¨uller-Gronbach, T. General multilevel adaptations for stochastic approximation algorithms of Robbins-Monro and Polyak-Ruppert type.Numer. Math. 142, 2 (2019), 279–328

    2019

  13. [21]

    Uniform convergence guarantees for the deep ritz method for nonlinear problems

    Dondl, P., M ¨uller, J., and Zeinhofer, M. Uniform convergence guarantees for the deep ritz method for nonlinear problems. Advances in Continuous and Discrete Models 2022, 1 (2022)

    2022

  14. [22]

    Deep learning-based numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equations

    E, W., Han, J., and Jentzen, A. Deep learning-based numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equations. Commun. Math. Stat. 5 , 4 (2017), 349–380

    2017

  15. [23]

    Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning

    E, W., Han, J., and Jentzen, A. Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning. Nonlinearity 35, 1 (2021), 278

    2021

  16. [24]

    The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems

    E, W., and Yu, B. The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat. 6 , 1 (2018), 1–12

    2018

  17. [25]

    Optimal non-asymptotic bound of the Ruppert-Polyak averaging without strong convexity

    Gadat, S., and Panloup, F. Optimal non-asymptotic bound of the Ruppert-Polyak averaging without strong convexity. arXiv:1709.03342 (2017), 41 pages

    2017 arXiv

  18. [26]

    Blow up phenomena for gradient de- scent optimization methods in the training of artificial neural networks

    Gallon, D., Jentzen, A., and Lindner, F. Blow up phenomena for gradient de- scent optimization methods in the training of artificial neural networks. arXiv:2211.15641 (2022), 84 pages

    2022 arXiv

  19. [27]

    Neural networks-based algorithms for stochastic control and PDEs in finance

    Germain, M., Pham, H., and W arin, X. Neural networks-based algorithms for stochastic control and PDEs in finance. arXiv:2101.08068 (2021), 27 pages

    2021 arXiv

  20. [28]

    Stochastic weight averaging revisited

    Guo, H., Jin, J., and Liu, B. Stochastic weight averaging revisited. Applied Sciences 13, 5 (2023), 2935

    2023

  21. [29]

    Han, J., Jentzen, A., and E, W.Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. USA 115 , 34 (2018), 8505–8510

    2018

  22. [30]

    Recent developments in machine learning methods for stochastic control and games

    Hu, R., and Lauri `ere, M. Recent developments in machine learning methods for stochastic control and games. Numer. Algebra Control Optim. 14 , 3 (2024), 435–525

    2024

  23. [31]

    Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv:1803.05407 (2018), 12 pages. 36

    2018 arXiv

  24. [32]

    Mathematical Introduc- tion to Deep Learning: Methods, Implementations, and Theory

    Jentzen, A., Kuckuck, B., and von Wurstemberger, P. Mathematical Introduc- tion to Deep Learning: Methods, Implementations, and Theory. arXiv:2310.20360 (2023), 712 pages

    2023 arXiv

  25. [33]

    On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

    Jentzen, A., and Riekert, A. On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. J. Mach. Learn. 1, 2 (2022), 141–246

    2022

  26. [34]

    P., and Ba, J

    Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014), 15 pages

    2014 arXiv

  27. [35]

    SAD Neural Net- works: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

    Kranz, J., Gallon, D., Dereich, S., and Jentzen, A. SAD Neural Net- works: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures. arXiv:2505.09572 (2025), 27 pages

    2025

  28. [36]

    Convergence of Adam Under Relaxed As- sumptions

    Li, H., Rakhlin, A., and Jadbabaie, A. Convergence of Adam Under Relaxed As- sumptions. arXiv:2304.13972 (2023), 35 pages

    2023 arXiv

  29. [37]

    Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression

    Li, X., and Gu, Q. Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression. arXiv:2502.14123 (2025), 34 pages

    2025 arXiv

  30. [38]

    Summary of ChatGPT-related research and perspective towards the future of large language models

    Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., and Ge, B. Summary of ChatGPT-related research and perspective towards the future of large language models. arXiv:2304.018...

    2023 arXiv

  31. [39]

    Decoupled weight decay regularization

    Loshchilov, I., and Hutter, F. Decoupled weight decay regularization. arXiv:1711.05101 (2017), 19 pages

    2017 arXiv

  32. [40]

    Gradient Descent Maximizes the Margin of Homogeneous Neural Networks

    Lyu, K., and Li, J. Gradient Descent Maximizes the Margin of Homogeneous Neural Networks. arXiv:1906.05890 (2020), 52 pages

    2020 arXiv

  33. [41]

    D., and Blei, D

    Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as ap- proximate Bayesian inference. arXiv:1704.04289 (2017), 35 pages

    2017 arXiv

  34. [42]

    Exponential moving average of weights in deep learning: Dynamics and benefits

    Morales-Brotons, D., Vogels, T., and Hendrikx, H. Exponential moving average of weights in deep learning: Dynamics and benefits. Transactions on Machine Learning Research (2024)

    2024

  35. [43]

    Topological properties of the set of functions generated by neural networks of fixed size

    Petersen, P., Raslan, M., and Voigtlaender, F. Topological properties of the set of functions generated by neural networks of fixed size. Found. Comput. Math. 21 , 2 (2021), 375–444

    2021

  36. [44]

    Continuous-time stochastic control and optimization with financial applications, vol

    Pham, H. Continuous-time stochastic control and optimization with financial applications, vol. 61 of Stochastic Modelling and Applied Probability . Springer-Verlag, Berlin, 2009. 37

    2009

  37. [45]

    Polyak, B. T. A new method of stochastic approximation type. Avtomat. i Telemekh. , 7 (1990), 98–107

    1990

  38. [46]

    T., and Juditsky, A

    Polyak, B. T., and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30 , 4 (1992), 838–855

    1992

  39. [47]

    Zero-shot text-to-image generation

    Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. arXiv:2102.12092 (2021), 20 pages

    2021 arXiv

  40. [48]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. On the Convergence of Adam and Beyond. arXiv:1904.09237 (2019), 23 pages

    2019 arXiv

  41. [49]

    High- resolution image synthesis with latent diffusion models.arXiv:2112.10752 (2022), 45 pages

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High- resolution image synthesis with latent diffusion models.arXiv:2112.10752 (2022), 45 pages

    2022 arXiv

  42. [50]

    An overview of gradient descent optimization algorithms

    Ruder, S. An overview of gradient descent optimization algorithms. arXiv:1609.04747 (2017), 14 pages

    2017 arXiv

  43. [51]

    Efficient estimations from a slowly convergent Robbins-Monro process.Cor- nell University Operations Research and Industrial Engineering, hdl.handle.net/1813/8664 (1988), 1–34

    Ruppert, D. Efficient estimations from a slowly convergent Robbins-Monro process.Cor- nell University Operations Research and Industrial Engineering, hdl.handle.net/1813/8664 (1988), 1–34

    1988

  44. [52]

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv:2205.11...

    2022 arXiv

  45. [53]

    Training tra- jectories, mini-batch losses and the curious role of the learning rate

    Sandler, M., Zhmoginov, A., Vladymyrov, M., and Miller, N. Training tra- jectories, mini-batch losses and the curious role of the learning rate. arXiv:2301.02312 (2023), 21 pages

    2023 arXiv

  46. [54]

    On Margin Maximization in Linear and ReLU Networks

    V ardi, G., Shamir, O., and Srebro, N. On Margin Maximization in Linear and ReLU Networks. arXiv:2110.02732 (2022), 30 pages

    2022 arXiv

  47. [55]

    Deep learning with Elastic Averaging SGD

    Zhang, S., Choromanska, A., and LeCun, Y. Deep learning with Elastic Averaging SGD. arXiv:1412.6651 (2014), 24 pages. 38

    2014 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.