Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing the tuned learning rate with the running maximum of distance traveled from the start, scaled by $1/\sqrt{d}$, lets AdaGrad and Adam keep their $O(1/\sqrt{T})$ convex convergence rates with nothing to tune.

desk verdict A simple, plausible parameter-free Adam variant with the first convergence proof of its kind, but the advertised worst-case O(1/sqrt(T)) rate rests on an unproven bounded-trajectory condition and an incorrect inequality in the proof. read the letter →

arxiv 2412.19444 v2 pith:JHK4HZLH submitted 2024-12-27 cs.LG math.OCstat.ML

Yuanzhe Tao , Yifeng Liu , Huizhuo Yuan , Xun Zhou , Yuan Cao , Quanquan Gu This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 90C2590C1568W27
keywords parameter-freeoptimizationadaptivegradientmethodsAdaGradAdamconvexlearning-rate-freedeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to remove the most-tuned hyperparameter in deep learning practice, the learning rate, from two standard optimizers without losing their convergence guarantees. Its proposal is to let the algorithm set its own step size at every iteration to the furthest distance it has traveled from the starting point, scaled by $1/\sqrt{d}$ and never decreased. For the two resulting methods, AdaGrad++ and Adam++, it proves convex-optimization bounds of the same $O(1/\sqrt{T})$ order as the original AdaGrad and Adam, under a pointwise stochastic-gradient norm bound that is weaker than a global Lipschitz assumption. If the proofs are right, this is the first convergence guarantee for a parameter-free Adam, and the reported experiments on CIFAR-10 image classification and GPT-2 pretraining show the better variant matching or slightly beating the tuned baselines. Nonconvex guarantees and a proof for the weight-decay variant used in the language-model tests are left as future work.

What carries the argument

The load-bearing object is the self-generated step size $\eta_t = \max(\eta_{t-1}, \|x_t - x_0\|_2/\sqrt{d})$: the learning rate at time $t$ is simply the furthest the iterates have yet wandered from start, scaled by $1/\sqrt{d}$, and it never shrinks. The displacement term adapts the rate to the actual scale of each problem, and the running maximum keeps $\eta_t$ non-decreasing, which is exactly what the Bregman-potential proof needs to telescope the errors into one sum. The rate is then controlled by the cumulative gradient norm $\|s_\tau\|_2 = \sqrt{\sum_{t\le\tau}\|g_t\|_2^2}$, which is $O(\sqrt{T})$ in the worst case but often much smaller when gradients shrink as the model converges. The analysis imports two tools from the DoG framework — a bound on non-decreasing sums and a martingale concentration inequality — to turn the pointwise gradient assumption into a high-probability convergence statement.

What would settle it

Run AdaGrad++ on a convex problem with bounded gradient norms, such as $\ell^1$ regression whose optimum is far from the start, without any projection, and record $D_\tau = \max_{t\le\tau}\|x_t - x_*\|$ along the trajectory; if $D_\tau$ grows with $T$ on such an instance, the uniform $O(1/\sqrt{T})$ reading of Corollary 4.3 fails, because the bound carries an explicit $D_\tau^2/T$ factor.

Watch

Extended reading notes

Core claim

The central claim is that the adaptive schedule $\eta_t = \max(\eta_{t-1}, \|x_t - x_0\|_2/\sqrt{d})$ — the running maximum of the normalized distance from initialization — is a drop-in replacement for the tuned learning rate in both AdaGrad and Adam, with the same worst-case guarantees. Under convexity and Assumption 4.1, Theorems 4.2 and 5.1 bound the optimality gap at the weighted average iterate $\bar{x}_\tau$ by $O\big((D_\tau^2\sqrt{d}\,\|s_\tau\|_2 + D_\tau \eta_0 \sqrt{\theta_{\tau,\delta}\|s_\tau\|_2^2 + L^2\theta_{\tau,\delta}^2})/(T\eta_0) \cdot \log(\eta_T/\eta_0)\big)$, where $D_\tau$ is the trajectory's largest distance from the reference point and $\|s_\tau\|_2$ the cumulative gradient norm. With bounded gradients this reads as $O(1/\sqrt{T})$ up to logarithms, matching AdaGrad, and if $\|s_\tau\|_2$ grows like $T^{1/2-\alpha}$ the rate improves to $O(1/T^{1/2+\alpha})$. The paper states that Theorem 5.1 is the first convergence guarantee for a parameter-free Adam and notes that it needs a decaying momentum parameter $\lambda < 1$, or an AMSGrad-style running maximum in Case 2, whereas the experiments run the $\lambda = 1$ configuration.

Load-bearing premise

The rates are $O(1/\sqrt{T})$ only if the iterates stay within a bounded distance $D_\tau$ of the optimum, and the paper never proves that distance stays bounded for its unconstrained, non-projected updates, so a trajectory that drifts with $T$ would make the displayed bounds degrade or become vacuous.

Editorial extensions

If this is right

  • Learning-rate tuning disappears for AdaGrad and Adam: the displacement schedule yields the same worst-case $O(1/\sqrt{T})$ convex rate up to logarithmic factors, so the main remaining hyperparameters are the optimizer's internal ones such as $\beta_1$ and $\beta_2$.
  • When the cumulative gradient norm stays small, $\|s_\tau\|_2 \le G\,T^{1/2-\alpha}$, both corollaries give $O(1/T^{1/2+\alpha})$, a faster rate the paper ties to the common observation that gradient norms decay during training.
  • Theorem 5.1 is presented as the first convergence guarantee for a parameter-free variant of Adam, under the stated conditions on the momentum schedule or the second-moment running maximum.
  • In the reported runs, the better Adam++ variant matches or tops tuned Adam on most CIFAR-10 architectures and beats AdamW on GPT-2 small and medium by 0.02 in both train and test loss, with the base step size fixed at 1.
  • Storing only $x_0$ in addition to the standard Adam or AdaGrad state makes these methods more memory-efficient than earlier parameter-free adaptive methods, which keep several parameter-sized quantities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theoretical conditions and the tested configuration differ in exactly the spots the proof needs (momentum decay $\lambda<1$ or the AMSGrad max in theory, constant $\lambda=1$ in practice), which suggests the proof's technical conditions are not the operative mechanism in experiments; closing the gap by proving the rate for $\lambda=1$ would make theory and practice coincide.
  • The bounds are trajectory-readable: a practitioner can record $D_\tau$ and $\|s_\tau\|_2$ on a convex problem and see which term actually controls the optimality gap, a cheap check of whether the $1/\sqrt{T}$ interpretation is the operating regime or an artifact of assuming $D_\tau$ constant.
  • Because $\eta_t$ depends only on positions while the preconditioner absorbs the gradient scale, the algorithm is roughly invariant to rescaling the objective once the stabilizer $\delta$ is negligible, a property the paper does not state but which would explain why one base factor of 1.0 works across tasks with very different loss scales.
  • The experiments train nonconvex networks while the theorems are convex-only, so whether the displacement schedule's benefits persist for geometric rather than convexity reasons is untested; running Adam++ on a small convex benchmark would check the predicted rate directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes AdaGrad++ and Adam++, two parameter-free variants of AdaGrad and Adam in which the step size is set adaptively to the maximum distance of the iterates from initialization, normalized by sqrt(d). The main theoretical results (Theorems 4.2 and 5.1) give high-probability bounds on the function value at a weighted average iterate in terms of D_tau = max_{t<=tau} ||x_t - x*|| and the cumulative gradient norm ||s_tau||_2, under convexity and a trajectory-wise stochastic gradient norm bound. Corollaries 4.3 and 5.2 simplify these to eO(D_tau^2 G sqrt(d/T)), which the paper interprets as an O(1/sqrt(T)) rate matching AdaGrad and Adam. The paper also presents experiments on CIFAR-10 and GPT-2 comparing Adam++ with Adam, D-Adaptation, and Prodigy.

Significance. If the claimed worst-case rates held, this would be a significant contribution: the proposed update rules are strikingly simple, and the paper would provide the first convergence guarantee for a parameter-free Adam-style method, a gap in the literature. The proof structure follows the DoG template and uses a martingale concentration inequality, yielding high-probability statements rather than expectation bounds. The experiments are extensive, covering multiple architectures, datasets, and LLM pretraining, and include ablations on the initial and base learning rates. However, the advertised worst-case rate is not established, because D_tau is not bounded a priori; the results are data-dependent bounds of the DoG type, not a uniform O(1/sqrt(T)) rate. This limits the significance relative to the abstract's claim.

major comments (3)
  1. [§4.2, Corollaries 4.3/5.2, Eq. (A.6)] The central claim of an O(1/sqrt(T)) worst-case rate is not supported by the theorems as stated. The bounds in Theorems 4.2 and 5.1 depend on D_tau = max_{t<=tau} ||x_t - x*||, yet the algorithms run unconstrained on R^d with no projection, and the paper provides no argument that this quantity remains bounded along the trajectory. Consequently, the corollaries' simplification to eO(D_tau^2 G sqrt(d/T)) only yields O(1/sqrt(T)) under an additional, unproven boundedness assumption. Eq. (A.6) compounds this by claiming eta_t <= D_t via a triangle inequality that is false as written: ||x_k - x_0||_2 <= ||x_k - x*||_2 + ||x_0 - x*||_2 <= 2 D_t, so at best eta_t <= 2 D_t / sqrt(d) + epsilon. While this constant factor is absorbable, the lack of control on D_t is not, and it affects the interpretation of both corollaries.
  2. [§5.1 and Section 6] The provable version of Adam++ requires lambda in (0,1) (Theorem 5.1), but all experiments in Section 6 use lambda = 1 and, in the LLM experiments, AdamW++ rather than Adam++. As Section 7 acknowledges, AdamW++ is used without proof. Thus the experimental validation does not exercise the algorithm for which convergence guarantees are proven, weakening the paper's claim that the proposed algorithms are both simple and provably convergent in practice.
  3. [Appendix B, final bound for Case 2] The proof of Theorem 5.1 for Case 2 ends with a term sum_{k=0}^{tau-1} ||g_k||_2 / sqrt(k+1), but the theorem statement contains no such sum, only ||s_tau||_2. The displayed proof does not show the step (e.g., Cauchy-Schwarz) that would turn this sum into O(sqrt(log tau) ||s_tau||_2) or similar; this is a missing derivation in a load-bearing step for the stated bound.
minor comments (4)
  1. [Theorem 4.2 and Theorem 5.1] The notation D_tau is used for both max ||x_t - x*||_infinity and max ||x_t - x*||_2 in the theorem statements and proofs (e.g., Theorem 4.2 statement and Appendix A vs. Appendix B), making the mathematical claims ambiguous.
  2. [Theorem 5.1] The definition of x_tau is mistyped: it reads 'x_tau = (sum_{t=0}^{T-1} eta_t x_tau) / (sum_{t=0}^{T-1} eta_t)' in the theorem statement; it should be x_tau = (sum_{t=0}^{tau-1} eta_t x_t) / (sum_{t=0}^{tau-1} eta_t) as in Theorem 4.2.
  3. [Section 3, preview of Adam++] The formula for the adaptive step size in the preview, eta_t = max_{i<=t} ||x_i - x_0||_2 / sqrt(d(t+1)), is inconsistent with Algorithm 2, where eta_t = max(eta_{t-1}, ||x_t - x_0||_2 / sqrt(d)) without the (t+1) factor; the (t+1) appears only in the definition of s_t in Case 2.
  4. [Throughout Appendix B] There are several typos and incomplete definitions, such as 'the inqeuality holds' in the proof of the main bound, and the switch from D_t being the infinity norm at the start of Appendix B to the 2-norm in the noise term; these should be cleaned up.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the convergence proofs are self-contained given the external DoG lemmas, and the only author-overlapping citation is non-load-bearing; the D_tau dependence is a correctness risk, not an input-output identity.

full rationale

The derivation chain is self-contained conditional on stated trajectory-dependent quantities. The algorithms define eta_t = max_{i <= t} ||x_i - x_0||_2 / sqrt(d) (Algorithm 1, line 4; Algorithm 2, line 4), so the step size is a nondecreasing function of the running displacement from x0; this is the DoG mechanism, not a hidden identity with the conclusion. Theorems 4.2 and 5.1 are proved from standard convex-analysis inequalities and the DoG lemmas C.1-C.2 imported from Ivgi et al. (2023), an external source, with no fitted parameter being relabeled as a prediction. The author-overlapping citation Zhou et al. (2018) appears in Section 4.2 only to remark that ||s_tau||_2 is often smaller than worst case in practice and to point at a similar quantity; it is not used to establish either theorem. The paper also states its own scope limits in Section 7 ('current convergence analyses ... limited to the convex setting', 'AdamW++ ... used in our experiments without proof') and in Section 5.1 notes that the provable theory requires lambda < 1 while lambda = 1 is used empirically; these are correctness and scope caveats, not circularity. The concern that the O(1/sqrt(T)) corollaries treat D_tau = max_{t <= tau} ||x_t - x*|| as constant without proving it bounded, and the apparent triangle-inequality slip in Eq. (A.6), are correctness risks external to the circularity question: the theorems' conclusions are not equal to their assumptions by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central proof has no fitted numerical constants: the step size is a function of the trajectory, delta, beta, and eta_0 are standard hyperparameters, and all rates are stated in terms of data-dependent quantities. The main hidden cost is the assumption that D_tau and ||s_tau||_2 remain benign, plus the LLM experiments' use of the tuned AdamW learning rate as eta_0.

free parameters (4)
  • initial step size eta_0 = CIFAR: 1e-6(1+||x0||^2); LLM: 6e-4(1+||x0||^2) / 3e-4(1+||x0||^2)
    The method is called parameter-free but needs an initial eta; in the LLM experiments this is set to the tuned AdamW learning rate, so the LLM comparison is not learning-rate-free.
  • delta (regularizer in H_t) = not specified; conventional small constant such as 1e-8
    Chosen by hand; the algorithms and bounds include it, but no sensitivity analysis is given.
  • beta_1, beta_2, lambda = CIFAR: (0.9, 0.999); GPT-2: (0.9, 0.95); lambda not reported, likely 1
    Moment hyperparameters are fixed by convention, not fitted, but the paper explicitly excludes learning rates from its parameter-free definition. The lambda = 1 value used empirically is outside the theorem's lambda < 1 condition.
  • base learning rate factor c = 1.0
    Applied multiplicatively on top of the adaptive step in all experiments; the ablation explores c between 0.5 and 4, but the main results fix c = 1.
assumptions (5)
  • domain assumption Assumption 4.1: there exists a continuous l(x) with ||G(x)|| <= l(x) almost surely
    Stated before Theorem 4.2; the high-probability guarantee holds unless l(x_t) exceeds L along the trajectory.
  • domain assumption f is convex and G is an unbiased subgradient oracle with E[G(x)|x] in df(x)
    Definitions in Sections 4 and 5; all convergence theorems are restricted to this setting.
  • domain assumption For Adam++ Theorem 5.1: 0 < beta_1 < sqrt(beta_2) and 0 < lambda < 1
    Algorithm 2 line 6 introduces decaying beta_1t = beta_1 lambda^{t-1}, and the proof in Appendix B uses gamma = beta_1 / sqrt(beta_2) < 1; the practical lambda = 1 case is not covered.
  • domain assumption Corollaries assume l(x) <= G, bounded gradients
    Used to turn ||s_tau||_2 = O(sqrt(T)) into the O(1/sqrt(T)) rate; this also implicitly assumes D_tau stays bounded.
  • standard math Lemmas C.1 and C.2 from Ivgi et al. 2023 are valid martingale and step-size inequalities
    Imported without proof; the noise term and the log(eta_T / eta_0) factors in Theorems 4.2 and 5.1 rest on them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Simple and Provable Parameter-Free Adaptive Gradient Methods." pith.science (2026). https://pith.science/paper/JHK4HZLH

@misc{pith2026241219444,
  author       = {Pith},
  title        = {Pith review of: Towards Simple and Provable Parameter-Free Adaptive Gradient Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JHK4HZLH}},
  note         = {Machine review of arXiv:2412.19444}
}
read the original abstract

Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, ad-hoc tuning of learning rates poses a challenge and leads to inefficiencies in practice. To address this issue, recent research has focused on developing ``parameter-free'' algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of Adam++.

Figures

Figures reproduced from arXiv: 2412.19444 by the authors.

Figure 1
Figure 1. The results of training ResNet-18, ResNet-50, and VGG16 on CIFAR-10 with a constant [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. The results of training ResNet-18, ResNet-50, and VGG16 on CIFAR-10 with a cosine [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Comparison of training GPT-2 Small (155M) on OpenWebText. Left: Test loss. Per [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison of training GPT-2 Medium (355M) on OpenWebText. Left: Test loss. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Effect of different choices of η0 on test accuracy and training losses. When η0 is less than 10−1 , its influence on final performance is marginal. 10 1 10 0 Base step size 65 70 75 80 85 Test Accuracy (%) Adam ++ (Case 2) 10 1 10 0 Base step size 0.0 0.1 0.2 0.3 0.4 0…
Figure 6
Figure 6. Figure 6: Effect of different choices of c on test accuracy and training losses. When c is between 0.5 and 4, its influence on final performance is limited. the simple intuition, AdaGrad++ and Adam++ are guaranteed to converge with a reasonable convergence rate, and also perform…
Figure 7
Figure 7. Figure 7: The results of training VGG16 on CIFAR-10 with cosine constant learning rate schedule. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: The results of training Vision Transformer on CIFAR-10, CIFAR-100, and SVHN with [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: The results of training Wide ResNet-50-2 on CIFAR-10, CIFAR-100, and SVHN with [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: The results of training DenseNet-121 on CIFAR-10, CIFAR-100, and SVHN with a [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: The performance comparison of various optimizers—Adam, D-Adapt Adam, Prodigy, [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: The performance comparison of various optimizers—Adam, D-Adapt Adam, Prodigy, [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: The performance comparison of various optimizers—Adam, D-Adapt Adam, Prodigy, [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Test accuracy and training loss with respect to wall-clock time for different algorithms [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Enlarged version of Figure [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Enlarged version of Figure [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LightSAM: Parameter-Agnostic Sharpness-Aware Minimization

    cs.LG 2025-05 reject novelty 6.0 of 10

    An adaptive SAM variant using AdaGrad and Adam steps for both perturbation and update is claimed to converge at O(ln T / T^{1/4}) without tuning, but the Adam version still needs decaying hyperparameters and the proof...

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    and Cevher, V

    Alacaoglu, A., Malitsky, Y., Mertikopoulos, P. and Cevher, V. (2020). A new regret analysis for adam-type algorithms. In International conference on machine learning . PMLR. Bernstein, J., V ahdat, A., Yue, Y.and Liu, M.-Y.(2020). On the distance between two neural 24 0 25 50 75 100 125 150 175 200 Epoch 0 20 40 60 80T est Accuracy (%) Adam D-Adapt Adam P...

    work page 2020

  2. [2]

    Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error

    (f) SVHN, training loss Figure 9: The results of training Wide ResNet-50-2 on CIF AR-10, CIF AR-100, and SVHN with a consine learning rate schedule. Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error. Adam++ achieves performance superior or comparable to Adam. networks and the stability of learning. Advance...

    work page 2020

  3. [3]

    Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error

    (f) SVHN, training loss Figure 10: The results of training DenseNet-121 on CIF AR-10, CIF AR-100, and SVHN with a consine learning rate schedule. Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error. Adam++ achieves performance superior or comparable to Adam. Bubeck, S. et al.(2015). Convex optimization: Algo...

    work page 2015

  4. [4]

    Chen, J., Zhou, D., Tang, Y., Yang, Z., Cao, Y

    AdaGrad AdaGrad++ (f) SVHN, training loss Figure 11: The performance comparison of various optimizers—Adam, D-Adapt Adam, Prodigy, Adam++ (Case 1), Adam++ (Case 2), AdaGrad, and AdaGrad++—conducted for training a Vision Transformer on CIF AR-10, CIF AR-100, and SVHN datasets using a cosine learning rate schedule. Chen, J., Zhou, D., Tang, Y., Yang, Z., Ca...

    arXiv 2018

  5. [5]

    gap of adaptive gradient methods in training deep neural networks

    AdaGrad AdaGrad++ (f) SVHN, training loss Figure 12: The performance comparison of various optimizers—Adam, D-Adapt Adam, Prodigy, Adam++ (Case 1), Adam++ (Case 2), AdaGrad, and AdaGrad++—conducted for training a Wide ResNet-50-2 on CIF AR-10, CIF AR-100, and SVHN datasets using a cosine learning rate schedule. gap of adaptive gradient methods in training...

    work page 2022

  6. [6]

    Wide residual networks

    Zagoruyko, S.(2016). Wide residual networks. arXiv preprint arXiv:1605.07146 . Zaheer, M., Reddi, S., Sachan, D., Kale, S. and Kumar, S. (2018). Adaptive methods for nonconvex optimization. Advances in neural information processing systems

    arXiv 2016

  7. [7]

    (b) ResNet-18, CIF AR-10, training loss 0 500 1000 1500 2000 2500 Time 10 20 30 40 50 60 70 80 90T est Accuracy (%) Adam D-Adapt Adam Prodigy Adam++ (Case

    work page 2000

  8. [8]

    (c) ResNet-50, CIF AR-10, test accuracy 0 500 1000 1500 2000 2500 Time 10 2 10 1 100 Train Loss Adam D-Adapt Adam Prodigy Adam++ (Case

    work page 2000

Show all 21 references
  1. [9]

    conference on machine learning

    (d) ResNet-50, CIF AR-10, training loss Figure 14: Test accuracy and training loss with respect to wall-clock time for different algorithms in training ResNet-18 and ResNet-50 on CIF AR-10 dataset. conference on machine learning . PMLR. Cutkosky, A. and Orabona, F. (2019). Mom...

    2019

  2. [12]

    Openwebtext corpus

    Gokaslan, A.and Cohen, V.(2019). Openwebtext corpus. Gower, R. M., Defazio, A.and Rabbat, M.(2021). Stochastic polyak stepsize with a moving target. arXiv preprint arXiv:2106.11851 . 30 0 25 50 75 100 125 150 175 200 Epoch 10 20 30 40 50 60 70 80T est Accuracy (%) Adam D-Adapt...

    2019 arXiv

  3. [13]

    Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error

    (f) VGG16, training loss Figure 15: Enlarged version of Figure 1 on the results of training ResNet-18, ResNet-50, and VGG16 on CIF AR-10 with a constant learning rate schedule. Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error....

    2018

  4. [14]

    Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error

    (f) VGG16, training loss Figure 16: Enlarged version of Figure 2 on the results of training ResNet-18, ResNet-50, and VGG16 on CIF AR-10 with a cosine learning rate schedule. Each curve represents the mean of 8 random runs, with the shaded area indicating the standard error. H...

    2016 arXiv

  5. [19]

    and Yang, T

    Guo, Z., Xu, Y., Yin, W., Jin, R. and Yang, T. (2021). A novel convergence analysis for algorithms of the adam family and beyond. arXiv preprint arXiv:2104.14840 . 31 0 25 50 75 100 125 150 175 200 Epoch 40 50 60 70 80 90T est Accuracy (%) Adam D-Adapt Adam Prodigy Adam++ (Case

    2021 arXiv

  6. [20]

    Adagrad stepsizes: Sharp convergence over nonconvex landscapes

    W ard, R., Wu, X.and Bottou, L.(2020). Adagrad stepsizes: Sharp convergence over nonconvex landscapes. Journal of Machine Learning Research 21 1–30. You, Y., Gitman, I.and Ginsburg, B.(2017a). Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 . Yo...

    2020 arXiv

  7. [21]

    Zhou, D., Chen, J., Cao, Y., Tang, Y., Yang, Z.and Gu, Q. (2018). On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671 . 34

    2018 arXiv

  8. [27]

    and Tommasi, T

    33 Orabona, F. and Tommasi, T. (2017). Training deep networks without learning rates through coin betting. Advances in Neural Information Processing Systems

    2017

  9. [30]

    and Loizou, N

    Orvieto, A., Lacoste-Julien, S. and Loizou, N. (2022). Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. Advances in Neural Information Processing Systems 35 26943–26954. Paquette, C. and Scheinberg, K. (2018). A stoch...

    2022 arXiv

  10. [31]

    Very deep convolutional networks for large-scale image recognition

    Simonyan, K.and Zisserman, A.(2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 . Streeter, M.and McMahan, H. B.(2010). Less regret via online conditioning. arXiv preprint arXiv:1002.4862 . V aswani, S., Mishkin, A., Lara...

    2014 arXiv

  11. [32]

    et al.(2024)

    Defazio, A., Mehta, H., Mishchenko, K., Khaled, A., Cutkosky, A. et al.(2024). The road less scheduled. arXiv preprint arXiv:2405.15682 . Defazio, A.and Mishchenko, K.(2023). Learning-rate-free learning by d-adaptation. In Inter- national Conference on Machine Learning . PMLR....

    2024 arXiv

  12. [36]

    and Yang, T

    Chen, Z., Xu, Y., Chen, E. and Yang, T. (2018b). Sadagrad: Strongly adaptive stochastic gradient methods. In International Conference on Machine Learning . PMLR. Cutkosky, A. and Mehta, H. (2020). Momentum improves normalized sgd. In International 29 0 200 400 600 800 1000 Tim...

    2018

  13. [2011]

    Orabona, F

    Granada. Orabona, F. (2014). Simultaneous model selection and optimization through parameter-free stochastic learning. Advances in Neural Information Processing Systems

    2014

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.