Pith. sign in

REVIEW 3 major objections 5 minor 56 references

AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AutoSGD automatically chooses the learning rate at every step of SGD, and the paper proves that the resulting method converges at a linear rate at selected episode times.

desk verdict Clever three-stream learning-rate selection with honest but consequential theory gap: the convergence proof doesn't cover the implemented decision rule. read the letter →

arxiv 2505.21651 v1 pith:D6EA4MWP submitted 2025-05-27 cs.LG math.OCstat.COstat.ML

Nikola Surjanovic , Alexandre Bouchard-Côté , Trevor Campbell This is my paper
classification cs.LGmath.OCstat.COstat.ML MSC 90C1568W40
keywords learningrateadaptationstochasticgradientdescentparameter-freeoptimizationadaptivestepsizeepisode-baseddecisionprocessconvergencetheorylinesearchalternatives
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the learning rate of stochastic gradient descent can be chosen automatically, with no user tuning, by comparing three nearby candidate rates and switching rates only after a statistical decision. The proposed method, AutoSGD, runs three parallel SGD streams at rates $c\gamma_t$, $\gamma_t$, and $C\gamma_t$ within each episode, and uses paired noisy objective evaluations to decide whether to increase, keep, decrease, or restart with a smaller rate. The authors prove that AutoSGD converges under assumptions on this decision process: at a sequence of random episode times $t_n$, the expected objective satisfies $\mathbb{E}[f(x_{t_n})] \le \nu^n f_0$ with $\mathbb{E}[t_n] \le a+bn$, and the deterministic counterpart AutoGD converges linearly at a rate related to $1/(\kappa\log\kappa)$ on Polyak–Łojasiewicz functions. If the claims hold, users can avoid hand-picking decay schedules and initial rates, since the method learns to warm up and anneal on its own. Experiments across classical optimization problems and machine learning training tasks show AutoSGD matching or beating tuned alternatives with minimal tuning.

What carries the argument

The engine of the paper is the episode-based three-stream comparison. An episode is a random-length block of SGD iterations during which a fixed grid of three learning rates $\{c\gamma_t, \gamma_t, C\gamma_t\}$ is held constant; at each inner step, three parallel streams are updated from the same starting point, and a stopping rule decides which of four moves happens next: increase to $C\gamma_t$, stay at $\gamma_t$, decrease to $c\gamma_t$, or restart at the previous episode start with a smaller rate. The decision rule is driven by a statistic $Z_{t,k}$ built from two independent noisy evaluations of the objective difference for each stream, normalized by their spread; it can be updated online in constant memory and is compared against a threshold $z^*=1.96$ after at least $M=30$ samples. For the convergence proof, the paper introduces a drift function $V(x,\gamma)$ on the joint iterate/learning-rate state and uses drift conditions of the type standard in Markov chain theory to bound the frequency of excursions into the 'bad' region where the rate is too large; this drift framework is what converts the inner-stream descent estimates into the geometric bound on the episode-time sequence.

What would settle it

On a one-dimensional stochastic quadratic with known smoothness constant $\alpha$, initialize $\gamma_0 > (C\alpha)^{-1}$ and run the implemented decision rule; if the 'increase' move is ever chosen while $\gamma_t > (C\alpha)^{-1}$, then Assumption C.6 fails and Theorem 4.6 does not apply to the implemented algorithm. A Monte Carlo estimate of $P(I_t \mid x_t, \gamma_t)$ in that regime, computed from many runs, would settle whether this condition is violated.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a gradient method can make its own learning-rate schedule by treating rate selection as an online comparison between neighboring rates, and this self-selection does not cost convergence. The deterministic algorithm AutoGD looks one step ahead at $c\gamma_t$, $\gamma_t$, and $C\gamma_t$, moves with the best rate, and, if all three increase the objective, stands still and shrinks the rate; the authors show its objective values always converge and that, under Polyak–Łojasiewicz, unimodality, and smoothness conditions, it converges linearly at rate roughly $1/(\kappa\log\kappa)$, compared with $1/\kappa$ for optimally tuned gradient descent. The stochastic algorithm AutoSGD extends the same idea with random episodes: three parallel SGD streams run from the same starting point with the three rates, a test statistic based on paired noisy objective differences decides the next move, and a restart move resets to the previous episode's starting point with a smaller rate when every rate appears to increase the objective. The main theorem, Theorem 4.6, states that under the paper's conditions there are random times $t_n<\infty$ almost surely with $\mathbb{E}[f(x_{t_n})] \le \nu^n f_0$ and $\mathbb{E}[t_n] \le a+bn$; the proof uses nonnegative-supermartingale and drift arguments drawn from Markov chain theory rather than the usual stochastic-optimization toolkit.

Load-bearing premise

The load-bearing premise is that the specific threshold-and-sample-size rule used in the algorithm's implementation satisfies the paper's decision-process assumptions; the paper says it does not verify this for its own recommended rule, and one of the assumptions (never increasing the rate once it is too large) it calls unrealistic to enforce in practice.

Editorial extensions

If this is right

  • Users no longer need to specify a learning-rate schedule or carefully pick an initial rate: AutoSGD itself performs warmup and annealing, and even a poor initial $\gamma_0$ is corrected within $|\log_2(\gamma/\gamma_0)|$ iterations.
  • If the theory holds, AutoSGD's convergence is linear in episode count at selected times, with the deterministic linear rate scaling like $1/(\kappa\log\kappa)$ rather than $1/\kappa$ for tuned GD.
  • The restart move makes the method robust: when all candidate rates increase the objective, the iterate returns to the prior episode start and the rate shrinks, which the authors show prevents the exponential divergence that occurs if one is forced to move at every step.
  • Empirically, AutoSGD matches or outperforms tuned SGD, invariant step-size methods, and stochastic line search across regression, matrix factorization, multiclass classification, and deep-network fine-tuning, with less oscillation near the optimum than the compared invariant method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the biggest open step is closing the gap between Theorem 4.6 and the implemented Algorithm 2; the paper states it does not verify its convergence assumptions for the default $z^*=1.96$, $M=30$ rule, so verifying or weakening Assumptions C.5–C.8 would make the practical method the one with a theorem.
  • Editorial inference: the paired-difference statistic $Z_{t,k}$ is a reusable subroutine: any optimizer that can evaluate the objective noisily could use the same constant-memory comparison to adapt a step size, per coordinate or per layer, not just a single global rate.
  • Editorial inference: the episode-with-restart design suggests a general fail-safe pattern for other adaptive methods — when every candidate step size increases the objective, reset and shrink — which could be grafted onto line-search or rate-scheduling algorithms to remove instability from too-large initial rates.
  • Editorial inference: a direct testable extension would replace the fixed threshold $z^*=1.96$ with a threshold that shrinks as $f(x_t)$ decreases, since the paper's assumptions require episodes to lengthen when the objective is small; this might match the theory more closely while preserving the constant-memory implementation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AutoSGD, an SGD method with automatic learning-rate selection. At each episode, three parallel SGD streams run with learning rates cγ_t, γ_t, and Cγ_t; a decision rule based on paired objective differences (Algorithm 2) chooses whether to increase, keep, decrease, or restart-and-decrease the learning rate. A deterministic counterpart, AutoGD, is also introduced. The authors claim convergence theory for both methods (Theorem 4.5 for AutoGD, Theorem 4.6/C.12 for AutoSGD) and report experiments on classical optimization problems and fine-tuning tasks where AutoSGD matches or outperforms tuned baselines.

Significance. If the convergence claims were fully supported, the paper would make a valuable contribution: a learning-rate-free SGD variant with linear convergence at episode times and seemingly robust empirical performance. The AutoGD theory (Proposition 4.1 and Theorem 4.5) is coherent under the stated unimodality and smoothness assumptions, and the empirical study is broad, covering multiple classical objectives and real ML fine-tuning tasks. However, the central AutoSGD convergence theorem is not established for the algorithm that is actually implemented and evaluated. The paper explicitly disclaims applicability of its own theory to the recommended decision process, and one key assumption is admitted to be unenforceable in practice. The result is that the abstract's claim 'We introduce theory supporting the convergence of AutoSGD' is currently unsupported for the evaluated method.

major comments (3)
  1. [C.2, Theorem C.12 / Theorem 4.6] The main AutoSGD convergence theorem is conditional on Assumptions C.4–C.8, but the paper states in Section C.2: 'We do not attempt to apply these results to our specific recommended decision process from Section C.1.' Since Algorithm 2 with z* = 1.96 and M = 30 is the decision process used in every AutoSGD experiment, Theorem 4.6/C.12 does not establish convergence of the evaluated algorithm. This is not merely a presentation issue: Assumption C.6 requires P(I_t | x_t, gamma_t) = 0 when gamma_t > (C alpha)^{-1}, and the paper admits this 'is not one we can realistically enforce in practice' because alpha is unknown to the algorithm. The central theoretical claim therefore needs either a proof that the implemented decision rule satisfies the assumptions (e.g., by constructing a data-dependent upper bound on gamma_t), or an explicit reframing of the theory as applying to an abstract family of decision processes, with the implemented rule presented as a heuristic whose convergence is not proven.
  2. [C.2, note after Theorem C.12] Theorem C.12 bounds expected values of f at episode-start times t_n and the expected number of episodes, but the paper immediately notes that 'Theorem C.12 does not say anything about the total number of SGD iterations; in fact, none of the assumptions above preclude the possibility that tau_t = infinity.' Since AutoSGD's computational cost, and every experimental plot in Section 5, is measured in SGD iterations, the theorem does not certify finite-time convergence of the implemented stochastic process. The authors should either prove tau_t < infinity almost surely (or supply a bound on the total iteration count), or state unambiguously that the episode-level result leaves open the possibility of infinite loops within an episode.
  3. [Algorithm 1, line 13 vs. Appendix E.1] There is a discrepancy between the analyzed/pseudocode restart rule and the rule used in the classical experiments. Algorithm 1 sets gamma_{t+1} <- c gamma_t when R_t = 1, and the theory in Section C.2 models a factor-c decrease. However, Appendix E.1 says: 'For iterations where a restart occurs (R_t = 1), we decreased the learning rates in the grid by a factor of c^2 for the classical optimization experiments.' Thus the classical experiments use a more aggressive restart decrease than the pseudocode and the theory. The pseudocode, theoretical model, and experimental implementation should be aligned, or the implemented rule should be reported consistently throughout.
minor comments (5)
  1. [Section 2] In the description of the empirical risk minimization setting, 'f(x, u) = f_u' should presumably read 'f(x, u) = f_u(x)'; as written, the dependence on x is missing.
  2. [Section 3.2.1 and Algorithm 1] The three SGD streams are denoted by typographically very similar symbols (x_{t,k}, x_{t,k}, and an accented variant), which makes the algorithm and the Z-statistic definitions hard to follow. Renaming the streams x^{low}, x^{mid}, x^{high} would improve readability substantially.
  3. [Theorem 4.5] The display defining t0 appears to contain unmatched parentheses: 't0 = max(0, log_{c^{-1}} gamma0/gamma , log_C ((sqrt(2)-1)/(L gamma0))' is missing a closing parenthesis after the second log argument. Please correct the typesetting.
  4. [Algorithm 2] The condition in Algorithm 2 that a candidate gamma' requires Z^{(gamma')} > z* and Z^{(gamma'')} >= -z* for all smaller gamma'' is never motivated in the main text; a short explanation of why all smaller rates must be non-significantly worse would help the reader understand the decision rule.
  5. [Section 5.1] The paper notes that the iteration count for NMLS is not scaled by the number of backtracks, but for other methods the x-axis counts all inner SGD iterations. Please state explicitly for every optimizer what one 'iteration' counts, so that the comparison is unambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the AutoSGD convergence theorem is a conditional result explicitly scoped away from the proposed decision process; remaining gaps are theory-practice mismatches, not circular reductions.

full rationale

The paper's derivation chain is not circular. Theorem 4.5 for AutoGD is proved from Polyak-Lojasiewicz, unimodality, and smoothness assumptions via Taylor remainder bounds and explicit constants; the target convergence rate is not assumed. For AutoSGD, Theorem 4.6/Theorem C.12 is a conditional theorem: it states that if a general decision process satisfies Assumptions C.4-C.8, then at selected episode times E[f] decays geometrically. The assumptions concern probabilities of the decision indicators, restart/increase behavior, and episode-length bounds; they are sufficient conditions, not the theorem's conclusion restated. The paper explicitly does not claim the theorem covers its recommended Algorithm 2 decision process: Section C.2 states, 'We do not attempt to apply these results to our specific recommended decision process from Section C.1,' and it concedes that Assumption C.6's requirement P(I_t | x_t, gamma_t) = 0 for gamma_t > (C alpha)^{-1} 'is not one we can realistically enforce in practice.' These are honest scoping gaps between theory and implementation, not instances of fitting a parameter and renaming it a prediction, nor of importing a conclusion through self-citation. The empirical evaluation is against external benchmarks and standard baselines, and the tail-averaging appendix explicitly identifies its method as known suffix averaging. No equation reduces to another by construction, and no load-bearing step is justified solely by the authors' prior work.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central theoretical claim rests on a nested set of assumptions. The AutoGD rate theorem requires unimodality along gradient lines (Assumption 4.3) and a technical smoothness bound (Assumption 4.4). The AutoSGD convergence theorem requires five assumptions (C.4-C.8), several of which are ad hoc properties of the decision process that the authors do not verify for the implemented Algorithm 2. No new physical or mathematical entities are introduced.

free parameters (4)
  • Expansion factor C = 2 (default)
    Hand-chosen multiplier for the large learning rate candidate in the grid {c*gamma, gamma, C*gamma}. The theory states AutoGD works for a wide range of values, so it is not fitted to data.
  • Contraction factor c = 1/2 (default)
    Hand-chosen multiplier for the small learning rate candidate and for shrinking the rate after a restart. Set via C = 1/c in all experiments.
  • Decision statistic threshold z* = 1.96
    Hand-chosen threshold in Algorithm 2, resembling a 95% normal quantile. Not fitted to data, but it controls the episode length and whether the algorithm increases, decreases, or restarts.
  • Minimum samples M = 30
    Hand-chosen minimum number of inner iterations before a decision can be made in Algorithm 2. Affects how quickly the learning rate can adapt.
assumptions (7)
  • domain assumption Assumption 4.3: For all x, f(x - gamma * grad f(x)) is unimodal in gamma with a unique minimum.
    Used in Theorem 4.5 to guarantee the best learning rate in the grid is the one closest to the position-dependent optimal rate. The paper notes it holds for strongly convex functions, but it is very strong for general non-convex objectives.
  • domain assumption Assumption 4.4: f is L-smooth, twice continuously differentiable, and there exists gamma > 1/L such that gamma - (gamma^2/2) G(x, gamma) <= 1/(2L).
    Technical condition needed in Theorem 4.5 to ensure the learning rate grid contains a rate that achieves a constant fraction of the maximum possible descent.
  • domain assumption Assumption C.4: f is twice continuously differentiable, mu-PL, and the stochastic gradient satisfies a variance bound with constants alpha, beta.
    Standard stochastic optimization assumption used in Lemma C.9 to bound the expected descent of a single inner SGD stream.
  • ad hoc to paper Assumption C.5: The decision indicators I_t, S_t, D_t are negatively correlated with the objective value at the corresponding stream endpoint.
    This assumption encodes that the decision process tends to select lower objective values. It is not verified for Algorithm 2; the paper says 'We expect that this assumption is satisfied by reasonable decision processes.'
  • ad hoc to paper Assumption C.6: The decision process has false-decision probability at most delta, and P(I_t | x_t, gamma_t) = 0 when gamma_t > (C alpha)^{-1}.
    The hard upper bound on gamma_t is required for the proof of Lemma C.10. The paper explicitly says 'this is not one we can realistically enforce in practice' because alpha is unknown.
  • ad hoc to paper Assumption C.7: Episode lengths satisfy the three conditions in Eqs. (7)-(9), including that the test takes at least O(1/f(x_t)) time to decide when the rate is small.
    These conditions are tailored to force non-negligible descent per episode. The paper says the proposed decision process 'should adhere to this behaviour' but does not prove it.
  • ad hoc to paper Assumption C.8: It is unlikely that any inner SGD stream terminates at a vanishingly small f(x_t) in a single episode.
    The paper itself says 'this assumption is probably not even necessary' and that a more careful analysis might avoid it. It is used in Lemma C.11 to bound the drift function.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent." pith.science (2026). https://pith.science/paper/D6EA4MWP

@misc{pith2026250521651,
  author       = {Pith},
  title        = {Pith review of: AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6EA4MWP}},
  note         = {Machine review of arXiv:2505.21651}
}
read the original abstract

The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks.

Figures

Figures reproduced from arXiv: 2505.21651 by the authors.

Figure 1
Figure 1. Performance of deterministic optimizers on the non-convex objective function [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of the AutoSGD learning rate selection procedure with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of the new learning rate selection procedure within AutoSGD (plotted on [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Classical optimization settings with learning rates chosen that are favourable to the majority [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Classical optimization settings with initially small learning rates for the different optimizers [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Training loss for various optimizers on three out of five ML data sets. Solid lines represent [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Validation loss for various optimizers on three out of five ML data sets. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: AutoSGD learning rates for each of the classical optimization problems. In the second row, [PITH_FULL_IMAGE:figures/full_fig_p038_8.png]
Figure 9
Figure 9. Figure 9: AutoSGD learning rates for each of the ML training optimization problems. [PITH_FULL_IMAGE:figures/full_fig_p038_9.png]
Figure 10
Figure 10. Figure 10: Classical optimization settings with learning rates chosen that are favourable to the [PITH_FULL_IMAGE:figures/full_fig_p039_10.png]
Figure 11
Figure 11. Figure 11: Classical optimization settings with initially small learning rates for the different optimizers [PITH_FULL_IMAGE:figures/full_fig_p039_11.png]
Figure 12
Figure 12. Figure 12: Additional matrix factorization results with [PITH_FULL_IMAGE:figures/full_fig_p040_12.png]
Figure 13
Figure 13. Figure 13: Training loss for various optimizers on additional ML data sets. Solid lines represent [PITH_FULL_IMAGE:figures/full_fig_p040_13.png]
Figure 14
Figure 14. Figure 14: Validation loss for various optimizers on additional ML data sets. [PITH_FULL_IMAGE:figures/full_fig_p041_14.png]
Figure 15
Figure 15. Figure 15: Simulation results for the Beale (left column), Matyas (middle column), and Rosenbrock [PITH_FULL_IMAGE:figures/full_fig_p041_15.png]
Figure 16
Figure 16. Figure 16: Simulation results for the three-hump camel (left column) and valley (right column) [PITH_FULL_IMAGE:figures/full_fig_p042_16.png]
Figure 17
Figure 17. Figure 17: AutoGD learning rates for each of the deterministic optimization problems. [PITH_FULL_IMAGE:figures/full_fig_p042_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 44 canonical work pages

  1. [1]

    How free is parameter-free stochastic optimization? arXiv:2402.03126 , 2024

    Amit Attia and Tomer Koren. How free is parameter-free stochastic optimization? arXiv:2402.03126 , 2024

    arXiv 2024

  2. [2]

    Calculus , volume 1

    Tom Apostol. Calculus , volume 1. John Wiley & Sons, 2^ nd edition, 1967

    work page 1967

  3. [3]

    Gradient descent converges linearly for logistic regression on separable data

    Kyriakos Axiotis and Maxim Sviridenko. Gradient descent converges linearly for logistic regression on separable data. In International Conference on Machine Learning , pages 1302--1319. PMLR, 2023

    work page 2023

  4. [4]

    Julia: A fresh approach to numerical computing

    Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM Review , 59(1):65--98, 2017

    work page 2017

  5. [5]

    Making SGD parameter-free

    Yair Carmon and Oliver Hinder. Making SGD parameter-free. In Conference on Learning Theory , pages 2360--2389. PMLR, 2022

    work page 2022

  6. [6]

    Understanding and detecting convergence for stochastic gradient descent with momentum

    Jerry Chee and Ping Li. Understanding and detecting convergence for stochastic gradient descent with momentum. In 2020 IEEE International Conference on Big Data , pages 133--140. IEEE, 2020

    work page 2020

  7. [7]

    Convergence diagnostics for stochastic gradient descent with constant step size

    Jerry Chee and Panos Toulis. Convergence diagnostics for stochastic gradient descent with constant step size. arXiv:1710.06382 , 2017

    work page Pith review arXiv 2017

  8. [8]

    Automatically constructing a corpus of sentential paraphrases

    Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing , 2005

    work page 2005

Show all 56 references
  1. [9]

    Robust, accurate stochastic optimization for variational inference

    Akash Kumar Dhaka, Alejandro Catalina, Michael R Andersen, M ns Magnusson, Jonathan Huggins, and Aki Vehtari. Robust, accurate stochastic optimization for variational inference. Advances in Neural Information Processing Systems , 33:10961--10973, 2020

    2020

  2. [10]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(7), 2011

    2011

  3. [11]

    Learning-rate-free learning by D - A daptation

    Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by D - A daptation. In International Conference on Machine Learning , pages 7449--7479. PMLR, 2023

    2023

  4. [12]

    Markov Chains

    Randal Douc, Eric Moulines, Pierre Priouret, and Philippe Soulier. Markov Chains . Springer, 2018

    2018

  5. [13]

    Probability: Theory and Examples

    Rick Durrett. Probability: Theory and Examples . Cambridge University Press, 4th edition, 2010

    2010

  6. [14]

    The road less scheduled

    Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. Advances in Neural Information Processing Systems , 37:9974--10007, 2024

    2024

  7. [15]

    Bayesian Data Analysis

    Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian Data Analysis . CRC Press, 2013

    2013

  8. [16]

    Handbook of convergence theorems for (stochastic) gradient methods

    Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv:2301.11235 , 2023

    2023 arXiv

  9. [17]

    Inference from iterative simulation using multiple sequences

    Andrew Gelman and Donald B Rubin. Inference from iterative simulation using multiple sequences. Statistical science , 7(4):457--472, 1992

    1992

  10. [18]

    Don't be so monotone: R elaxing stochastic line search in over-parameterized models

    Leonardo Galli, Holger Rauhut, and Mark Schmidt. Don't be so monotone: R elaxing stochastic line search in over-parameterized models. Advances in Neural Information Processing Systems , 36:34752--34764, 2023

    2023

  11. [19]

    Variance-reduced methods for machine learning

    Robert M Gower, Mark Schmidt, Francis Bach, and Peter Richt \'a rik. Variance-reduced methods for machine learning. Proceedings of the IEEE , 108(11):1968--1983, 2020

    1968

  12. [20]

    Srivastava, and K

    Geoffrey Hinton, N. Srivastava, and K. Swersky. Neural networks for machine learning lecture 6a: Overview of mini-batch gradient descent, 2012

    2012

  13. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770--778, 2016

    2016

  14. [22]

    DoG is SGD 's best friend: A parameter-free dynamic step size schedule

    Maor Ivgi, Oliver Hinder, and Yair Carmon. DoG is SGD 's best friend: A parameter-free dynamic step size schedule. In International Conference on Machine Learning , pages 14465--14499. PMLR, 2023

    2023

  15. [23]

    Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification

    Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of Machine Learning Research , 18(223):1--42, 2018

    2018

  16. [24]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014

    2014 arXiv

  17. [25]

    Accelerated parameter-free stochastic optimization

    Itai Kreisler, Maor Ivgi, Oliver Hinder, and Yair Carmon. Accelerated parameter-free stochastic optimization. In The Thirty Seventh Annual Conference on Learning Theory , pages 3257--3324. PMLR, 2024

    2024

  18. [26]

    Tuning-free stochastic optimization

    Ahmed Khaled and Chi Jin. Tuning-free stochastic optimization. arXiv:2402.07793 , 2024

    2024 arXiv

  19. [27]

    Linear convergence of black-box variational inference: S hould we stick the landing? In International Conference on Artificial Intelligence and Statistics , pages 235--243

    Kyurae Kim, Yian Ma, and Jacob Gardner. Linear convergence of black-box variational inference: S hould we stick the landing? In International Conference on Artificial Intelligence and Statistics , pages 235--243. PMLR, 2024

    2024

  20. [28]

    DoWG unleashed: A n efficient universal parameter-free gradient descent method

    Ahmed Khaled, Konstantin Mishchenko, and Chi Jin. DoWG unleashed: A n efficient universal parameter-free gradient descent method. Advances in Neural Information Processing Systems , 36:6748--6769, 2023

    2023

  21. [29]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  22. [30]

    RoBERTa : A robustly optimized BERT pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa : A robustly optimized BERT pretraining approach. arXiv:1907.11692 , 2019

    1907 arXiv

  23. [31]

    Stochastic polyak step-size for SGD : A n adaptive learning rate for fast convergence

    Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for SGD : A n adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics , pages 1306--1314. PMLR, 2021

    2021

  24. [32]

    Using statistics to automate stochastic optimization

    Hunter Lang, Lin Xiao, and Pengchuan Zhang. Using statistics to automate stochastic optimization. Advances in Neural Information Processing Systems , 32, 2019

    2019

  25. [33]

    Prodigy: A n expeditiously adaptive parameter-free learner

    Konstantin Mishchenko and Aaron Defazio. Prodigy: A n expeditiously adaptive parameter-free learner. arXiv:2306.06101 , 2023

    2023 arXiv

  26. [34]

    Adaptive gradient descent without descent

    Yura Malitsky and Konstantin Mishchenko. Adaptive gradient descent without descent. arXiv:1910.09529 , 2019

    1910 arXiv

  27. [35]

    Beating SGD saturation with tail-averaging and minibatching

    Nicole M \"u cke, Gergely Neu, and Lorenzo Rosasco. Beating SGD saturation with tail-averaging and minibatching. Advances in Neural Information Processing Systems , 32, 2019

    2019

  28. [36]

    Let's make block coordinate descent converge faster: F aster greedy rules, message-passing, active-set complexity, and superlinear convergence

    Julie Nutini, Issam Laradji, and Mark Schmidt. Let's make block coordinate descent converge faster: F aster greedy rules, message-passing, active-set complexity, and superlinear convergence. Journal of Machine Learning Research , 23(131):1--74, 2022

    2022

  29. [37]

    Dynamics of SGD with stochastic P olyak stepsizes: T ruly adaptive variants and convergence to exact solution

    Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of SGD with stochastic P olyak stepsizes: T ruly adaptive variants and convergence to exact solution. Advances in Neural Information Processing Systems , 35:26943--26954, 2022

    2022

  30. [38]

    Training deep networks without learning rates through coin betting

    Francesco Orabona and Tatiana Tommasi. Training deep networks without learning rates through coin betting. Advances in Neural Information Processing Systems , 30, 2017

    2017

  31. [39]

    On convergence-diagnostic based step sizes for stochastic gradient descent

    Scott Pesme, Aymeric Dieuleveut, and Nicolas Flammarion. On convergence-diagnostic based step sizes for stochastic gradient descent. In International conference on machine learning , pages 7641--7651. PMLR, 2020

    2020

  32. [40]

    On the determination of the step size in stochastic quasigradient methods

    Georg Pflug. On the determination of the step size in stochastic quasigradient methods. Technical report, International Institute for Applied Systems Analysis, 1983

    1983

  33. [41]

    Non-asymptotic confidence bounds for stochastic approximation algorithms with constant step size

    Georg Pflug. Non-asymptotic confidence bounds for stochastic approximation algorithms with constant step size. Monatshefte f \"u r Mathematik , 110:297--314, 1990

    1990

  34. [42]

    Py T orch: A n imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Py T orch: A n imperative style, high-performance deep learning library. In Advances in Neural Information Processing System...

    2019

  35. [43]

    https://huggingface.co/microsoft/resnet-18, 2025

    ResNet-18 . https://huggingface.co/microsoft/resnet-18, 2025

    2025

  36. [44]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics , pages 400--407, 1951

    1951

  37. [45]

    https://huggingface.co/FacebookAI/roberta-base, 2025

    RoBERTa base model. https://huggingface.co/FacebookAI/roberta-base, 2025

    2025

  38. [46]

    Anytime tail averaging

    Nicolas Le Roux. Anytime tail averaging. arXiv:1902.05083 , 2019

    1902 arXiv

  39. [47]

    Making gradient descent optimal for strongly convex stochastic optimization

    Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. arXiv:1109.5647 , 2011

    2011 arXiv

  40. [48]

    Sticking the landing: S imple, lower-variance gradient estimators for variational inference

    Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: S imple, lower-variance gradient estimators for variational inference. Advances in Neural Information Processing Systems , 30, 2017

    2017

  41. [49]

    SQuAD : 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD : 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383--2392, 2016

    2016

  42. [50]

    Virtual library of simulation experiments: T est functions and datasets, 2013

    Sonja Surjanovic and Derek Bingham. Virtual library of simulation experiments: T est functions and datasets, 2013

    2013

  43. [51]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Proce...

    2013

  44. [52]

    Stochastic gradient descent for non-smooth optimization: C onvergence results and optimal averaging schemes

    Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: C onvergence results and optimal averaging schemes. In International Conference on Machine Learning , pages 71--79. PMLR, 2013

    2013

  45. [53]

    Painless stochastic gradient: I nterpolation, line-search, and convergence rates

    Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: I nterpolation, line-search, and convergence rates. Advances in Neural Information Processing Systems , 32, 2019

    2019

  46. [54]

    A framework for improving the reliability of black-box variational inference

    Manushi Welandawe, Michael Riis Andersen, Aki Vehtari, and Jonathan H Huggins. A framework for improving the reliability of black-box variational inference. Journal of Machine Learning Research , 25(219):1--71, 2024

    2024

  47. [55]

    GLUE : A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations , 2019

    2019

  48. [56]

    Fluctuation-dissipation relations for stochastic gradient descent

    Sho Yaida. Fluctuation-dissipation relations for stochastic gradient descent. arXiv:1810.00004 , 2018

    2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.