REVIEW 4 major objections 4 minor 26 references
Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper derives per-step optimal learning rate and momentum from mini-batch statistics and reports parity with exhaustive search.
desk verdict AutoOpt is a plausible extension of Schaul et al. to momentum, but the Section 4 derivation collapses a 2-vector into a scalar under the very assumption the experiments violate, and the CIFAR10 Adam gap is real. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is the reparameterization of the gradient estimator as $\hat g_t = g_t - G_t\gamma$, with $G_t = [g_t,\ g_t-\hat g_{t-1}]$ and $\gamma = [\gamma_1,\gamma_2]^T$ linked to the learning rate and momentum by $\alpha = 1-\gamma_1$ and $\beta = \gamma_2/(1-\gamma_1)$. Plugging this into a quadratic Taylor model of the loss produces an expected next-step loss whose minimizer is $\gamma_t^O = A_t^{-1}b_t$, where $A_t$ is the expected Hessian-weighted Gram matrix of $G_t$ and $b_t$ is a vector of covariance and bias terms. The simplification $b_t = \tfrac12 V(g_t|\hat H_t)$—a scalar variance term—holds only when the optimizer's Hessian estimator equals the true Hessian; the practical estimator uses the unbiased per-example variance $\hat V(g_t|\hat H_t)$ and the per-step sample matrix $\hat A_t = G_t^T\hat H_t^{-1}G_t$, followed by exponential smoothing. These components convert an oracle quantity into a per-step, per-layer schedule.
What would settle it
Re-run the CIFAR10 experiment with AutoAdam using the paper's architecture: if the test error again lands near the reported 47.8% while the exhaustive-search optimum stays near 36.4%, the claim that AutoOpt matches the best manual settings fails for Adam.
Extended reading notes
Core claim
The paper's central claim is that the optimal learning rate and momentum at iteration $t$ can be written as $\gamma_t^O = A_t^{-1} b_t$, where the $2\times 2$ matrix $A_t$ and vector $b_t$ come from the Hessian-weighted covariance of the gradient estimator and its history. Because the true gradient is unknown, this is called the oracle solution; the paper's practical contribution is an estimator $\hat\gamma_t^O$ that replaces expectations with the current mini-batch, estimates the variance term $V(g_t|\hat H_t)$ from per-example gradients, and smooths the result with an exponential moving average. The claim is that plugging this estimated optimal vector into the update rule for SGD, Adam, or AdaGrad yields the same or comparable train and test error as the best configuration found by exhaustive search, in one run instead of hundreds. The paper reports this for MNIST and CIFAR10 CNN experiments, with the notable exception that AutoAdam on CIFAR10 reaches 47.8% test error versus 36.4% for manually tuned Adam, a gap the paper attributes to the difference between the unknown Hessian and Adam's diagonal estimator.
Load-bearing premise
The method assumes that the optimizer's approximation of the loss curvature is exact; when that approximation is wrong, the automatically chosen learning rate and momentum are no longer the best choices.
Editorial extensions
If this is right
- With AutoOpt, the learning rate and momentum are set anew each step and each layer; the paper observes larger mini-batches producing larger learning rates, followed by automatic decay, while momentum rises as the step size shrinks.
- Using AutoOpt removes the exhaustive search: for SGD the 324 settings collapse to 1, and for Adam and AdaGrad the 54 settings collapse to 1, with comparable errors.
- Because the method's per-step cost is the same order as backpropagation for diagonal Hessian estimators, it is practical for CNN training.
- The method is generic across optimizers that fit the update rule (9), so it can be deployed on other SGD variants beyond SGD, Adam, and AdaGrad.
- In the paper's runs, AutoSGD, AutoAdam, and AutoAdaGrad reach comparable performance to each other after automatic tuning, making it easy to switch optimizers without manual re-tuning.
Reading between the lines
- The failure pattern on CIFAR10 AutoAdam suggests that the biggest gains from this approach will come from optimizers whose Hessian estimates track true curvature; a testable extension is to replace Adam's diagonal curvature estimate with a low-rank or block-diagonal correction and see whether the 10-point gap closes.
- Because AutoOpt produces per-layer schedules, it implies that a single global learning rate is a real bottleneck; one could compare layer-wise AutoOpt schedules against the best global schedule and quantify the gain.
- The method's estimator needs per-example gradients, so in models where storing those is prohibitive, an approximate variance estimator based on gradient chunks or running statistics would be needed; this follows from the method's mechanics, not from the paper.
- The online and non-stationary extension the authors mention in their conclusion is a natural stress test: if the variance estimator reacts quickly enough, AutoOpt should re-adapt learning rate and momentum when the data distribution changes mid-training.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AutoOpt, a method to automatically and simultaneously set the learning rate and momentum of stochastic gradient optimizers (SGD, Adam, AdaGrad) at each step and per layer. The authors derive an 'oracle' optimal parameter vector by minimizing the expected quadratic approximation of the loss after a step, then estimate that oracle from current mini-batch statistics. They report experiments on MNIST and CIFAR10 CNN classifiers, comparing AutoOpt against an exhaustive manual search, and claim that the automatic settings match or beat the best manually tuned configurations. The central theoretical step is in Section 4, where the 2x1 vector b_t is simplified to a scalar variance term under the assumption that the optimizer's Hessian estimator equals the true Hessian.
Significance. If the central derivation were sound, automatic per-layer adjustment of learning rate and momentum would be a practically valuable contribution, and the experimental comparison against thousands of manual configurations with multiple seeds is a genuine strength. The idea of using the second-order statistics of the unbiased gradient estimator is appealing, and the per-layer adaptation shown in Figures 2-4 is informative. However, the derivation in Section 4 contains a load-bearing dimensional inconsistency, and the paper's own Table 2 contradicts the headline claim for one of the three optimizers on CIFAR10. As a result, the method's theoretical basis and its stated empirical scope are both unsupported in the present form.
major comments (4)
- [§4, Eqs. (18)–(25)] Equation (18) defines b_t as a 2x1 vector, but after the assumption \hat H_t = \bar H_t the paper replaces it by the scalar \hat b_t = (1/2)\hat V(g_t|\hat H_t) in Eq. (23). Substituting \hat H_t = \bar H_t into Eq. (18) gives b_t = E{G_t^T \hat H_t^{-1}(g_t - \bar g_t)}, whose two entries are V(g_t|\hat H_t) and V(g_t|\hat H_t) - E{\hat g_{t-1}^T \hat H_t^{-1}(g_t - \bar g_t)}; it is not a scalar. Consequently Eq. (25), \hat \gamma_{Ot} = \hat A_t^{-1} \hat b_t, is dimensionally undefined as written: \hat A_t^{-1} is 2x2 and \hat b_t is a scalar. This is not a presentational slip but the core step that connects the oracle to the practical estimator.
- [§4, Eq. (20)] The simplification b_t = (1/2)V(g_t|\hat H_t) is not derived from Eqs. (16)–(18). A direct expansion of the second component of b_t under \hat H_t = \bar H_t gives E{g_t^T \hat H_t^{-1}(g_t - \bar g_t)} - E{\hat g_{t-1}^T \hat H_t^{-1}(g_t - \bar g_t)}, which equals V(g_t|\hat H_t) only if E{\hat g_{t-1}^T \hat H_t^{-1}(g_t - \bar g_t)} = 0; this independence condition is not stated or proved. The factor 1/2 also does not follow from the expansion in Eq. (16): the linear term in γ from Eq. (16) has coefficient -b_t, and minimizing 1/2 γ^T A_t γ - γ^T b_t gives γ = A_t^{-1} b_t, not A_t^{-1}(b_t/2). The factor 1/2 therefore appears without justification.
- [§5, Table 2 and Abstract/Conclusions] The central claim that 'the results match the performance of the best settings obtained through an exhaustive search' is contradicted by the paper's own CIFAR10 results for AutoAdam: test error 47.8% versus 36.4% for manually tuned Adam, and train error 42.6% versus 27.2%. The text attributes this gap to the discrepancy between Adam's diagonal Hessian estimator and the true Hessian, but that discrepancy is exactly the assumption used in Section 4 to derive the oracle. Since the empirical failure occurs precisely where the derivation's key assumption fails, the method's stated scope ('in all cases the proposed method automatically attains the lowest, or comparable test and train errors') is unsupported.
- [§4, Eq. (26) and §5] The proposed method is described as automatic, but the exponentially weighted moving average in Eq. (26) introduces a smoothing factor υ that is never specified or tuned in the experiments. No value for υ is given in Algorithm 1 or in Section 5, and no sensitivity analysis is reported. Without a stated value or default, the method still depends on at least one hyperparameter, and the experiments are not fully reproducible as described.
minor comments (4)
- [Abstract and §1] The sentence 'The results match the performance of the best settings obtained through an exhaustive search and therefore, removes the need for a tedious manual tuning' has a subject-verb agreement error ('results ... removes') that should be corrected.
- [§3, Eq. (16) and §4, Eq. (24)] The estimator \hat A_t in Eq. (24) replaces the expectation in Eq. (17) by a single sample product G_t^T \hat H_t^{-1} G_t and is then inverted at every step. The paper does not discuss what happens when the two columns of G_t are nearly collinear or when \hat A_t is ill-conditioned; a regularization or averaging mechanism would make the algorithm more robust.
- [§5, Table 2 caption] The caption states that the automatic counterparts 'achieve comparable, or better results' than manual tuning, but the CIFAR10 AutoAdam row in the same table shows a more than 10% degradation. The caption should be qualified to reflect the actual entries in the table.
- [Appendix B] The complexity claim that Algorithm 1 has the same time complexity as back-propagation relies on the per-observation gradients being available at every layer. In standard deep-learning frameworks these per-sample gradients are not computed during a normal backward pass, so the practical overhead may be larger than stated unless the implementation accumulates them explicitly.
Circularity Check
No significant circularity: AutoOpt's oracle is an internal optimization objective, and the paper validates against an external exhaustive search.
full rationale
The paper derives the optimal learning rate and momentum by minimizing the expected loss of the update rule under a quadratic approximation (Eq. 16-19). The resulting oracle gamma_O = A^{-1}b is an internal optimization target, not a prediction derived from the target quantity being predicted. The estimator in Section 4 plugs sample statistics into A and b, which is standard plug-in estimation rather than a self-definitional reduction. Validation is external: Table 2 and Figures 3-4 compare AutoOpt against an exhaustive grid search over 324 settings for SGD and 54 each for Adam and AdaGrad, so the central claim is not forced by construction. There are no load-bearing self-citations; the method builds on standard references (e.g., Schaul et al., Tan et al.) but does not rely on an unverified self-citation chain. The assumption that the optimizer's Hessian estimator equals the true Hessian is mathematically fragile and empirically contradicted by AutoAdam on CIFAR10 (47.8% vs 36.4% manual), but an assumption failure is a correctness gap, not circular dependence. Overall the derivation is self-contained and the empirical comparison is external, so no circularity is found.
Assumptions & free parameters
free parameters (1)
- smoothing factor upsilon
assumptions (4)
- domain assumption The true loss J(w) is continuously differentiable and can be approximated by a second-order Taylor expansion (Eq. 5).
- ad hoc to paper The estimated Hessian equals the true Hessian (Section 4, first paragraph).
- domain assumption The mini-batch observations are i.i.d. and the individual per-sample gradients are independent (used in Appendix A).
- ad hoc to paper The expectation over the product of the previous gradient estimate and the current gradient deviation is zero, which makes the second component of b_t zero (implied by the b_t = 1/2 V simplification).
Cite this review
Pith. "Pith review of Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent." pith.science (2026). https://pith.science/paper/Z7RW2PBT
@misc{pith2026190807607,
author = {Pith},
title = {Pith review of: Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z7RW2PBT}},
note = {Machine review of arXiv:1908.07607}
}
read the original abstract
Stochastic Gradient Descent (SGD) methods are prominent for training machine learning and deep learning models. The performance of these techniques depends on their hyperparameter tuning over time and varies for different models and problems. Manual adjustment of hyperparameters is very costly and time-consuming, and even if done correctly, it lacks theoretical justification which inevitably leads to "rule of thumb" settings. In this paper, we propose a generic approach that utilizes the statistics of an unbiased gradient estimator to automatically and simultaneously adjust two paramount hyperparameters: the learning rate and momentum. We deploy the proposed general technique for various SGD methods to train Convolutional Neural Networks (CNN's). The results match the performance of the best settings obtained through an exhaustive search and therefore, removes the need for a tedious manual tuning.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Introduction to stochastic search and optimization: estimation, simulation, and control, volume 65
James C Spall. Introduction to stochastic search and optimization: estimation, simulation, and control, volume 65. John Wiley & Sons, 2005
2005
-
[2]
Variance reduction for stochastic gradient optimization
Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors,Advances in Neural Information Processing Systems 26, pages 181–189. Curran Associates, Inc., 2013
work page 2013
-
[3]
Accelerating stochastic gradient descent using predictive variance reduction
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pages 315–323, 2013
work page 2013
-
[4]
On variance reduction in stochastic gradient descent and its asynchronous variants
Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alexander J Smola. On variance reduction in stochastic gradient descent and its asynchronous variants. In Advances in Neural Information Processing Systems, pages 2647–2655, 2015
work page 2015
-
[5]
Stochastic variance reduced multiplicative update for nonnegative matrix fac- torization
Hiroyuki Kasai. Stochastic variance reduced multiplicative update for nonnegative matrix fac- torization. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6338–6342. IEEE, 2018
work page 2018
-
[6]
Convergence of variance-reduced learning under random reshuffling
Bicheng Ying, Kun Yuan, and Ali H Sayed. Convergence of variance-reduced learning under random reshuffling. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2286–2290. IEEE, 2018
work page 2018
-
[7]
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147, 2013
2013
-
[8]
Reddi, Satyen Kale, and Sanjiv Kumar
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018
2018
Show all 26 references
-
[9]
Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011
2011
-
[10]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[11]
Adadelta: an adaptive learning rate method
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012
2012 arXiv
-
[12]
Incorporating nesterov momentum into adam
Timothy Dozat. Incorporating nesterov momentum into adam. In International Conference on Learning Representations, Workshop Track, 2016
2016
-
[13]
No more pesky learning rates
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Conference on Machine Learning, pages 343–351, 2013
2013
-
[14]
Barzilai-borwein step size for stochastic gradient descent
Conghui Tan, Shiqian Ma, Yu-Hong Dai, and Yuqiu Qian. Barzilai-borwein step size for stochastic gradient descent. In Advances in Neural Information Processing Systems , pages 685–693, 2016
2016
-
[15]
Two-point step size gradient methods
Jonathan Barzilai and Jonathan M Borwein. Two-point step size gradient methods. IMA journal of numerical analysis, 8(1):141–148, 1988
1988
-
[16]
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25 , pages 1097–1105. Curran Asso...
2012
-
[17]
C. M. Bishop. Pattern recognition and machine learning, volume 4. Springer, New York, 2006
2006
-
[18]
Sgd-qn: Careful quasi-newton stochastic gradient descent
Antoine Bordes, Léon Bottou, and Patrick Gallinari. Sgd-qn: Careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 10(Jul):1737–1754, 2009
2009
-
[19]
Res: Regularized stochastic bfgs algorithm
Aryan Mokhtari and Alejandro Ribeiro. Res: Regularized stochastic bfgs algorithm. IEEE Transactions on Signal Processing, 62(23):6089–6104, 2014
2014
-
[20]
Limited-memory bfgs optimization of recurrent neural network language models for speech recognition
Xunying Liu, Shansong Liu, Jinze Sha, Jianwei Yu, Zhiyuan Xu, Xie Chen, and Helen Meng. Limited-memory bfgs optimization of recurrent neural network language models for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP...
2018
-
[21]
Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533, 1986
1986
-
[22]
The MNIST database of handwritten digits
Yann Lecun and Corinna Cortes. The MNIST database of handwritten digits. 2009
2009
-
[23]
Learning multiple layers of features from tiny images
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009
2009
-
[24]
MIT press Cambridge, 2016
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.Deep learning, volume 1. MIT press Cambridge, 2016
2016
-
[25]
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pages 807–814, 2010. Appendix A: Derivation of ˆV ( gt| ˆHt ) (22) The expressionV ( gt| ˆHt ) (21...
2010
-
[26]
can be utilized in fully-connected neural networks (FCNN’s) andconvolution neural networks (CNN’s). Practically, when the Hessian estimator is a diagonal matrix, as in SGD, Adam, and AdaGrad (see Table 1) , the time complexity of the proposed algorithm remains the same as for ...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.