Pith. sign in

REVIEW 2 major objections 6 minor 61 references

AlphaGrad: Non-Linear Gradient Normalization Optimizer

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AlphaGrad more than doubles PPO reward over Adam, the paper says

desk verdict A genuinely new but simple optimizer with sound, self-aware theory and unverified empirical claims: single runs, no code, alpha chosen post hoc on the test bed. read the letter →

arxiv 2504.16020 v2 pith:FVFIOUYN submitted 2025-04-22 cs.LG cs.AIcs.NEstat.ML

Soham Sane This is my paper · ORCID
classification cs.LGcs.AIcs.NEstat.ML
keywords AlphaGradgradientnormalizationtanhclippingmemory-efficientoptimizerreinforcementlearningPPOnon-convexconvergenceAdamcomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces AlphaGrad, an optimizer that updates each weight with $\tanh(\alpha \tilde{g})$, where $\tilde{g}$ is the layer's gradient divided by its L2 norm. The author is trying to establish that this one-knob, nearly stateless rule can deliver the scale-invariance of adaptive methods without storing per-parameter moments, and that on reinforcement learning it can be a stronger choice than Adam: substantially larger final reward in on-policy PPO, competitive peak reward with smoother training in TD3, though unstable in DQN. It also proves, under standard smoothness assumptions, convergence to stationary points at an $O(1/\sqrt{T})$ rate. A sympathetic reader would care because, if true, AlphaGrad is a memory-lean alternative to Adam for on-policy training and a concrete demonstration that per-parameter adaptivity is not required for strong RL optimization.

What carries the argument

The load-bearing object is the normalized-gradient-tanh map $g' = \tanh(\alpha \cdot g/(\|g\|_2+\epsilon))$, applied per parameter tensor. The L2 normalization removes layer-scale differences, the tanh keeps every coordinate in $(-1,1)$, and $\alpha$ sets how much of the normalized vector saturates into sign-like behavior. The convergence proof is carried by an alignment inequality, $\langle g_t, g'_t\rangle \ge \tanh(\alpha)\,\|g_t\|_2^2/(\|g_t\|_2+\epsilon)$, proven by concavity of $\phi(x)=\tanh(\alpha x)-x\tanh(\alpha)$ on $[0,1]$; substituting that inequality into the L-smoothness descent inequality makes the error terms telescope into the stated rates. The paper also proposes the dimensionality heuristic $\alpha\approx k\sqrt{d}$ for choosing the steepness per layer, while emphasizing that empirical tuning typically overrides it.

What would settle it

Run AlphaGrad with $\alpha=98$ against Adam on HalfCheetah-v5 with PPO across at least ten independent seeds and compare the full distributions of final returns; if Adam's median falls within the spread of AlphaGrad's runs, the paper's headline empirical claim fails. A complementary check repeats the Hopper-v4 TD3 comparison with $\alpha\in[100,250]$ over multiple seeds to see whether the reported smoothness survives seed variance.

Watch

Extended reading notes

Core claim

The paper's central claim is that the map $g' = \tanh(\alpha\, g/(\|g\|_2+\epsilon))$, applied tensor-wise, is a complete optimizer: it enforces scale invariance, bounds every update component in $(-1,1)$, and interpolates smoothly between normalized gradient descent ($\alpha\to 0$) and sign-based updates ($\alpha\to\infty$). On benchmarks, the paper reports AlphaGrad at $\alpha=98$ reaching rewards above 3000 on HalfCheetah-v5 with PPO, more than double Adam's plateau around 1500, with higher explained variance; on Hopper-v4 with TD3, $\alpha\in[100,250]$ gives Adam-competitive peaks with markedly smoother learning curves; on CartPole-v1 with DQN, AlphaGrad is unstable. The convergence analysis proves stationarity in the smooth non-convex case with the $O(1/\sqrt{T})$ bound stated for the quantity $\|g_t\|_2^2/(\|g_t\|_2+\epsilon)$, plus a convex average-iterate bound whose rate degrades as gradients vanish near the optimum.

Load-bearing premise

The empirical claims assume that the training curves shown, from single runs with alpha chosen on the same benchmark it is then measured on, represent typical optimizer behavior; if the runs are noisy or the alpha selection is post hoc, the reported advantages over Adam do not generalize.

Editorial extensions

If this is right

  • Training with AlphaGrad removes per-parameter first and second moment buffers, so memory per weight drops to zero beyond the weights themselves, with only an optional single momentum scalar per parameter.
  • In on-policy PPO, the paper's result implies a stateless, bounded update can take larger effective trust-region steps than Adam while keeping value-function learning accurate, since reported explained variance is higher.
  • In off-policy DQN, the direct corollary is that stateless reactivity without temporal smoothing amplifies TD-loss spikes; the paper says alpha scheduling or added momentum would be needed.
  • For TD3-style actor-critic training, tuned AlphaGrad implies stability can be separated from peak performance: learning curves can be smooth and monotonic while matching Adam's ceiling.
  • The non-convex theorem means the optimizer is guaranteed to reach stationarity in deterministic smooth problems, though the stationarity measure is the modified quantity rather than the squared gradient norm itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the paper, the bounded $(-1,1)$ update makes AlphaGrad a natural candidate for fixed-point or low-precision training; a direct test would be training a quantized network with AlphaGrad and checking whether the boundedness removes the need for gradient scaling.
  • A testable prediction outside the paper: adding Adam-style momentum or a short EMA of normalized gradients to AlphaGrad should recover DQN stability while keeping most of the PPO gain, isolating statelessness as the cause of the DQN failure.
  • Because the PPO gain is far larger than the formal convergence bound predicts, the mechanism is likely interaction with the clipped surrogate objective rather than stationarity; comparing AlphaGrad with signSGD under PPO would separate tanh shaping from sign-like saturation.
  • The paper's alpha-as-directional-filter examples suggest that a saturation-ratio scheduler, maintaining a target fraction of saturated coordinates, is a plausible alpha-tuning replacement that could remove the per-benchmark grid search.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes AlphaGrad, an optimizer that layer-wise L2-normalizes gradients and then applies a smooth tanh nonlinearity, g' = tanh(alpha * g/(||g||_2 + epsilon)), with alpha acting as a single steepness parameter. The manuscript claims three contributions: (1) the algorithm formulation with a heuristic alpha proportional to sqrt(d_L); (2) a deterministic convergence analysis in convex and non-convex settings; and (3) empirical results on CartPole/DQN, Hopper/TD3, and HalfCheetah/PPO, comparing AlphaGrad against Adam. The reported results are context-dependent: instability in DQN, stability with competitive performance in TD3 when alpha is tuned, and a large PPO advantage for alpha=98. The theoretical sections are self-contained, and the authors explicitly acknowledge several limitations, including the nonstandard convergence measure h(N_t) = N_t^2/(N_t + epsilon) and the possibility that the convex rate becomes vacuous when gamma_min approaches zero.

Significance. If the empirical claims were reproducible, AlphaGrad would be an attractive stateless alternative to Adam for on-policy RL and for memory-constrained training, and the paper's transparent discussion of the theory's limitations is a genuine strength. The proofs of Lemma 2 and Theorems 1-2 are standard and appear correct under the stated assumptions, and the alpha scaling heuristic is clearly motivated. However, the headline empirical claims rest on single-run curves with post hoc alpha selection, and the convergence analysis applies to global gradient normalization rather than the layer-wise normalization used in the implemented algorithm. The empirical contribution therefore needs substantially stronger evidence before the paper's central claims can be accepted.

major comments (2)
  1. [Section 6.3, Figure 3] The central empirical claim that AlphaGrad 'achieves substantially superior performance' over Adam in PPO is based on single-run learning curves with no seeds, no confidence intervals, and alpha values (98, 196, 294) selected by a grid search on the same HalfCheetah-v5 benchmark. Because PPO returns are highly variable, the reported more-than-double reward gap for alpha=98 is not statistically supported and could be due to seed luck or post hoc selection. The same concern applies to the TD3 stability claim in Section 6.2. The paper should report multiple seeds (at least five) with median/interquartile ranges, and should either pre-register the alpha grid or use a separate validation environment for alpha selection before evaluating on the final benchmark.
  2. [Section 3, Eqs. (8)-(10)] The convergence analysis considers the full parameter vector x and normalizes the complete gradient by its global L2 norm, x_{t+1} = x_t - eta * tanh(alpha * grad f(x_t)/||grad f(x_t)||). The algorithm in Section 2, however, normalizes each layer or tensor independently (Eq. (2)) and uses a layer-specific alpha_L. Consequently, Theorems 1 and 2 do not directly establish convergence for the layer-wise AlphaGrad variant actually evaluated in the experiments. A block-wise or tensor-wise analysis is needed, or the theory must be explicitly restricted to the single-tensor case; as written, the claim of a formal convergence guarantee for the proposed optimizer is not fully supported.
minor comments (6)
  1. [Figures 1-3] The figures do not indicate the number of seeds or show any measure of variance; please add per-seed curves or median/interquartile ranges, and state the seed protocol in Section 5.1.
  2. [Section 5.3] The claim that tensor-wise normalization leads to improved convergence on a 'private regression dataset (not shown in this paper)' is unverifiable and should be removed or replaced with a citable, documented experiment.
  3. [Section 2, Eq. (2)] The notation is inconsistent: Eq. (2) uses subscript p for the normalized gradient, while Eqs. (1) and (3) use subscript L for the layer; please harmonize the notation.
  4. [Abstract and Section 7] The abstract's 'extensive empirical evaluation' overstates the scope, as only three environments are tested, and Section 7, titled 'Further Empirical Validation', contains no new experiments; consider renaming Section 7 to 'Future Work'.
  5. [Section 2, Eqs. (5)-(6)] The paper describes AlphaGrad as 'conditionally stateless' but the momentum variant stores one variable per parameter; the memory comparison with Adam should state that momentum-enabled AlphaGrad uses half the per-parameter state of Adam, while the stateless regime is memory-free.
  6. [General] No code or configuration files are provided; given the strong dependence of the results on alpha and the 'reduced learning rate' mentioned in Section 6.2, releasing the exact CleanRL configurations is necessary for reproducibility.

Circularity Check

2 steps flagged · score 5.0 of 10

Empirical superiority claims are post-hoc α selections; the formal convergence analysis is self-contained and not circular.

  1. fitted input called prediction [Section 6.3, HalfCheetah-v5 (PPO)]
    "Here, we utilized a larger range of alpha values due to limited compute resources (α∈{98,196,294}) against Adam. The results, shown in Figure 3, were striking and demonstrated a clear advantage for AlphaGrad. ... AlphaGrad decisively outperformed Adam in this PPO setting. All AlphaGrad variants achieved substantially higher asymptotic rewards, with α=98 peaking above 3000—more than double Adam's plateau around 1500."

    The 'more than double Adam's reward' number is the best of the three α values tested on HalfCheetah-v5, and α=98 was not chosen by a prior rule but from observing these same training curves. The paper concedes in Section 2.1 that 'optimal performance frequently requires extensive empirical validation and tuning tailored to the specific task,' so the reported PPO advantage is a post-hoc selection outcome, i.e., a fitted result, not an independent prediction of AlphaGrad's superiority. The theory is unaffected, but the headline empirical claim reduces to 'for the α grid point that worked best on this benchmark, AlphaGrad beats Adam.'

  2. fitted input called prediction [Section 6.2, Hopper-v4 (TD3)]
    "Recognizing the inadequacy of the theoretical scaling law (Section 2.1), we performed an empirical search for effective α values, testing α∈{100,150,200,250,457} against Adam... The most significant finding was the superior stability exhibited by these well-tuned AlphaGrad variants."

    The stability conclusion is drawn from the subset α∈[100,250] that was selected by searching on the same Hopper-v4 environment; α=457 (the value closest to the √d guideline) was discarded because it 'failed to learn effectively.' Reporting the survivors of an in-sample grid search as AlphaGrad's 'superior stability' is fitting then re-describing the fit; it cannot serve as independent evidence. As with PPO, this does not bear on the deterministic convergence theorems.

full rationale

The mathematical content of the paper (Section 3) is self-contained: Lemma 1 bounds ||g'|| by √n, Lemma 2 derives ⟨g,g'⟩≥γN from tanh(αx)≥x tanh(α) on [0,1], and Theorems 1–2 telescope the L-smoothness inequality to stationarity. No external fitted quantity enters, and no load-bearing self-citation exists. The circularity, if any, is confined to the empirical contribution. In Sections 6.2 and 6.3 the paper performs a grid search over α on the same environments whose outcomes are then reported as AlphaGrad's superior stability (TD3/Hopper) and substantially superior performance (PPO/HalfCheetah). The abstract itself concedes the profile is 'highly context-dependent' and requires 'careful α tuning.' Thus the headline empirical findings are in-sample selections over the defining hyperparameter, presented as optimizer-level conclusions. They are not mathematically forced by the update rule, but they are fitted outcomes rather than out-of-sample predictions. Because the central theoretical claim is independent and the empirical overfitting is openly acknowledged, the overall circularity score is moderate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central algorithm is simple and does not introduce new physical or mathematical entities. The load-bearing free parameters are alpha and the per-task learning rate; both are fitted empirically. The theoretical results require standard smoothness assumptions plus a user-chosen alpha. The alpha scaling law is an extra heuristic with its own free constant k.

free parameters (3)
  • alpha (steepness parameter) = CartPole: {93,186,279}; Hopper: {100,150,200,250,457}, best [100,250]; HalfCheetah: {98,196,294}, best 98.
    The single hyperparameter that controls the tanh transform. The theoretical guideline (alpha = k sqrt(d_L)) is acknowledged as unreliable, and optimal values are chosen by grid search on each benchmark, so the empirical claims depend on this fitted choice.
  • learning rate eta = Not specified numerically; 'reduced learning rate' for TD3.
    Learning rate is tuned per experiment and interacts with alpha; without reporting its value, the comparison to Adam is incomplete and the stability claims are not reproducible.
  • k in alpha = k sqrt(d_L) = Heuristic range [1,3], or derived from target saturation fraction.
    The scaling guideline in Eq. 7 contains an arbitrary constant k, so the guideline is not a parameter-free prediction.
assumptions (4)
  • standard math f is L-smooth and bounded below (Assumptions 1-3).
    Used in Section 3 for the convex and non-convex convergence theorems.
  • domain assumption Normalized gradient components in a layer are approximately Gaussian N(0,1/d_L).
    Section 2.1 uses this to derive alpha ~ sqrt(d_L); the paper later states real gradients deviate from this model.
  • standard math tanh is concave on [0,1] and satisfies the alignment inequality in Lemma 2.
    Used to lower-bound the inner product between raw and transformed gradients.
  • domain assumption CleanRL reference implementations provide fair, correctly configured baselines for Adam and the RL algorithms.
    The comparison in Section 6 assumes the Adam curves are strong baselines; no hyperparameter tuning for Adam is reported beyond defaults.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AlphaGrad: Non-Linear Gradient Normalization Optimizer." pith.science (2026). https://pith.science/paper/FVFIOUYN

@misc{pith2026250416020,
  author       = {Pith},
  title        = {Pith review of: AlphaGrad: Non-Linear Gradient Normalization Optimizer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVFIOUYN}},
  note         = {Machine review of arXiv:2504.16020}
}
abstract

We introduce AlphaGrad, a memory-efficient, conditionally stateless optimizer addressing the memory overhead and hyperparameter complexity of adaptive methods like Adam. AlphaGrad enforces scale invariance via tensor-wise L2 gradient normalization followed by a smooth hyperbolic tangent transformation, $g' = \tanh(\alpha \cdot \tilde{g})$, controlled by a single steepness parameter $\alpha$. Our contributions include: (1) the AlphaGrad algorithm formulation; (2) a formal non-convex convergence analysis guaranteeing stationarity; (3) extensive empirical evaluation on diverse RL benchmarks (DQN, TD3, PPO). Compared to Adam, AlphaGrad demonstrates a highly context-dependent performance profile. While exhibiting instability in off-policy DQN, it provides enhanced training stability with competitive results in TD3 (requiring careful $\alpha$ tuning) and achieves substantially superior performance in on-policy PPO. These results underscore the critical importance of empirical $\alpha$ selection, revealing strong interactions between the optimizer's dynamics and the underlying RL algorithm. AlphaGrad presents a compelling alternative optimizer for memory-constrained scenarios and shows significant promise for on-policy learning regimes where its stability and efficiency advantages can be particularly impactful.

Figures

Figures reproduced from arXiv: 2504.16020 by the authors.

Figure 1
Figure 1. Performance on CartPole-v1 (DQN): Episodic Return, Length, TD Loss, and Q-Values. AlphaGrad variants ( [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Performance on Hopper-v4 (TD3): Training Rewards, Episode Length, Q-Values, and Actor Loss. AlphaGrad variants (tuned [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Performance on HalfCheetah-v5 (PPO): Training Rewards, Explained Variance, Entropy Loss, and Approx KL Divergence. [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 41 canonical work pages

  1. [1]

    He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385

    arXiv 2015

  2. [2]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805

    arXiv 2018

  3. [3]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning.Nature, 518(7540):529–533

    work page 2015

  4. [4]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980

    arXiv 2014

  5. [5]

    Ruder, S. (2016). An overview of gradient descent optimization algorithms.arXiv preprint arXiv:1609.04747

    arXiv 2016

  6. [6]

    C., Roelofs, R., Stern, M., Srebro, N., and Recht, B

    Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. (2017). The marginal value of adaptive gradient methods in machine learning.arXiv preprint arXiv:1705.08292

    arXiv 2017

  7. [7]

    Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). On the importance of initializa- tion and momentum in deep learning. InProceedings of the 30th International Conference on Machine Learning (ICML), volume 28(3), pages 1139–1147. PMLR

    work page 2013

  8. [8]

    You, Y., Gitman, I., and Ginsburg, B. (2017). Large batch training of convolutional net- works.arXiv preprint arXiv:1708.03888

    arXiv 2017

Show all 61 references
  1. [9]

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Hsieh, C.-J., and Zhao, Y. (2019). Large batch optimization for deep learning: Training BERT in 76 minutes.arXiv preprint arXiv:1904.00962

    2019 arXiv

  2. [10]

    Bernstein, J., Wang, J., Azizzadenesheli, K., and Anandkumar, A. (2018). signSGD: Com- pressed optimisation for non-convex problems.arXiv preprint arXiv:1802.04434

    2018 arXiv

  3. [11]

    Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods.arXiv preprint arXiv:1802.09477

    2018 arXiv

  4. [12]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347

    2017 arXiv

  5. [13]

    Ball, K. (1997). An elementary introduction to modern convex geometry. In Levy, S., editor, Flavors of Geometry, pages 1–58. Cambridge University Press

    1997

  6. [14]

    (2018).High-Dimensional Probability: An Introduction with Applications in Data Science

    Vershynin, R. (2018).High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press

    2018

  7. [15]

    (2016).Deep Learning

    Goodfellow, I., Bengio, Y., and Courville, A. (2016).Deep Learning. MIT Press

    2016

  8. [16]

    (2004).Introductory Lectures on Convex Optimization: A Basic Course

    Nesterov, Y. (2004).Introductory Lectures on Convex Optimization: A Basic Course. Ap- plied Optimization, volume 87. Kluwer Academic Publishers

    2004

  9. [17]

    Bubeck, S. (2015). Convex optimization: Algorithms and complexity.Foundations and Trends®in Machine Learning, 8(3-4):231–357. AlphaGrad

    2015

  10. [18]

    Polyak, B. T. (1963). Gradient methods for minimizing functionals.Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki, 3(4):643–653

    1963

  11. [19]

    Karimi, H., Nutini, J., and Schmidt, M. (2016). Linear convergence of gradient and proximal- gradient methods under the Polyak- Lojasiewicz condition. InMachine Learning and Knowl- edge Discovery in Databases (ECML PKDD), volume 9851 of Lecture Notes in Computer Science, pages...

    2016

  12. [20]

    Nesterov, Y. (2013). Gradient methods for minimizing composite functions.Mathematical Programming, 140(1):125–161

    2013

  13. [21]

    and Lan, G

    Ghadimi, S. and Lan, G. (2013). Stochastic first- and zeroth-order methods for nonconvex stochastic programming.SIAM Journal on Optimization, 23(4):2341–2368

    2013

  14. [22]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of Adam and beyond. arXiv preprint arXiv:1904.09237

    2019 arXiv

  15. [23]

    E., and Nocedal, J

    Bottou, L., Curtis, F. E., and Nocedal, J. (2018). Optimization methods for large-scale machine learning.SIAM Review, 60(2):223–311

    2018

  16. [24]

    and Keshet, J

    Levy, K. and Keshet, J. (2018). Online normalization for training neural networks. InAd- vances in Neural Information Processing Systems 31 (NeurIPS), pages 4776–4786

    2018

  17. [25]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. (2021). Deep reinforcement learning at the edge of the statistical precipice. InAdvances in Neural Information Processing Systems 34 (NeurIPS), pages 29304–29320

    2021

  18. [26]

    and Hinton, G

    Tieleman, T. and Hinton, G. (2012). Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning. Available at:http://www.cs.toronto.edu/ ~tijmen/csc321/slides/lecture_ slides_lec6.pdf

    2012

  19. [27]

    Pascanu, R., Mikolov, T., and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. InProceedings of the 30th International Conference on Machine Learning (ICML), volume 28(3), pages 1310–1318. PMLR

    2013

  20. [28]

    Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. (2001). Gradient flow in recur- rent nets: the difficulty of learning long-term dependencies. InA Field Guide to Dynamical Recurrent Networks, pages 237–243. IEEE Press

    2001

  21. [29]

    Huang, S., Dossa, R. F. J., Ye, C., and Braga, J. (2022). CleanRL: High-quality single- file implementations of deep reinforcement learning algorithms.Journal of Open Source Software, 7(70):3759

    2022

  22. [30]

    Todorov, E., Erez, T., and Tassa, Y. (2012). MuJoCo: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026–5033. IEEE

    2012

  23. [31]

    Towers, M., Kunz, J., Jim´ enez, F., Andreas, K., Soma, A., Tangri, R., Batra, A., Agarwal, K., Jiang, M., Alami, R., et al. (2023). Gymnasium.Journal of Open Source Software, 8(85):5272

    2023

  24. [32]

    Sutton, R. S. and Barto, A. G. (2018).Reinforcement Learning: An Introduction. The MIT Press, Second edition. Available at:http://incompleteideas.net/book/the-book-2nd. html. AlphaGrad

    2018

  25. [33]

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. InAdvances in Neural Information Processing Systems 32, pages 8024–8035...

    2019

  26. [34]

    S., Davis, A., Dean, J., Devin, M., et al

    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016). TensorFlow: Large-scale machine learning on heterogeneous distributed systems. In12th USENIX Symposium on Operating Systems Design and Implementa...

    2016

  27. [35]

    A., and Peters, J

    Kober, J., Bagnell, J. A., and Peters, J. (2013). Reinforcement learning in robotics: A survey.The International Journal of Robotics Research, 32(11):1238–1274

    2013

  28. [36]

    I., Kemker, R., Part, J

    Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019). Continual lifelong learning with neural networks: A review.Neural Networks, 113:54–71

    2019

  29. [37]

    S., and Pennington, J

    Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S. S., and Pennington, J. (2018). Dynam- ical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks.arXiv preprint arXiv:1806.05393

    2018 arXiv

  30. [38]

    Gupta, V., Koren, T., and Singer, Y. (2018). Shampoo: Preconditioned stochastic tensor optimization. InAdvances in Neural Information Processing Systems 31 (NeurIPS), pages 9523–9533

    2018

  31. [39]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. InInternational Conference on Learning Representations (ICLR)

    2017

  32. [40]

    J., Fu, P., Saparov, A., Choudhury, S., Huang, W., Hurtado, J., Khailany, B., Lien, H.-H

    Banbury, C., Reddi, V. J., Fu, P., Saparov, A., Choudhury, S., Huang, W., Hurtado, J., Khailany, B., Lien, H.-H. S., Melik-Adamyan, D., et al. (2021). Benchmarking TinyML systems: Challenges and direction. In2021 IEEE International Symposium on Performance Analysis of Systems ...

    2021

  33. [41]

    Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. (2015). Deep learning with limited numerical precision.arXiv preprint arXiv:1502.02551

    2015 arXiv

  34. [42]

    Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer- arithmetic-only inference. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition...

    2018

  35. [43]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models.arXiv preprint arXiv:2001.08361

    2020 arXiv

  36. [44]

    Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. (2018). Visualizing the loss land- scape of neural nets. InAdvances in Neural Information Processing Systems 31 (NeurIPS), pages 6391–6401

    2018

  37. [45]

    LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). The MNIST database of hand- written digits. Available at:http://yann.lecun.com/exdb/mnist/

    1998

  38. [46]

    Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto. AlphaGrad

    2009

  39. [47]

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee

    2009

  40. [48]

    Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019). Do ImageNet classifiers gener- alize to ImageNet?arXiv preprint arXiv:1902.10811

    2019 arXiv

  41. [49]

    Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for SQuAD.arXiv preprint arXiv:1806.03822

    2018 arXiv

  42. [50]

    J., Li, J., Paduraru, C., Gowal, S., and Hester, T

    Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. (2019). An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901

    2019 arXiv

  43. [51]

    Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2018). Deep reinforcement learning that matters. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32

    2018

  44. [52]

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man´ e, D. (2016). Concrete problems in AI safety.arXiv preprint arXiv:1606.06565

    2016 arXiv

  45. [53]

    J., and R´ e, C

    Zhang, J., Liu, H., Wright, S. J., and R´ e, C. (2022). On the overlooked structure of stochas- tic gradients. InAdvances in Neural Information Processing Systems 35 (NeurIPS), pages 35262–35276

    2022

  46. [54]

    U., Cordonnier, J.-B., and Jaggi, M

    Stich, S. U., Cordonnier, J.-B., and Jaggi, M. (2019). Convergence of SGD with biased gradients from gradient sparsification. InProceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 5976–5985. PMLR

    2019

  47. [55]

    and Stern, M

    Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. InProceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR

    2018

  48. [56]

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Gong, L., Ma, X., et al. (2023). Symbolic discovery of optimization algorithms.arXiv preprint arXiv:2302.03669

    2023 arXiv

  49. [57]

    H., Ma, T., Anandkumar, A., and R´ e, C

    Liu, H., Li, Z., Hsieh, D., Li, E. H., Ma, T., Anandkumar, A., and R´ e, C. (2023). Sophia: A scalable stochastic second-order optimizer.arXiv preprint arXiv:2305.14342

    2023 arXiv

  50. [58]

    Biedenkapp, A., Rajan, R., Hutter, F., and Lindauer, M. (2021). Hyperparameter optimiza- tion for model-based reinforcement learning. InProceedings of the 35th AAAI Conference on Artificial Intelligence, Workshop on Reinforcement Learning in Games

    2021

  51. [59]

    Dalal, G., Mania, H., Nachum, O., Scherrer, B., and Tamar, A. (2023). On the sample com- plexity of actor-critic for reinforcement learning with function approximation. InAdvances in Neural Information Processing Systems 36 (NeurIPS)

    2023

  52. [60]

    and Bengio, Y

    Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR Workshop and Conference Pro- ceedings, pa...

    2010

  53. [61]

    He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE Interna- tional Conference on Computer Vision (ICCV), pages 1026–1034

    2015

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.