REVIEW 5 major objections 5 minor 1 cited by
EXAdam: The Power of Adaptive Cross-Moments
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read EXAdam modifies Adam with cross-moment debiasing and a gradient-acceleration term, reporting 38.46% faster convergence and 90.83% CIFAR-10 test accuracy, beating Adam's 89.66%.
desk verdict Genuinely new Adam variant with a clearly specified update rule, but the empirical case is internally inconsistent—impossible loss values, miscomputed margins, and a language-model run that contradicts the generalization claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the cross-moment debiasing pair $\tilde m$ and $\tilde v$ together with the gradient accelerator $\tilde g$ in Algorithm 1. The pair couples the first and second moment estimates: $\tilde m$ scales the standard first-moment debias by $1 + \frac{v}{v+\epsilon}\beta_2^t$, and $\tilde v$ scales the second-moment debias by $1 + \frac{m^2}{m^2+\epsilon}\beta_1^t$. Because $\beta_1^t$ and $\beta_2^t$ decay, these factors are largest in the first iterations, so the new terms mainly change early training and then fade toward Adam's corrections. The same variance ratio appears in $\tilde g$, which adds the current gradient directly to the momentum term in the update, so steep regions with large $v$ get more weight from the instantaneous gradient while flat regions rely more on momentum.
What would settle it
Re-run the CIFAR-10 CNN experiment across a grid of learning rates from $10^{-3}$ to $10^{-5}$ with several seeds; if Adam, AdamW, or RMSProp reaches or exceeds EXAdam's 90.83% test accuracy or matches its convergence speed under its own best configuration, the claimed general superiority fails.
Extended reading notes
Core claim
The paper claims that Adam's independent treatment of the first and second moment estimates leaves bias correction poorly matched to the local loss geometry, and that injecting the cross-moment factors $\left(1 + \frac{v}{v+\epsilon}\beta_2^t\right)$ and $\left(1 + \frac{m^2}{m^2+\epsilon}\beta_1^t\right)$ into the debiased estimates, together with a gradient accelerator $\tilde g = \frac{g}{1-\beta_1^t}\left(1 + \frac{v}{v+\epsilon}\beta_2^t\right)$, yields an update $\theta \leftarrow \theta - \alpha \frac{\tilde m + \tilde g}{\sqrt{\tilde v} + \epsilon}$ that is more responsive during early training and more stable later. As $t \to \infty$, the correction factors tend to $1$, so EXAdam asymptotically reduces to Adam. On CIFAR-10 with a CNN, the paper reports 38.46% faster convergence than Adam, gains of 1.96, 2.17, and 1.17 percentage points in training, validation, and test accuracy, and a test accuracy of 90.83% versus 89.66% for Adam; in a MinGPT text-generation run it reports lower training loss than Adam, AdamW, and AdEMAMix with competitive validation loss among the stable optimizers.
Load-bearing premise
The superiority claim rests on comparing every optimizer at learning rate $10^{-4}$ with default hyperparameters and one seed (1234), and on the unproven assumption that the new cross-moment terms preserve Adam's update stability.
Editorial extensions
If this is right
- EXAdam can replace Adam in existing training code with small elementwise additions, raising CIFAR-10 test accuracy to 90.83% from 89.66% for Adam and 89.71% for AdamW at the same learning rate and schedule.
- At the same 100-epoch schedule, EXAdam reaches its final training accuracy about 38% faster than Adam, so practitioners can stop training earlier or spend the saved compute elsewhere.
- The cross-moment corrections fade as $t$ grows, so EXAdam's early-training behavior differs from Adam but its asymptotic updates coincide with Adam's, making the change most consequential at the start of training.
- On the MinGPT/Shakespeare task, EXAdam achieves lower training loss than Adam, AdamW, and AdEMAMix with competitive validation loss, suggesting the benefit is not specific to convolutional image models.
- The reported 2.5% computational overhead is small enough that the optimizer can be used as a drop-in replacement without major throughput cost.
Reading between the lines
- The correction factors $\frac{v}{v+\epsilon}\beta_2^t$ and $\frac{m^2}{m^2+\epsilon}\beta_1^t$ are largest at $t=1$ and decay quickly, so a testable prediction is that EXAdam's advantage is concentrated in the first epochs and would shrink when training starts with a learning-rate warm-up.
- Because $\tilde g$ is added to $\tilde m$ inside the update, EXAdam is not a pure rescaling of Adam's direction; measuring the cosine similarity between EXAdam and Adam update vectors would isolate whether the gain comes from the cross-moment corrections or from the raw gradient injection.
- The 38.46% convergence gain is measured under a ReduceLROnPlateau scheduler that reacts to validation loss, so a natural extension is to check whether the gain survives under a fixed or cosine learning-rate schedule, since the scheduler itself can amplify or mask optimizer differences.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EXAdam, an Adam variant that modifies the debiasing terms (Eq. 1) and adds a gradient-based acceleration term (Eq. 2) to the update rule (Eq. 3). The stated contributions are improved moment estimation, faster convergence, and higher accuracy relative to Adam, supported by a CNN experiment on CIFAR-10 (38.46% faster convergence; 90.83% vs 89.66% test accuracy) and a MinGPT text-generation benchmark against several baseline optimizers. The manuscript also claims to provide a theoretical analysis of the new components. The central issue is that the theory is absent and the empirical evidence is reported with internal inconsistencies and insufficient rigor to support the claims.
Significance. If the claims were true, EXAdam would be a valuable low-overhead drop-in optimizer: the algorithm is simple to implement, the update cost is claimed to be only 2.5% above Adam, and the paper includes pseudocode and comparisons against seven baselines. However, the theoretical analysis promised in the abstract does not exist, the CIFAR-10 loss values are numerically inconsistent with the reported accuracies, and the paper's own MinGPT table contradicts the generalization claim. The empirical advantage over Adam is therefore not established, and the significance of the contribution cannot be assessed from the current evidence.
major comments (5)
- [Section 2, Eqs. (1)–(3)] The abstract and introduction promise a 'theoretical analysis,' but Section 2 contains no theorem, lemma, proposition, or proof. Statements such as 'the new terms asymptotically converge to the original Adam bias correction terms' are simple algebra, while claims of 'more stable and robust optimization process,' 'improved initial convergence and stability,' and 'enhanced ability to escape saddle points' are asserted without mathematical support. The convergence and stability properties of the update rule (Eq. 3) are central to the paper's value and need to be stated formally and proved (e.g., under standard smoothness or convexity assumptions, or at least with a bounded-update argument).
- [Section 3.1, Table 1] The reported loss values are not compatible with the reported accuracies under a standard cross-entropy loss. For Adam at epoch 1, training accuracy is 39.85%, so at least 60.15% of examples are misclassified; each misclassified example has loss at least ln 2 ≈ 0.693 (its true-class probability cannot exceed 1/2 if it is not the argmax), which already gives an average loss of at least about 0.42. The table reports a training loss of 0.01271. This is off by more than an order of magnitude and suggests the loss curves are rescaled or computed with an unspecified formula. Because the 38.46% faster-convergence claim is read off those loss curves (Figure 1 and Table 1), the central empirical claim cannot be checked without a full clarification and the raw cross-entropy values.
- [Section 3.1, paragraph after Table 2] The accuracy-gap calculations in the text are incorrect. From Table 2, EXAdam's testing accuracy exceeds Adam's by 1.17 percentage points, AdamW's by 1.12, and RMSProp's by 0.69, not by the stated '1.96, 2.44, and 2.39 percentage points, respectively.' Please correct these numbers and ensure the abstract and all other claims are consistent with Table 2, and distinguish percentage-point differences from relative percentage changes.
- [Section 3.2, Table 3] Table 3 contradicts the claim that EXAdam generalizes better than the stable baselines. At iteration 8000, EXAdam has training loss 0.892 and validation loss 1.571, while Adam, AdamW, and AdEMAMix have validation losses of 1.505, 1.494, and 1.502, respectively. The text's statement that the table 'demonstrates the superior performance of EXAdam compared to Adam, AdamW, SGD with Momentum, and AdEMAMix, as evidenced by its lower loss values on both the training and validation sets' is false on the validation side. EXAdam's training-validation gap is 0.679, versus 0.320 for Adam, indicating more overfitting rather than better generalization.
- [Section 3.1] The fair-comparison premise is not met. All optimizers use the same learning rate (0.0001) and a single random seed (1234), with no repeated runs or error bars; Section 3 itself concedes that this learning rate 'may not be optimal for each individual algorithm.' The advantage over Adam is a 1.17 percentage-point test-accuracy difference (90.83 vs 89.66), which could be attributable to the choice of seed or learning rate rather than to the algorithm. The claimed general superiority is therefore not established by the reported experiments; several seeds and a hyperparameter check are required.
minor comments (5)
- [Algorithm 1, line 9] The parentheses in the gradient-accelerator term are ambiguous: the text writes (1 + v/(v + εβ_t^2)) while Algorithm 1 line 9 can be read as (1 + v/(v+ε)β_t^2). Use the same bracketing in both places.
- [Title and Section 2] The title introduces 'cross-moments,' but the paper never defines a cross-moment or explains how Equations (1)–(2) constitute cross-moment estimation; please define the term or consider a more descriptive title.
- [Throughout] The writing alternates between first-person singular ('my experiments,' 'my contribution') and first-person plural ('we'), and includes informal passages such as the hiking analogy; a uniform, formal style would make the paper easier to assess.
- [Figures 1–3] The figure captions do not specify axes, whether the curves are plotted from raw batches or averaged, or whether any smoothing is applied; without this information the convergence-speed comparison cannot be reproduced from the figures.
- [Section 3.1, Table 1] The loss columns are given to five significant figures even though the loss function is never named; please state explicitly that these are cross-entropy values (or correct them if they are not).
Circularity Check
No significant circularity: EXAdam's empirical claims are tested against external optimizers, with no fitted parameters, self-citation chain, or uniqueness theorem making the conclusion equivalent to its inputs.
full rationale
EXAdam's central empirical claim (38.46% faster convergence and 90.83% test accuracy on CIFAR-10) is evaluated against Adam, AdamW, RMSProp, AdaDelta, and SGD with Momentum under a fixed training protocol. No parameter is fitted to the reported test outcome, and no result from the author's prior work is invoked to force the algorithm's choice. The update rules in Eqs. (1)-(3) define the proposed algorithm; the prose explanations such as 'When the gradients are noisy, indicated by a large second moment estimate v, the correction factor is closer to 1, which means the bias correction is more aggressive. This makes sense' are qualitative interpretations of the definition, not predictions derived from the equations in a way that reduces to the desired conclusion. There is no uniqueness theorem, no self-citation load-bearing step, and no fitted parameter renamed as a prediction. The stated 'theoretical analysis' is largely heuristic and lacks formal convergence proofs, and the empirical comparison rests on a single seed and one learning rate, but these are rigor and reproducibility concerns rather than circularity. The accuracy and loss comparisons are made against external baselines and are not constructed to equal EXAdam's claimed advantage by definition. Therefore the paper shows no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Adam's independent debiasing of first and second moments is a limitation that causes suboptimal updates.
- ad hoc to paper EXAdam's modified debiasing and added gradient term preserve or improve Adam's convergence and stability.
- domain assumption Comparing all optimizers at the same learning rate and without per-optimizer tuning is a fair test of superiority.
Cite this review
Pith. "Pith review of EXAdam: The Power of Adaptive Cross-Moments." pith.science (2026). https://pith.science/paper/DLDRIOY7
@misc{pith2026241220302,
author = {Pith},
title = {Pith review of: EXAdam: The Power of Adaptive Cross-Moments},
year = {2026},
howpublished = {\url{https://pith.science/paper/DLDRIOY7}},
note = {Machine review of arXiv:2412.20302}
}
abstract
This paper introduces EXAdam ($\textbf{EX}$tended $\textbf{Adam}$), a novel optimization algorithm that builds upon the widely-used Adam optimizer. EXAdam incorporates two key enhancements: (1) new debiasing terms for improved moment estimation and (2) a gradient-based acceleration mechanism for increased responsiveness to the current loss landscape. These innovations work synergistically to address limitations of the original Adam algorithm, potentially offering improved convergence properties, enhanced ability to escape saddle points, and potentially greater robustness to hyperparameter choices, though this requires further investigation. We provide a theoretical analysis of EXAdam's components and their interactions, highlighting the algorithm's potential advantages in navigating complex optimization landscapes. Empirical evaluations demonstrate EXAdam's superiority over Adam, achieving 38.46% faster convergence and yielding improvements of 1.96%, 2.17%, and 1.17% in training, validation, and testing accuracies, respectively, when applied to a CNN trained on the CIFAR-10 dataset. While these results are promising, further empirical validation across diverse tasks is essential to fully gauge EXAdam's efficacy. Nevertheless, EXAdam represents a significant advancement in adaptive optimization techniques, with promising implications for a wide range of machine learning applications. This work aims to contribute to the ongoing development of more efficient, adaptive, and universally applicable optimization methods in the field of machine learning and artificial intelligence.
Figures
Forward citations
Cited by 1 Pith paper
-
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.
Reference graph
Works this paper leans on
-
[1]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2017
2017
-
[2]
Alex Krizhevsky, Vinod Nair, Geoffrey Hinton, et al. The cifar-10 dataset, 2014
work page 2014
-
[3]
Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning, 2018
work page 2018
-
[4]
On large-batch training for deep learning: Generalization gap and sharp minima, 2017
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima, 2017
2017
-
[5]
Sharp minima can generalize for deep nets, 2017
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets, 2017
2017
-
[6]
A. S. Nemirovskii and Yurii Nesterov. Optimal methods of smooth convex minimization. Ussr Computational Mathematics and Mathematical Physics, 25:21–30, 1986
work page 1986
-
[7]
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, 2013
work page 2013
-
[8]
On the momentum term in gradient descent learning algorithms
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks : the official journal of the International Neural Network Society, 12:145–151, 02 1999
work page 1999
Show all 22 references
-
[9]
An overview of gradient descent optimization algorithms
Sebastian Ruder. An overview of gradient descent optimization algorithms. ArXiv, abs/1609.04747, 2016
2016 arXiv
-
[10]
Why momentum really works
Gabriel Goh. Why momentum really works. 2017
2017
-
[11]
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4:26–31, 2012
2012
-
[12]
Reddi, Satyen Kale, and Sanjiv Kumar
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019
1904 arXiv
-
[13]
Decoupled weight decay regularization, 2019
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019
2019
-
[14]
Adaptive gradient methods with dynamic bound of learning rate
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. ArXiv, abs/1902.09843, 2019
1902 arXiv
-
[15]
Pytorch: An imperative style, high-performance deep learning library, 2019
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K ¨opf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu F...
2019
-
[16]
Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012
2012
-
[17]
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
2018
-
[18]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023
2023
-
[19]
Adafactor: Adaptive learning rates with sublinear memory cost, 2018
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018
2018
-
[20]
The ademamix optimizer: Better, faster, older, 2024
Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older, 2024
2024
-
[21]
signsgd: Compressed optimisation for non-convex problems, 2018
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems, 2018. 12 EXAdam: The Power of Adaptive Cross-Moments A PREPRINT
2018
-
[22]
Deconstructing what makes a good optimizer for language models, 2024
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models, 2024. 13
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.