{"id":"e4b5726a-6699-4d54-8d67-009fb48a64a5","arxiv_id":"2412.20302","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"EXAdam changes Adam's bias correction and adds a gradient acceleration term, reporting modest accuracy and convergence gains on one CIFAR-10 run.","lead":"This paper proposes EXAdam, a modified version of the Adam optimizer with new bias-correction formulas and an extra gradient term meant to make training respond faster to the current loss landscape. On a single CIFAR-10 image-classification run, it reports around 1 to 2 percentage points higher accuracy and faster convergence than Adam, though the evidence is preliminary.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central superiority claim hinges on a single-seed CIFAR-10 run at one learning rate; the loss values in Table 1 are internally inconsistent with the accuracies, and the MinGPT validation results contradict the generalization claim, so the claimed 38.46% speedup and accuracy gains are not…","rationale":"I read the paper in good faith: it proposes a modified Adam optimizer and supports it mainly with a CIFAR-10 CNN comparison plus a MinGPT language-model comparison. For the central claim to hold, EXAdam would need to show robustly faster convergence and higher accuracy than Adam under an honest comparison. That condition fails on the present evidence. The reader's weakest assumption (fair comparison at a single learning rate and seed) is real, but I would put the emphasis even more sharply: the loss numbers that define convergence are internally implausible, and the second experiment's validation numbers point the other way. These are not accusations of misconduct; they are checkable properties of the reported tables. A multi-seed, multi-learning-rate reproduction would settle whether the observed advantage is robust. Since the paper itself concedes the learning-rate caveat and provides no error bars, the REJECT verdict stands without adjustment.","tokens_in":12338,"tokens_out":10025,"duration_ms":98593,"concrete_test":"Run the released EXAdam code on CIFAR-10 for at least 10 seeds with EXAdam and Adam at LR=1e-4, 3e-4, and 1e-3, using a predefined convergence metric (e.g., epochs to reach 85% validation accuracy) and reporting mean and standard deviation of test accuracy. If the 90.83 vs 89.66 accuracy gap and the 38.46% speedup do not persist across seeds and at least one alternative learning rate for Adam, the claimed general superiority is not established. In the same run, recompute the epoch-1 training loss from the checkpoint using standard cross-entropy to verify whether the reported 0.01271 is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the reported empirical comparison is a fair, reproducible demonstration that EXAdam converges faster and generalizes better than Adam. This premise is not met. Section 3.1 fixes one seed (1234) and one learning rate (1e-4) for all optimizers, and Section 3 concedes this may not be optimal for each individual algorithm. No error bars or repeated runs are reported, so the 1.17 percentage-point test-accuracy gap (90.83 vs 89.66) could be seed- or learning-rate-specific. More seriously, the loss columns in Table 1 are not consistent with a cross-entropy loss: Adam at epoch 1 has training loss 0.01271 at 39.85% accuracy, whereas any cross-entropy loss with 40% accuracy is at least about 0.42 (incorrect examples alone contribute at least ln 2 each) and typically near 1. This suggests the loss curves are rescaled or computed by an unspecified formula, so the 38.46% faster convergence derived from them cannot be checked. The second experiment does not rescue the claim: Table 3 shows EXAdam's final validation loss (1.571) is worse than Adam (1.505), AdamW (1.494), and AdEMAMix (1.502), while its training loss is lowest (0.892), i.e., more overfitting. The paper even misstates the CIFAR-10 gaps (calling the 1.17-point gain over Adam a 2.44-point gain). The central claim is therefore unsupported by the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EXAdam, an Adam variant that modifies the debiasing terms (Eq. 1) and adds a gradient-based acceleration term (Eq. 2) to the update rule (Eq. 3). The stated contributions are improved moment estimation, faster convergence, and higher accuracy relative to Adam, supported by a CNN experiment on CIFAR-10 (38.46% faster convergence; 90.83% vs 89.66% test accuracy) and a MinGPT text-generation benchmark against several baseline optimizers. The manuscript also claims to provide a theoretical analysis of the new components. The central issue is that the theory is absent and the empirical evidence is reported with internal inconsistencies and insufficient rigor to support the claims.","tokens_in":12658,"tokens_out":9821,"duration_ms":86118,"significance":"If the claims were true, EXAdam would be a valuable low-overhead drop-in optimizer: the algorithm is simple to implement, the update cost is claimed to be only 2.5% above Adam, and the paper includes pseudocode and comparisons against seven baselines. However, the theoretical analysis promised in the abstract does not exist, the CIFAR-10 loss values are numerically inconsistent with the reported accuracies, and the paper's own MinGPT table contradicts the generalization claim. The empirical advantage over Adam is therefore not established, and the significance of the contribution cannot be assessed from the current evidence.","major_comments":[{"comment":"The abstract and introduction promise a 'theoretical analysis,' but Section 2 contains no theorem, lemma, proposition, or proof. Statements such as 'the new terms asymptotically converge to the original Adam bias correction terms' are simple algebra, while claims of 'more stable and robust optimization process,' 'improved initial convergence and stability,' and 'enhanced ability to escape saddle points' are asserted without mathematical support. The convergence and stability properties of the update rule (Eq. 3) are central to the paper's value and need to be stated formally and proved (e.g., under standard smoothness or convexity assumptions, or at least with a bounded-update argument).","section":"Section 2, Eqs. (1)–(3)"},{"comment":"The reported loss values are not compatible with the reported accuracies under a standard cross-entropy loss. For Adam at epoch 1, training accuracy is 39.85%, so at least 60.15% of examples are misclassified; each misclassified example has loss at least ln 2 ≈ 0.693 (its true-class probability cannot exceed 1/2 if it is not the argmax), which already gives an average loss of at least about 0.42. The table reports a training loss of 0.01271. This is off by more than an order of magnitude and suggests the loss curves are rescaled or computed with an unspecified formula. Because the 38.46% faster-convergence claim is read off those loss curves (Figure 1 and Table 1), the central empirical claim cannot be checked without a full clarification and the raw cross-entropy values.","section":"Section 3.1, Table 1"},{"comment":"The accuracy-gap calculations in the text are incorrect. From Table 2, EXAdam's testing accuracy exceeds Adam's by 1.17 percentage points, AdamW's by 1.12, and RMSProp's by 0.69, not by the stated '1.96, 2.44, and 2.39 percentage points, respectively.' Please correct these numbers and ensure the abstract and all other claims are consistent with Table 2, and distinguish percentage-point differences from relative percentage changes.","section":"Section 3.1, paragraph after Table 2"},{"comment":"Table 3 contradicts the claim that EXAdam generalizes better than the stable baselines. At iteration 8000, EXAdam has training loss 0.892 and validation loss 1.571, while Adam, AdamW, and AdEMAMix have validation losses of 1.505, 1.494, and 1.502, respectively. The text's statement that the table 'demonstrates the superior performance of EXAdam compared to Adam, AdamW, SGD with Momentum, and AdEMAMix, as evidenced by its lower loss values on both the training and validation sets' is false on the validation side. EXAdam's training-validation gap is 0.679, versus 0.320 for Adam, indicating more overfitting rather than better generalization.","section":"Section 3.2, Table 3"},{"comment":"The fair-comparison premise is not met. All optimizers use the same learning rate (0.0001) and a single random seed (1234), with no repeated runs or error bars; Section 3 itself concedes that this learning rate 'may not be optimal for each individual algorithm.' The advantage over Adam is a 1.17 percentage-point test-accuracy difference (90.83 vs 89.66), which could be attributable to the choice of seed or learning rate rather than to the algorithm. The claimed general superiority is therefore not established by the reported experiments; several seeds and a hyperparameter check are required.","section":"Section 3.1"}],"minor_comments":[{"comment":"The parentheses in the gradient-accelerator term are ambiguous: the text writes (1 + v/(v + εβ_t^2)) while Algorithm 1 line 9 can be read as (1 + v/(v+ε)β_t^2). Use the same bracketing in both places.","section":"Algorithm 1, line 9"},{"comment":"The title introduces 'cross-moments,' but the paper never defines a cross-moment or explains how Equations (1)–(2) constitute cross-moment estimation; please define the term or consider a more descriptive title.","section":"Title and Section 2"},{"comment":"The writing alternates between first-person singular ('my experiments,' 'my contribution') and first-person plural ('we'), and includes informal passages such as the hiking analogy; a uniform, formal style would make the paper easier to assess.","section":"Throughout"},{"comment":"The figure captions do not specify axes, whether the curves are plotted from raw batches or averaged, or whether any smoothing is applied; without this information the convergence-speed comparison cannot be reproduced from the figures.","section":"Figures 1–3"},{"comment":"The loss columns are given to five significant figures even though the loss function is never named; please state explicitly that these are cross-entropy values (or correct them if they are not).","section":"Section 3.1, Table 1"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for publication: the theoretical content promised in the abstract is absent, and the experimental evidence has internal inconsistencies (Table 1 loss vs. accuracy; Table 3 validation losses; Table 2 gap arithmetic). These are load-bearing issues rather than cosmetic ones. I would be open to a resubmission if the authors rerun the experiments with a proper protocol (multiple seeds, correct loss reporting, a learning-rate ablation) and either add a real convergence analysis or remove the theoretical claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a new Adam variant with two genuinely new algebraic modifications—cross-moment debiasing (Eq. 1) and a gradient acceleration term (Eq. 2). The update rule is clearly specified and cheap to implement. That's the good news. The bad news is that the evidence offered for \"superiority\" does not hold up, and one of the tables contains numbers that cannot be real cross-entropy losses.\n\nWhat's new: the debiasing terms and the extra gradient term are not in the cited Adam/AdamW/AdEMAMix papers. The author gives a lot of prose about why they should help, and the algorithm itself is a legitimate variant. The paper is honest in places—it admits the shared learning rate may not be optimal, and it concedes the saddle-point claim is unproven.\n\nWhere it falls apart: Section 2 promises a theoretical analysis but has no theorem or proof. Section 3 reports one seed, one learning rate, no error bars. Table 1 lists Adam's epoch-1 training loss as 0.01271 at 39.85% accuracy; that is not a cross-entropy loss by any reasonable calculation—random-ish predictions on 10 classes give a loss around 2.3, and even perfect classification of 40% of examples plus no better than chance on the rest gives a floor above 0.4. So the loss curves, and the 38.46% convergence speedup derived from them, are unverifiable. The text also miscomputes the accuracy margins: EXAdam beats Adam by 1.17 points, not 2.44, and beats RMSProp by 0.69, not 1.96. The MinGPT experiment doesn't rescue anything: EXAdam's final validation loss (1.571) is worse than Adam's (1.505), AdamW's (1.494), and AdEMAMix's (1.502), while its training loss is lowest—a more-overfitting story, not a generalization win.\n\nSo: the central claim—consistent, faster, better-generalizing—is unsupported by the supplied evidence. The algorithm might be a decent Adam variant, but this paper doesn't show it.\n\nWho's this for? Someone who wants a concrete new optimizer to test could implement it from the pseudo-code. But as a scientific submission, it doesn't meet the evidentiary bar. I'd desk-reject this version, and tell the author to come back with multi-seed runs, error bars, proper per-optimizer tuning or sensitivity analysis, and loss curves that match the stated loss function. If the code is actually on GitHub, an independent run could settle it quickly.","headline":"Genuinely new Adam variant with a clearly specified update rule, but the empirical case is internally inconsistent—impossible loss values, miscomputed margins, and a language-model run that contradicts the generalization claim.","tokens_in":13187,"tokens_out":3683,"would_cite":false,"duration_ms":34319,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EXAdam modifies Adam with cross-moment debiasing and a gradient-acceleration term, reporting 38.46% faster convergence and 90.83% CIFAR-10 test accuracy, beating Adam's 89.66%.","keywords":["Adam optimizer","bias correction","cross-moment","gradient acceleration","adaptive learning rates","CIFAR-10","convergence speed","deep learning optimization"],"falsifier":"Re-run the CIFAR-10 CNN experiment across a grid of learning rates from $10^{-3}$ to $10^{-5}$ with several seeds; if Adam, AdamW, or RMSProp reaches or exceeds EXAdam's 90.83% test accuracy or matches its convergence speed under its own best configuration, the claimed general superiority fails.","tokens_in":12103,"feed_emoji":"📈","tokens_out":7552,"duration_ms":66791,"temperature":0.7,"pith_summary":"EXAdam is a proposed drop-in replacement for the Adam optimizer. Its update rule replaces Adam's bias-corrected moments $\\hat m$ and $\\hat v$ with cross-moment corrected versions $\\tilde m$ and $\\tilde v$, and adds a gradient-acceleration term $\\tilde g$ so that each parameter update mixes the smoothed momentum with the current gradient. The paper's central claim is that these two changes together converge faster and generalize better than Adam, with the headline evidence a CIFAR-10 CNN run where EXAdam reaches the same training accuracy in 38.46% fewer steps and reaches 90.83% test accuracy versus 89.66% for Adam. If true, this matters because EXAdam adds only about 2.5% computational overhead while improving on Adam, AdamW, and RMSProp in the reported experiments, offering practitioners a cheap way to speed up and improve training without changing model code.","feed_headline":"New Adam variant trains 38% faster on CIFAR-10","feed_subtitle":"EXAdam's cross-moment corrections and gradient boost lift test accuracy to 90.83% from Adam's 89.66%.","key_machinery":"The load-bearing mechanism is the cross-moment debiasing pair $\\tilde m$ and $\\tilde v$ together with the gradient accelerator $\\tilde g$ in Algorithm 1. The pair couples the first and second moment estimates: $\\tilde m$ scales the standard first-moment debias by $1 + \\frac{v}{v+\\epsilon}\\beta_2^t$, and $\\tilde v$ scales the second-moment debias by $1 + \\frac{m^2}{m^2+\\epsilon}\\beta_1^t$. Because $\\beta_1^t$ and $\\beta_2^t$ decay, these factors are largest in the first iterations, so the new terms mainly change early training and then fade toward Adam's corrections. The same variance ratio appears in $\\tilde g$, which adds the current gradient directly to the momentum term in the update, so steep regions with large $v$ get more weight from the instantaneous gradient while flat regions rely more on momentum.","core_discovery":"The paper claims that Adam's independent treatment of the first and second moment estimates leaves bias correction poorly matched to the local loss geometry, and that injecting the cross-moment factors $\\left(1 + \\frac{v}{v+\\epsilon}\\beta_2^t\\right)$ and $\\left(1 + \\frac{m^2}{m^2+\\epsilon}\\beta_1^t\\right)$ into the debiased estimates, together with a gradient accelerator $\\tilde g = \\frac{g}{1-\\beta_1^t}\\left(1 + \\frac{v}{v+\\epsilon}\\beta_2^t\\right)$, yields an update $\\theta \\leftarrow \\theta - \\alpha \\frac{\\tilde m + \\tilde g}{\\sqrt{\\tilde v} + \\epsilon}$ that is more responsive during early training and more stable later. As $t \\to \\infty$, the correction factors tend to $1$, so EXAdam asymptotically reduces to Adam. On CIFAR-10 with a CNN, the paper reports 38.46% faster convergence than Adam, gains of 1.96, 2.17, and 1.17 percentage points in training, validation, and test accuracy, and a test accuracy of 90.83% versus 89.66% for Adam; in a MinGPT text-generation run it reports lower training loss than Adam, AdamW, and AdEMAMix with competitive validation loss among the stable optimizers.","pith_inferences":["The correction factors $\\frac{v}{v+\\epsilon}\\beta_2^t$ and $\\frac{m^2}{m^2+\\epsilon}\\beta_1^t$ are largest at $t=1$ and decay quickly, so a testable prediction is that EXAdam's advantage is concentrated in the first epochs and would shrink when training starts with a learning-rate warm-up.","Because $\\tilde g$ is added to $\\tilde m$ inside the update, EXAdam is not a pure rescaling of Adam's direction; measuring the cosine similarity between EXAdam and Adam update vectors would isolate whether the gain comes from the cross-moment corrections or from the raw gradient injection.","The 38.46% convergence gain is measured under a ReduceLROnPlateau scheduler that reacts to validation loss, so a natural extension is to check whether the gain survives under a fixed or cosine learning-rate schedule, since the scheduler itself can amplify or mask optimizer differences."],"forward_implications":["EXAdam can replace Adam in existing training code with small elementwise additions, raising CIFAR-10 test accuracy to 90.83% from 89.66% for Adam and 89.71% for AdamW at the same learning rate and schedule.","At the same 100-epoch schedule, EXAdam reaches its final training accuracy about 38% faster than Adam, so practitioners can stop training earlier or spend the saved compute elsewhere.","The cross-moment corrections fade as $t$ grows, so EXAdam's early-training behavior differs from Adam but its asymptotic updates coincide with Adam's, making the change most consequential at the start of training.","On the MinGPT/Shakespeare task, EXAdam achieves lower training loss than Adam, AdamW, and AdEMAMix with competitive validation loss, suggesting the benefit is not specific to convolutional image models.","The reported 2.5% computational overhead is small enough that the optimizer can be used as a drop-in replacement without major throughput cost."],"supporting_citations":[{"why":"Defines the Adam baseline whose moment estimates and update rule EXAdam extends; all comparisons are against it.","marker":"[1]"},{"why":"Supplies the CIFAR-10 benchmark and its 10-class accuracy metric used for the headline 90.83% result.","marker":"[2]"},{"why":"Provides RMSProp, one of the optimizers compared against EXAdam on CIFAR-10 (90.83% vs 90.14% test accuracy).","marker":"[11]"},{"why":"Provides AdamW, a state-of-the-art baseline with decoupled weight decay that EXAdam also outperforms on CIFAR-10.","marker":"[13]"},{"why":"Supplies the deep-learning library used to implement the model, training loop, and all optimizer comparisons in the reported runs.","marker":"[15]"},{"why":"Provides AdaDelta, one of the optimizers in the CIFAR-10 comparison, representing the low end of the accuracy range.","marker":"[16]"},{"why":"Supplies the GPT architecture from which the MinGPT text-generation model is derived.","marker":"[17]"},{"why":"Supplies the transformer architecture used for the MinGPT text-generation experiment.","marker":"[18]"},{"why":"Provides AdaFactor, an optimizer compared in the MinGPT experiment that achieves lower training loss but higher validation loss.","marker":"[19]"},{"why":"Provides AdEMAMix, a recent optimizer used as a comparison baseline in the MinGPT experiment.","marker":"[20]"}],"fun_headline_variants":["EXAdam boosts Adam with cross-moments and gradient acceleration","Adam upgrade: cross-moment debiasing speeds up training by 38%","Cross-moment optimizer EXAdam trains neural nets 38% faster","EXAdam: adaptive cross-moments beat Adam on CIFAR-10","Adam with cross-moments: faster convergence, higher accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The superiority claim rests on comparing every optimizer at learning rate $10^{-4}$ with default hyperparameters and one seed (1234), and on the unproven assumption that the new cross-moment terms preserve Adam's update stability.","fun_headline_variants_meta":{"raw":{"variants":["EXAdam boosts Adam with cross-moments and gradient acceleration","Adam upgrade: cross-moment debiasing speeds up training by 38%","Cross-moment optimizer EXAdam trains neural nets 38% faster","EXAdam: adaptive cross-moments beat Adam on CIFAR-10","Adam with cross-moments: faster convergence, higher accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1740,"prompt_tokens":1093,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":552}},"tokens_in":709,"tokens_out":647,"duration_ms":6106,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:00.309022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CIFAR-10 CNN experiment across a grid of learning rates from $10^{-3}$ to $10^{-5}$ with several seeds; if Adam, AdamW, or RMSProp reaches or exceeds EXAdam's 90.83% test accuracy or matches its convergence speed under its own best configuration, the claimed general superiority fails.","supporting_citations":[{"cited_title":"The cifar-10 dataset, 2014","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 benchmark and its 10-class accuracy metric used for the headline 90.83% result."},{"cited_title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","cited_arxiv_id":null,"evidence_quote":"Provides RMSProp, one of the optimizers compared against EXAdam on CIFAR-10 (90.83% vs 90.14% test accuracy)."},{"cited_title":"Adafactor: Adaptive learning rates with sublinear memory cost, 2018","cited_arxiv_id":null,"evidence_quote":"Provides AdaFactor, an optimizer compared in the MinGPT experiment that achieves lower training loss but higher validation loss."}],"review_version":1}